Re: [testng-users] Test Suites, Ant, Surefire, and JunitReport

[Steve Loughran <st...@gmail.com>]

On Jan 24, 2008 10:12 PM, Dan Fabulich <da...@fabulich.com> wrote:
>

> Most of the JUnit reporting tools, including Ant's JunitReport task and
> Maven's surefire-report plugin (but also including Hudson and others)
> incorrectly assume that all of those testcases are methods of MyTestSuite.
> As a result, it makes it look like all of your tests are methods of one
> class in one package.  If you've got two "testOutput" methods, they'll be
> indistinguishable (except by stacktrace, if one is present).
>
> The same thing happens when you run TestNG tests: it always generates all
> of the test results in one XML file.  (Though it contains more
> information, see below.)
>
> More generally, both Ant's JunitReport and surefire-report incorrectly
> assume a one-to-one relationship between TestCase classes, XML files, and
> "suites" of tests.  They assume that every XML file contains one suite of
> tests, which is the same thing as a TestCase class, and all of the tests
> in the suite are just methods of the TestCase class.  This assumption is
> wrong.


I guess the reason for the assumption is that nobody normally writes a
test suite for JUnit, as it is needless coding that a pattern like
**/*Test.class could do at the build file level, where you can also
turn different tests on and off. But as you point out, it doesnt hold
for other test systems.

 > Ant 1.6 (I think?) introduced a helpful attribute on all of its <testcase>
> classes to include the class name of the class on every <testcase>
> element:
>
> Example 7
>      <testsuite name="com.mycompany.MyTestSuite" failures="0" errors="0" tests="4" time="0.031">
>        <properties><!-- ... --></properties
>        <testcase name="test1a" time="0.0" classname="com.mycompany.Test1" />
>        <testcase name="test1b" time="0.0" classname="com.mycompany.Test1" />
>        <testcase name="test2a" time="0.0" classname="com.mycompany.Test2" />
>        <testcase name="test2b" time="0.0" classname="com.mycompany.Test2" />
>        <testcase name="test3a" time="0.0" classname="com.mycompany.Test3" />
>        <testcase name="test3b" time="0.0" classname="com.mycompany.Test3" />
>      </testsuite>

I forget when, but take your word for it.

> This is good, because it allows you to figure out which class the method
> really belongs to.  This is what TestNG outputs when it generates
> JUnit-like results.  Unfortunately, nobody honors the "classname"
> attribute, including Ant 1.7's JunitReport task!

You contributions to the XSL file are welcome :)

>
> Even if we did try to honor that information and report on it in the HTML,
> how would we do it?  Right now, the reports are organized in terms of
> packages, and within the packages you'll get a list of classes (assumed
> incorrectly to be the same thing as "suites"), and within the list of
> classes you'll get a list of test method names.  There's no room for
> "suites" (as distinct from classes) in these reports at all.
>
> And arguably there shouldn't be a place for this information, because
> apparently 80-90% of the time suites aren't being used; adding a "suites"
> section would be redundant in those cases.
>
> We could throw away the "suite" wrapper and pretend as if the tests were
> just classes and methods, but note something else: the sum of the "time"
> attribute on the <testcase> elements is normally different from the "time"
> attribute on the <testsuite>.  That's because the timer has a 16ms
> resolution on most operating systems.

>
> In bug SUREFIRE-433, it was argued that we should generate separate JUnit
> XML files for every class.  You could even imagine converting the results
> from Example 7 into results that looked like Example 2, throwing away all
> of the suite-level information.
>
> I certainly want to be conservative in what I emit and liberal in what I
> accept, but I *really* don't want to throw away information, especially at
> the XML level.  With that said, as we're trying to make the report look
> something like the way it looks today, we're going to lose suite-level
> information in the HTML.
>
> What I propose is that, in order to avoid destroying information, Surefire
> should generate XML that looks like Example 7 (all-in-one-file), and not
> try to fake it to look like Example 2 (one-file-per-class).  (TestNG's
> junit-like output also generates files like Example 7.)  However, when it
> comes time to generate an HTML report, surefire-reports will discard
> suite-level information, and treat large suites like Example 7 as if they
> had been presented in separate files like Example 2.  I'd argue that all
> of the other JunitReport-like tools (including Ant) should probably follow
> the same lead.

I think that  if everyone else has a bug, its hard to call it a bug.
More a presentation choice :)

> My proposal is not obviously right, because, again, most other JUnit
> report tools also have this reporting bug; when they try to format TestNG
> output or Surefire output, they'll incorrectly report all methods to be
> members of the suite class.  Maybe since we expect EVERYONE to have this
> bug, we should be even more conservative in what we emit and generate
> multi-file output, just so the other tools will know how to interpret it
> correctly.  That would make it more likely that the HTML output would
> contain complete information.
>
> What do other people think?  Agree, disagree?

I do need to allocate time to fix reporting in Ant, both what we
generate (assuming a junit4 task), and what the tool processes. Me,
I'm more concerned with the problem of trying to merge log results
from a server and a client, as the current reports only pick the
stdout and stderr from one side of the party,

-steve

Re: [testng-users] Re: Test Suites, Ant, Surefire, and JunitReport

[Steve Loughran <st...@gmail.com>]

On Jan 25, 2008 5:04 PM, Dan Fabulich <da...@fabulich.com> wrote:
>
> Steve Loughran wrote:
>
> >> What I propose is that, in order to avoid destroying information, Surefire
> >> should generate XML that looks like Example 7 (all-in-one-file), and not
> >> try to fake it to look like Example 2 (one-file-per-class).  (TestNG's
> >> junit-like output also generates files like Example 7.)  However, when it
> >> comes time to generate an HTML report, surefire-reports will discard
> >> suite-level information, and treat large suites like Example 7 as if they
> >> had been presented in separate files like Example 2.  I'd argue that all
> >> of the other JunitReport-like tools (including Ant) should probably follow
> >> the same lead.
> >
> > I think that if everyone else has a bug, its hard to call it a bug. More
> > a presentation choice :)
>
> I'd thought somebody might say that! :-)  Still, do you think TestNG and
> other tools should therefore generate multiple XML files, to be compatible
> with the other (arguably broken) reporting tools?

No,

* junit report will take other sets of XSL sheets than those built in
to Ant's own JAR. This allows anyone to fix the XSLs without waiting
for ant's (fairly mature) release cycle.
* CI tools are a separate issue. Don't know there

Looking at the reports -and trying to imagine a time where I ever get
to stop making meta-RPM-RPMs (don't ask), I do want to push the test
formats forward in a way everyone is happy. I really dont like the way
the current stuff sticks summary info as toplevel attributes (time,
results), as it stops us streaming out the XML as we go. I'm starting
to wonder if we couldnt move to a new format, and have the option of
XSL generation of the legacy stuff, purely for those tools that havent
been upgraded in sync.

-steve

Re: [testng-users] Re: Test Suites, Ant, Surefire, and JunitReport

[Dan Fabulich <da...@fabulich.com>]

Steve Loughran wrote:

>> What I propose is that, in order to avoid destroying information, Surefire
>> should generate XML that looks like Example 7 (all-in-one-file), and not
>> try to fake it to look like Example 2 (one-file-per-class).  (TestNG's
>> junit-like output also generates files like Example 7.)  However, when it
>> comes time to generate an HTML report, surefire-reports will discard
>> suite-level information, and treat large suites like Example 7 as if they
>> had been presented in separate files like Example 2.  I'd argue that all
>> of the other JunitReport-like tools (including Ant) should probably follow
>> the same lead.
>
> I think that if everyone else has a bug, its hard to call it a bug. More 
> a presentation choice :)

I'd thought somebody might say that! :-)  Still, do you think TestNG and 
other tools should therefore generate multiple XML files, to be compatible 
with the other (arguably broken) reporting tools?

-Dan