You are viewing a plain text version of this content. The canonical link for it is here.
Posted to surefire-dev@maven.apache.org by Dan Fabulich <da...@fabulich.com> on 2008/01/24 23:12:15 UTC
Test Suites, Ant, Surefire, and JunitReport
I spent some time solving a thorny problem over the last couple of days;
I'm not 100% sure I like the solution to it, so I wanted to write a bit
about it to see what other people think.
The problem occurs in cases where one test suite incorporates tests from
multiple test classes. This is the normal use case for a JUnit TestSuite.
You'll typically write a JUnit suite like this:
Example 1
public class MyTestSuite extends TestSuite {
public static Test suite() {
MyTestSuite suite = new MyTestSuite();
suite.addTestSuite(FooTestCase.class);
suite.addTest(BarTestSuite.suite());
return suite;
}
}
(In TestNG, suites are defined using XML files, and are considerably more
common in TestNG-land.)
The question I'm trying to answer is: how should the results of running a
TestSuite be represented in XML, and how should that XML be represented in
HTML? You might think this would be a solved problem, but it doesn't
appear to be. :-( Bugs SUREFIRE-433 and TESTNG-204 all relate to this, as
well as Ant bug 24106.
http://jira.codehaus.org/browse/SUREFIRE-433
http://jira.opensymphony.com/browse/TESTNG-204
http://issues.apache.org/bugzilla/show_bug.cgi?id=24106
First, it seems like most people don't even use test suites, preferring to
simply run every test in their src/test/java directory. If you do this,
you get a bunch of XML files, one file per TestCase class, like this:
Example 2
TEST-com.mycompany.Test1.xml
TEST-com.mycompany.Test2.xml
TEST-com.mycompany.Test3.xml
The individual XML files look like this:
Example 3
<testsuite name="com.mycompany.Test1" failures="1" errors="0" tests="2" time="0.031">
<properties><!-- ... --></properties
<testcase name="test1a" time="0.0" />
<testcase name="test1b" time="0.0">
<failure type="java.lang.AssertionError" message="I don't like this movie!">java.lang.AssertionError: I don't like this movie!
at org.testng.Assert.fail(Assert.java:84)
at org.testng.Assert.fail(Assert.java:91)
at com.mycompany.Test1.test1b(Test1.java:14)
</failure>
</testcase>
</testsuite>
That's all fine, but what if you want to use a test suite? When you do
that in Ant or in Maven today, you get just one XML file:
Example 4
TEST-com.mycompany.MyTestSuite.xml
In Surefire 2.4, the XML unfortunately looks like this:
Example 5
<testsuite name="com.mycompany.MyTestSuite" failures="0" errors="0" tests="4" time="0.031">
<properties><!-- ... --></properties
<testcase name="test1a" time="0.0" />
<testcase name="test1b" time="0.0" />
<testcase name="test2a" time="0.0" />
<testcase name="test2b" time="0.0" />
<testcase name="test3a" time="0.0" />
<testcase name="test3b" time="0.0" />
</testsuite>
Note that information has been lost here, and the information that *is*
provided is misleading. Specifically, we lost the fact that test1a was a
method of com.mycompany.Test1; all we know is its method name.
Furthermore, the report misleadingly seems to suggest that test1a is a
method of MyTestSuite, whereas in fact MyTestSuite just has a suite()
method and no test methods of its own.
Most of the JUnit reporting tools, including Ant's JunitReport task and
Maven's surefire-report plugin (but also including Hudson and others)
incorrectly assume that all of those testcases are methods of MyTestSuite.
As a result, it makes it look like all of your tests are methods of one
class in one package. If you've got two "testOutput" methods, they'll be
indistinguishable (except by stacktrace, if one is present).
The same thing happens when you run TestNG tests: it always generates all
of the test results in one XML file. (Though it contains more
information, see below.)
More generally, both Ant's JunitReport and surefire-report incorrectly
assume a one-to-one relationship between TestCase classes, XML files, and
"suites" of tests. They assume that every XML file contains one suite of
tests, which is the same thing as a TestCase class, and all of the tests
in the suite are just methods of the TestCase class. This assumption is
wrong.
Ant 1.6 (I think?) introduced a helpful attribute on all of its <testcase>
classes to include the class name of the class on every <testcase>
element:
Example 7
<testsuite name="com.mycompany.MyTestSuite" failures="0" errors="0" tests="4" time="0.031">
<properties><!-- ... --></properties
<testcase name="test1a" time="0.0" classname="com.mycompany.Test1" />
<testcase name="test1b" time="0.0" classname="com.mycompany.Test1" />
<testcase name="test2a" time="0.0" classname="com.mycompany.Test2" />
<testcase name="test2b" time="0.0" classname="com.mycompany.Test2" />
<testcase name="test3a" time="0.0" classname="com.mycompany.Test3" />
<testcase name="test3b" time="0.0" classname="com.mycompany.Test3" />
</testsuite>
This is good, because it allows you to figure out which class the method
really belongs to. This is what TestNG outputs when it generates
JUnit-like results. Unfortunately, nobody honors the "classname"
attribute, including Ant 1.7's JunitReport task!
Even if we did try to honor that information and report on it in the HTML,
how would we do it? Right now, the reports are organized in terms of
packages, and within the packages you'll get a list of classes (assumed
incorrectly to be the same thing as "suites"), and within the list of
classes you'll get a list of test method names. There's no room for
"suites" (as distinct from classes) in these reports at all.
And arguably there shouldn't be a place for this information, because
apparently 80-90% of the time suites aren't being used; adding a "suites"
section would be redundant in those cases.
We could throw away the "suite" wrapper and pretend as if the tests were
just classes and methods, but note something else: the sum of the "time"
attribute on the <testcase> elements is normally different from the "time"
attribute on the <testsuite>. That's because the timer has a 16ms
resolution on most operating systems. By starting the timer at the
beginning of the entire suite, we can capture overall time that we lose
when we turn the timer on and off. (And what if the tests had been run in
parallel?)
In bug SUREFIRE-433, it was argued that we should generate separate JUnit
XML files for every class. You could even imagine converting the results
from Example 7 into results that looked like Example 2, throwing away all
of the suite-level information.
I certainly want to be conservative in what I emit and liberal in what I
accept, but I *really* don't want to throw away information, especially at
the XML level. With that said, as we're trying to make the report look
something like the way it looks today, we're going to lose suite-level
information in the HTML.
What I propose is that, in order to avoid destroying information, Surefire
should generate XML that looks like Example 7 (all-in-one-file), and not
try to fake it to look like Example 2 (one-file-per-class). (TestNG's
junit-like output also generates files like Example 7.) However, when it
comes time to generate an HTML report, surefire-reports will discard
suite-level information, and treat large suites like Example 7 as if they
had been presented in separate files like Example 2. I'd argue that all
of the other JunitReport-like tools (including Ant) should probably follow
the same lead.
My proposal is not obviously right, because, again, most other JUnit
report tools also have this reporting bug; when they try to format TestNG
output or Surefire output, they'll incorrectly report all methods to be
members of the suite class. Maybe since we expect EVERYONE to have this
bug, we should be even more conservative in what we emit and generate
multi-file output, just so the other tools will know how to interpret it
correctly. That would make it more likely that the HTML output would
contain complete information.
What do other people think? Agree, disagree?
-Dan
Re: [testng-users] Re: Test Suites, Ant, Surefire, and JunitReport
Posted by Steve Loughran <st...@gmail.com>.
On Jan 25, 2008 5:04 PM, Dan Fabulich <da...@fabulich.com> wrote:
>
> Steve Loughran wrote:
>
> >> What I propose is that, in order to avoid destroying information, Surefire
> >> should generate XML that looks like Example 7 (all-in-one-file), and not
> >> try to fake it to look like Example 2 (one-file-per-class). (TestNG's
> >> junit-like output also generates files like Example 7.) However, when it
> >> comes time to generate an HTML report, surefire-reports will discard
> >> suite-level information, and treat large suites like Example 7 as if they
> >> had been presented in separate files like Example 2. I'd argue that all
> >> of the other JunitReport-like tools (including Ant) should probably follow
> >> the same lead.
> >
> > I think that if everyone else has a bug, its hard to call it a bug. More
> > a presentation choice :)
>
> I'd thought somebody might say that! :-) Still, do you think TestNG and
> other tools should therefore generate multiple XML files, to be compatible
> with the other (arguably broken) reporting tools?
No,
* junit report will take other sets of XSL sheets than those built in
to Ant's own JAR. This allows anyone to fix the XSLs without waiting
for ant's (fairly mature) release cycle.
* CI tools are a separate issue. Don't know there
Looking at the reports -and trying to imagine a time where I ever get
to stop making meta-RPM-RPMs (don't ask), I do want to push the test
formats forward in a way everyone is happy. I really dont like the way
the current stuff sticks summary info as toplevel attributes (time,
results), as it stops us streaming out the XML as we go. I'm starting
to wonder if we couldnt move to a new format, and have the option of
XSL generation of the legacy stuff, purely for those tools that havent
been upgraded in sync.
-steve
Re: [testng-users] Re: Test Suites, Ant, Surefire, and JunitReport
Posted by Dan Fabulich <da...@fabulich.com>.
Steve Loughran wrote:
>> What I propose is that, in order to avoid destroying information, Surefire
>> should generate XML that looks like Example 7 (all-in-one-file), and not
>> try to fake it to look like Example 2 (one-file-per-class). (TestNG's
>> junit-like output also generates files like Example 7.) However, when it
>> comes time to generate an HTML report, surefire-reports will discard
>> suite-level information, and treat large suites like Example 7 as if they
>> had been presented in separate files like Example 2. I'd argue that all
>> of the other JunitReport-like tools (including Ant) should probably follow
>> the same lead.
>
> I think that if everyone else has a bug, its hard to call it a bug. More
> a presentation choice :)
I'd thought somebody might say that! :-) Still, do you think TestNG and
other tools should therefore generate multiple XML files, to be compatible
with the other (arguably broken) reporting tools?
-Dan
Re: [testng-users] Test Suites, Ant, Surefire, and JunitReport
Posted by Steve Loughran <st...@gmail.com>.
On Jan 24, 2008 10:12 PM, Dan Fabulich <da...@fabulich.com> wrote:
>
> Most of the JUnit reporting tools, including Ant's JunitReport task and
> Maven's surefire-report plugin (but also including Hudson and others)
> incorrectly assume that all of those testcases are methods of MyTestSuite.
> As a result, it makes it look like all of your tests are methods of one
> class in one package. If you've got two "testOutput" methods, they'll be
> indistinguishable (except by stacktrace, if one is present).
>
> The same thing happens when you run TestNG tests: it always generates all
> of the test results in one XML file. (Though it contains more
> information, see below.)
>
> More generally, both Ant's JunitReport and surefire-report incorrectly
> assume a one-to-one relationship between TestCase classes, XML files, and
> "suites" of tests. They assume that every XML file contains one suite of
> tests, which is the same thing as a TestCase class, and all of the tests
> in the suite are just methods of the TestCase class. This assumption is
> wrong.
I guess the reason for the assumption is that nobody normally writes a
test suite for JUnit, as it is needless coding that a pattern like
**/*Test.class could do at the build file level, where you can also
turn different tests on and off. But as you point out, it doesnt hold
for other test systems.
> Ant 1.6 (I think?) introduced a helpful attribute on all of its <testcase>
> classes to include the class name of the class on every <testcase>
> element:
>
> Example 7
> <testsuite name="com.mycompany.MyTestSuite" failures="0" errors="0" tests="4" time="0.031">
> <properties><!-- ... --></properties
> <testcase name="test1a" time="0.0" classname="com.mycompany.Test1" />
> <testcase name="test1b" time="0.0" classname="com.mycompany.Test1" />
> <testcase name="test2a" time="0.0" classname="com.mycompany.Test2" />
> <testcase name="test2b" time="0.0" classname="com.mycompany.Test2" />
> <testcase name="test3a" time="0.0" classname="com.mycompany.Test3" />
> <testcase name="test3b" time="0.0" classname="com.mycompany.Test3" />
> </testsuite>
I forget when, but take your word for it.
> This is good, because it allows you to figure out which class the method
> really belongs to. This is what TestNG outputs when it generates
> JUnit-like results. Unfortunately, nobody honors the "classname"
> attribute, including Ant 1.7's JunitReport task!
You contributions to the XSL file are welcome :)
>
> Even if we did try to honor that information and report on it in the HTML,
> how would we do it? Right now, the reports are organized in terms of
> packages, and within the packages you'll get a list of classes (assumed
> incorrectly to be the same thing as "suites"), and within the list of
> classes you'll get a list of test method names. There's no room for
> "suites" (as distinct from classes) in these reports at all.
>
> And arguably there shouldn't be a place for this information, because
> apparently 80-90% of the time suites aren't being used; adding a "suites"
> section would be redundant in those cases.
>
> We could throw away the "suite" wrapper and pretend as if the tests were
> just classes and methods, but note something else: the sum of the "time"
> attribute on the <testcase> elements is normally different from the "time"
> attribute on the <testsuite>. That's because the timer has a 16ms
> resolution on most operating systems.
>
> In bug SUREFIRE-433, it was argued that we should generate separate JUnit
> XML files for every class. You could even imagine converting the results
> from Example 7 into results that looked like Example 2, throwing away all
> of the suite-level information.
>
> I certainly want to be conservative in what I emit and liberal in what I
> accept, but I *really* don't want to throw away information, especially at
> the XML level. With that said, as we're trying to make the report look
> something like the way it looks today, we're going to lose suite-level
> information in the HTML.
>
> What I propose is that, in order to avoid destroying information, Surefire
> should generate XML that looks like Example 7 (all-in-one-file), and not
> try to fake it to look like Example 2 (one-file-per-class). (TestNG's
> junit-like output also generates files like Example 7.) However, when it
> comes time to generate an HTML report, surefire-reports will discard
> suite-level information, and treat large suites like Example 7 as if they
> had been presented in separate files like Example 2. I'd argue that all
> of the other JunitReport-like tools (including Ant) should probably follow
> the same lead.
I think that if everyone else has a bug, its hard to call it a bug.
More a presentation choice :)
> My proposal is not obviously right, because, again, most other JUnit
> report tools also have this reporting bug; when they try to format TestNG
> output or Surefire output, they'll incorrectly report all methods to be
> members of the suite class. Maybe since we expect EVERYONE to have this
> bug, we should be even more conservative in what we emit and generate
> multi-file output, just so the other tools will know how to interpret it
> correctly. That would make it more likely that the HTML output would
> contain complete information.
>
> What do other people think? Agree, disagree?
I do need to allocate time to fix reporting in Ant, both what we
generate (assuming a junit4 task), and what the tool processes. Me,
I'm more concerned with the problem of trying to merge log results
from a server and a client, as the current reports only pick the
stdout and stderr from one side of the party,
-steve
Re: Test Suites, Ant, Surefire, and JunitReport
Posted by Benjamin Bentmann <be...@udo.edu>.
dfabulich wrote:
>
> As I said, I don't think you're using the right tool for the job here.
>
Hm, I am not really in the mood to start off a discussion with my project
manager ;-)
dfabulich wrote:
>
> You can get the exact behavior you want from JUnit.
>
And I could get it from TestNG if I passed each class in as an individual
suite, couldn't I?
dfabulich wrote:
>
> Again, if you want class-level logging, I encourage you to take up the
> matter with the TestNG team.
>
Might be a good idea. As some kind of explanation why I started this
discussion on the Surefire corner at all: Surefire 2.3.1 did what I wanted
so the first impression after updating to Surefire 2.4 was that it got
messed up.
dfabulich wrote:
>
> Doesn't this option smell bad to you?
>
It's a rather technical/sophisticated option, I agree about that. But aren't
"useSystemClassLoader", "childDelegation" and "forkMode" likewise? Whoever
develops tests needs to have proper knowledge about their requirements on
the execution runtime and hence, needs to configure the test runner
appropriately. The same would apply for this option, default it to "all",
let the experts change it to meet their special needs.
dfabulich wrote:
>
> Instead of making this option, wouldn't it be nice if it just logged
> correctly? If TestNG notified us when to log, we could log anything we
> wanted.
>
I fear having just another lister in the TestNG framework is not enough. The
problem I see here is the following: If Sufefire passes all test classes
(say A and B) as a single suite to TestNG, what would TestNG prevent from
running them like this:
A.testOne
B.testOne
A.testTwo
B.testTwo
A.testThree
B.testThree
Our test methods have no dependencies that would require a deterministic
order. Even if TestNG exposed events for starting/stopping a test class, I
still would not get a guarantee that all methods from a single class are run
without inter-leaving with methods from other classes to give me that
class-level testing feeling. I hardly can see an argument how one could
request a change on this behavior. A suite is the outer-most group of tests
and as long as I do not specify dependencies, TestNG is free in ordering
them.
dfabulich wrote:
>
> Surefire is meant to be a wrapper around the underlying test framework;
>
No problems with that. But it's Surefire and not the test framework that
aggregates my test classes into a single suite. I am basically asking for an
option to control the way how Surefire assembles the suites *before* it
passes control to the test framework.
Benjamin Bentmann
--
View this message in context: http://www.nabble.com/Test-Suites%2C-Ant%2C-Surefire%2C-and-JunitReport-tp15076378p15138325.html
Sent from the Surefire - Developer mailing list archive at Nabble.com.
Re: Test Suites, Ant, Surefire, and JunitReport
Posted by Dan Fabulich <da...@fabulich.com>.
Benjamin Bentmann wrote:
> But what I really want is to avoid bothering with suites at all. Sure,
> TestNG has a powerful suite concept, nice, maybe we may want to use it
> someday in the future but for now, we have just the simple use-case with
> non-related test classes, i.e. we are happy by grouping at the class
> level.
As I said, I don't think you're using the right tool for the job here.
TestNG is meant to support all that fancy stuff, and behaves a little
differently for that reason. You can get the exact behavior you want from
JUnit.
> We have no suite(s), only a directory with some unrelated test classes.
> Now Surefire goes ahead and creates a suite, automatically, without my
> control, assuming the entire directory is a single suite. This
> assumption does not fit our needs.
That's fair, but I claim that this assumption is a good one for people who
are using TestNG, most of whom pick TestNG over JUnit because they need to
do that weird stuff.
Again, if you want class-level logging, I encourage you to take up the
matter with the TestNG team. TestNG isn't meant to work this way today.
> dfabulich wrote:
>>
>> That option sounds really confusing. What would you call it? How would
>> you document it? You'd have to know a lot about the guts of Surefire and
>> TestNG for an option like that to make much sense.
>>
> Could think of something like "suiteMode=class|package|all":
> This option controls how Surefire will assemble your test classes into a
> suite. "class" means to run each class individually in its own suite.
> "package" groups all classes from the same package into a separate suite.
> "all" will aggregate all found test classes in a single suite. Each created
> suite will then be passed to the testing framework for execution and will be
> run independently of the other suites. Note that test classes that have
> dependencies on each other, must be executed in the same suite. Therefore,
> the values "class" and "package" should be used with care because Surefire
> itself cannot detect such inter-class dependencies.
Doesn't this option smell bad to you? It does to me!
Instead of making this option, wouldn't it be nice if it just logged
correctly? If TestNG notified us when to log, we could log anything we
wanted.
Surefire is meant to be a wrapper around the underlying test framework;
it's not meant to provide features that the frameworks lack. I think this
is one of those cases where trying to provide the feature in Surefire
would be way harder, and would also be fighting the framework.
> Say one starts of with JUnit, where Surefire executes classes
> individually. Then one goes and converts his tests to use TestNG (they
> have a tool for this). Conceptually, you have the same test design, but
> now Surefire insists on running all classes in a single job/black-box. I
> am sorry, but I still find this odd since I am lacking the required
> knowledge about the guts of Surefire and TestNG ;-)
For the record, I agree that the difference between TestNG and JUnit is
strange. I claim that we only find it odd BECAUSE we have painfully
learned about these guts... most people don't notice or care about the
difference.
-Dan
Re: Test Suites, Ant, Surefire, and JunitReport
Posted by Jason van Zyl <ja...@maven.org>.
On 27-Jan-08, at 12:31 PM, Benjamin Bentmann wrote:
>
>
> Jason van Zyl-2 wrote:
>>
>> Correct me if I'm wrong here Benjamin but I think he's talking about
>> stating what tests you want to run, and how they are grouped
>> declaratively. As a best practice don't use suites and come up with a
>> way in Surefire to group things and avoid grouping of things in code.
>>
>
> Not quite sure if I understand you correctly. A TestNG suite XML
> file is
> quite declarative so I already have the possibility to declaratively
> select
> and group tests. In fear that I can't get Surefire where I wished
> (one-class-per-suite), I opened http://jira.codehaus.org/browse/SUREFIRE-438
> which would allow to easily feed a bunch of suite XML files into the
> plugin.
>
> But what I really want is to avoid bothering with suites at all.
Sorry if I didn't explain it well but we are in 100% agreement. Make
tests simple, no suites, and group declaratively.
> Sure,
> TestNG has a powerful suite concept, nice, maybe we may want to use it
> someday in the future but for now, we have just the simple use-case
> with
> non-related test classes, i.e. we are happy by grouping at the class
> level.
> Explicitly maintaining suites, either in code or via XML, would just
> be lost
> worktime here (although I already thought about a plugin that
> automatically
> creates single-class suites from the existing test classes and
> prepares the
> dozens of suite files before the actual Surefire run...)
>
> We have no suite(s), only a directory with some unrelated test
> classes. Now
> Surefire goes ahead and creates a suite, automatically, without my
> control,
> assuming the entire directory is a single suite. This assumption
> does not
> fit our needs.
>
>
> dfabulich wrote:
>>
>> That option sounds really confusing. What would you call it? How
>> would
>> you document it? You'd have to know a lot about the guts of
>> Surefire and
>> TestNG for an option like that to make much sense.
>>
> Could think of something like "suiteMode=class|package|all":
> This option controls how Surefire will assemble your test classes
> into a
> suite. "class" means to run each class individually in its own suite.
> "package" groups all classes from the same package into a separate
> suite.
> "all" will aggregate all found test classes in a single suite. Each
> created
> suite will then be passed to the testing framework for execution and
> will be
> run independently of the other suites. Note that test classes that
> have
> dependencies on each other, must be executed in the same suite.
> Therefore,
> the values "class" and "package" should be used with care because
> Surefire
> itself cannot detect such inter-class dependencies.
>
>
>
> dfabulich wrote:
>>
>> "TestNG is designed to cover all categories of tests: unit, [...]"
>> [...]
>> If all you're doing is unit tests, you may prefer to use JUnit 4. :-)
>>
>
> Well, no comments ;-)
>
> Say one starts of with JUnit, where Surefire executes classes
> individually.
> Then one goes and converts his tests to use TestNG (they have a tool
> for
> this). Conceptually, you have the same test design, but now Surefire
> insists
> on running all classes in a single job/black-box. I am sorry, but I
> still
> find this odd since I am lacking the required knowledge about the
> guts of
> Surefire and TestNG ;-)
>
> Regards,
>
>
> Benjamin Bentmann
> --
> View this message in context: http://www.nabble.com/Test-Suites%2C-Ant%2C-Surefire%2C-and-JunitReport-tp15076378p15123679.html
> Sent from the Surefire - Developer mailing list archive at Nabble.com.
>
Thanks,
Jason
----------------------------------------------------------
Jason van Zyl
Founder, Apache Maven
jason at sonatype dot com
----------------------------------------------------------
Simplex sigillum veri. (Simplicity is the seal of truth.)
Re: Test Suites, Ant, Surefire, and JunitReport
Posted by Benjamin Bentmann <be...@udo.edu>.
Jason van Zyl-2 wrote:
>
> Correct me if I'm wrong here Benjamin but I think he's talking about
> stating what tests you want to run, and how they are grouped
> declaratively. As a best practice don't use suites and come up with a
> way in Surefire to group things and avoid grouping of things in code.
>
Not quite sure if I understand you correctly. A TestNG suite XML file is
quite declarative so I already have the possibility to declaratively select
and group tests. In fear that I can't get Surefire where I wished
(one-class-per-suite), I opened http://jira.codehaus.org/browse/SUREFIRE-438
which would allow to easily feed a bunch of suite XML files into the plugin.
But what I really want is to avoid bothering with suites at all. Sure,
TestNG has a powerful suite concept, nice, maybe we may want to use it
someday in the future but for now, we have just the simple use-case with
non-related test classes, i.e. we are happy by grouping at the class level.
Explicitly maintaining suites, either in code or via XML, would just be lost
worktime here (although I already thought about a plugin that automatically
creates single-class suites from the existing test classes and prepares the
dozens of suite files before the actual Surefire run...)
We have no suite(s), only a directory with some unrelated test classes. Now
Surefire goes ahead and creates a suite, automatically, without my control,
assuming the entire directory is a single suite. This assumption does not
fit our needs.
dfabulich wrote:
>
> That option sounds really confusing. What would you call it? How would
> you document it? You'd have to know a lot about the guts of Surefire and
> TestNG for an option like that to make much sense.
>
Could think of something like "suiteMode=class|package|all":
This option controls how Surefire will assemble your test classes into a
suite. "class" means to run each class individually in its own suite.
"package" groups all classes from the same package into a separate suite.
"all" will aggregate all found test classes in a single suite. Each created
suite will then be passed to the testing framework for execution and will be
run independently of the other suites. Note that test classes that have
dependencies on each other, must be executed in the same suite. Therefore,
the values "class" and "package" should be used with care because Surefire
itself cannot detect such inter-class dependencies.
dfabulich wrote:
>
> "TestNG is designed to cover all categories of tests: unit, [...]"
> [...]
> If all you're doing is unit tests, you may prefer to use JUnit 4. :-)
>
Well, no comments ;-)
Say one starts of with JUnit, where Surefire executes classes individually.
Then one goes and converts his tests to use TestNG (they have a tool for
this). Conceptually, you have the same test design, but now Surefire insists
on running all classes in a single job/black-box. I am sorry, but I still
find this odd since I am lacking the required knowledge about the guts of
Surefire and TestNG ;-)
Regards,
Benjamin Bentmann
--
View this message in context: http://www.nabble.com/Test-Suites%2C-Ant%2C-Surefire%2C-and-JunitReport-tp15076378p15123679.html
Sent from the Surefire - Developer mailing list archive at Nabble.com.
Re: Test Suites, Ant, Surefire, and JunitReport
Posted by Dan Fabulich <da...@fabulich.com>.
Benjamin Bentmann wrote:
> TestNG is not designed only for the fancy tests, it can run the simple unit
> tests, too. Dan, to me it looks you/Surefire are pushing something like "If
> you want unit tests, use JUnit, if you want function/integration tests, use
> TestNG" by punishing those that employ TestNG for the simple use-case. I am
> neither a JUnit nor TestNG fanatic but I can't follow your XOR-like argument
> here.
You CAN use TestNG for unit tests; the simple use case does work. It's
just not as nice as you might wish it were. My argument is that this
isn't very important because there's a better alternative available for
unit tests.
-Dan
Re: Test Suites, Ant, Surefire, and JunitReport
Posted by Benjamin Bentmann <be...@udo.edu>.
dfabulich wrote:
>
> Inter-class dependencies are one of TestNG's fundamental features
>
Use of this feature is optional. For my sake, call it an "abuse" of TestNG
if I run that simple unit tests without inter-class dependencies ;-)
dfabulich wrote:
>
> [...] but TestNG is meant to support functional/integration tests.
>
If I may requote the TestNG marketing slogan, again:
> TestNG is designed to cover all categories of tests: *unit*, [...]
TestNG is not designed only for the fancy tests, it can run the simple unit
tests, too. Dan, to me it looks you/Surefire are pushing something like "If
you want unit tests, use JUnit, if you want function/integration tests, use
TestNG" by punishing those that employ TestNG for the simple use-case. I am
neither a JUnit nor TestNG fanatic but I can't follow your XOR-like argument
here.
Benjamin Bentmann
--
View this message in context: http://www.nabble.com/Test-Suites%2C-Ant%2C-Surefire%2C-and-JunitReport-tp15076378p15136338.html
Sent from the Surefire - Developer mailing list archive at Nabble.com.
Re: Test Suites, Ant, Surefire, and JunitReport
Posted by Dan Fabulich <da...@fabulich.com>.
Jason van Zyl wrote:
> Correct me if I'm wrong here Benjamin but I think he's talking about
> stating what tests you want to run, and how they are grouped
> declaratively. As a best practice don't use suites and come up with a
> way in Surefire to group things and avoid grouping of things in code.
Inter-class dependencies are one of TestNG's fundamental features; it's
arguably the most important architectural difference between TestNG and
JUnit. As Benjamin pointed out, they are declarative.
> I think it would just be a practice for class to be self-contained, test
> one thing, and allow wiring up of sets/groups externally.
That's a good practice for unit tests, but TestNG is meant to support
functional/integration tests. Unlike unit tests, which are meant to be
isolated and test one thing, TestNG tests are supposed to test more than
one thing together, and are therefore designed to inter-depend on one
another.
It's a pretty significant philosophical shift; the TestNG guys wrote a
whole book about it, blah blah blah. :-)
Dependent test methods
http://beust.com/weblog/archives/000171.html
Are dependent test methods really evil?
http://beust.com/weblog/archives/000259.html
Why unit tests are disappearing
http://beust.com/weblog/archives/000319.html
> If this model breaks down then maybe, as I've asked before, it makes sense to
> create separate plugins for the various test tools.
I'm still on the fence about that, but regardless I don't think this is a
good candidate reason. Surefire has to handle this problem in the case of
JUnit suites as well as TestNG suites; the only difference is that in
JUnit this problem is less important because we think people mostly
shouldn't write unit tests like that.
-Dan
Re: Test Suites, Ant, Surefire, and JunitReport
Posted by Jason van Zyl <ja...@maven.org>.
On 25-Jan-08, at 10:59 AM, Dan Fabulich wrote:
> Benjamin Bentmann wrote:
>
>> I see. Now, if the user can provide an option to Surefire,
>> indicating that the unit tests do not have inter-class
>> dependencies, wouldn't this allow to get back the old behavior?
>> I.e. enabling Surefire to pass test classes individually into TestNG?
>
> That option sounds really confusing. What would you call it? How
> would you document it? You'd have to know a lot about the guts of
> Surefire and TestNG for an option like that to make much sense.
>
Correct me if I'm wrong here Benjamin but I think he's talking about
stating what tests you want to run, and how they are grouped
declaratively. As a best practice don't use suites and come up with a
way in Surefire to group things and avoid grouping of things in code.
If this model breaks down then maybe, as I've asked before, it makes
sense to create separate plugins for the various test tools.
> Not to mention that it'd be error prone.
I think it would just be a practice for class to be self-contained,
test one thing, and allow wiring up of sets/groups externally.
> Developer X would assess the tests, verifying that there are no
> inter-class dependencies, and turn on the flag. Later, developer Y
> could introduce an inter-class dependency and go to run the tests,
> but find that it's not working. It'd be hard to figure out why
> without knowing a lot about how Surefire is doing its job.
>
> More generally, I don't think class-level reporting really makes
> sense in TestNG as it's designed today. Right now, TestNG provides
> listeners with the following notifications:
>
> onStart
> onTestStart
> onTestSuccess
> onTestFailure
> onTestSkipped
> onTestFailedButWithinSuccessPercentage
> onConfigurationSuccess
> onConfigurationFailure
> onConfigurationSkip
> onFinish
>
> Note that there's nothing here about "onClassStart/Finish" or even
> "onGroupStart/Finish".
>
> If TestNG exposed a listener, we could easily attach to it, but I
> see that as a feature of TestNG. We could certainly try to write
> TestNG features in Surefire, but the TestNG code knows best whether,
> for example, your classes have any inter-dependencies, when your
> classes start and stop, and more generally how to construct a full
> test run comprised of suites, groups, classes and tests.
>
> Surefire shouldn't be in the business of reimplementing that logic,
> especially since we expect this feature to be generally useful. It
> should be a part of TestNG, if it matters at all.
>
>> I would not have any problems with a single file per method.
>> Indeed, I think
>> I like the idea: Test methods are the units that succeed or fail,
>> so if test
>> method TestClass.testFoo() fails, I would investigate
>> TestClass.testFoo-output.txt and can easily concentrate on its output
>> without any need to skip over the output from all the other
>> successful
>> methods. I think that would be worth another option ;-)
>
> "TestNG is designed to cover all categories of tests: unit,
> functional, end-to-end, integration, etc..."
>
> In TestNG, especially when one method @dependsOn another method, the
> test failures aren't units, and aren't even meant to be units.
> (This can be a huge benefit for slow integration tests, where
> earlier tests leave behind fixtures that later tests are meant to
> consume.)
>
> If all you're doing is unit tests, you may prefer to use JUnit 4. :-)
>
> Anyway, in thinking about it a little harder I realized that we
> can't even necessarily split the stdout into one file per method,
> since the tests may be run in parallel!
>
> I'm pretty sure we can do that logging thing, at least. :-p
>
> -Dan
Thanks,
Jason
----------------------------------------------------------
Jason van Zyl
Founder, Apache Maven
jason at sonatype dot com
----------------------------------------------------------
We know what we are, but know not what we may be.
-- Shakespeare
Re: Test Suites, Ant, Surefire, and JunitReport
Posted by Dan Fabulich <da...@fabulich.com>.
Benjamin Bentmann wrote:
> I see. Now, if the user can provide an option to Surefire, indicating
> that the unit tests do not have inter-class dependencies, wouldn't this
> allow to get back the old behavior? I.e. enabling Surefire to pass test
> classes individually into TestNG?
That option sounds really confusing. What would you call it? How would
you document it? You'd have to know a lot about the guts of Surefire and
TestNG for an option like that to make much sense.
Not to mention that it'd be error prone. Developer X would assess the
tests, verifying that there are no inter-class dependencies, and turn on
the flag. Later, developer Y could introduce an inter-class dependency
and go to run the tests, but find that it's not working. It'd be hard to
figure out why without knowing a lot about how Surefire is doing its job.
More generally, I don't think class-level reporting really makes sense in
TestNG as it's designed today. Right now, TestNG provides listeners with
the following notifications:
onStart
onTestStart
onTestSuccess
onTestFailure
onTestSkipped
onTestFailedButWithinSuccessPercentage
onConfigurationSuccess
onConfigurationFailure
onConfigurationSkip
onFinish
Note that there's nothing here about "onClassStart/Finish" or even
"onGroupStart/Finish".
If TestNG exposed a listener, we could easily attach to it, but I see that
as a feature of TestNG. We could certainly try to write TestNG features
in Surefire, but the TestNG code knows best whether, for example, your
classes have any inter-dependencies, when your classes start and stop, and
more generally how to construct a full test run comprised of suites,
groups, classes and tests.
Surefire shouldn't be in the business of reimplementing that logic,
especially since we expect this feature to be generally useful. It should
be a part of TestNG, if it matters at all.
> I would not have any problems with a single file per method. Indeed, I think
> I like the idea: Test methods are the units that succeed or fail, so if test
> method TestClass.testFoo() fails, I would investigate
> TestClass.testFoo-output.txt and can easily concentrate on its output
> without any need to skip over the output from all the other successful
> methods. I think that would be worth another option ;-)
"TestNG is designed to cover all categories of tests: unit, functional,
end-to-end, integration, etc..."
In TestNG, especially when one method @dependsOn another method, the test
failures aren't units, and aren't even meant to be units. (This can be a
huge benefit for slow integration tests, where earlier tests leave behind
fixtures that later tests are meant to consume.)
If all you're doing is unit tests, you may prefer to use JUnit 4. :-)
Anyway, in thinking about it a little harder I realized that we can't even
necessarily split the stdout into one file per method, since the tests may
be run in parallel!
I'm pretty sure we can do that logging thing, at least. :-p
-Dan
Re: Test Suites, Ant, Surefire, and JunitReport
Posted by Benjamin Bentmann <be...@udo.edu>.
dfabulich wrote:
>
> Typically it's used for slow tests; you can identify a subset of tests
> that you want to run and just run those. (You can even pull just one
> method from Class X, and another method from Class Y, and so on.)
>
Thanks for the explanation. To me, this seems more useful for ad-hoc
testing. During an automated build, I want all existing tests for maximum
test coverage of the artifact.
dfabulich wrote:
>
> For TestNG, we have to hand it the entire directory, because any one class
> may @dependsOn methods in other classes.
>
I see. Now, if the user can provide an option to Surefire, indicating that
the unit tests do not have inter-class dependencies, wouldn't this allow to
get back the old behavior? I.e. enabling Surefire to pass test classes
individually into TestNG?
dfabulich wrote:
>
> As for splitting them all up into separate txt files, the only way we
> could do it would be to split the tests up into one txt file per METHOD,
> which is way too many files IMO.
>
I would not have any problems with a single file per method. Indeed, I think
I like the idea: Test methods are the units that succeed or fail, so if test
method TestClass.testFoo() fails, I would investigate
TestClass.testFoo-output.txt and can easily concentrate on its output
without any need to skip over the output from all the other successful
methods. I think that would be worth another option ;-)
dfabulich wrote:
>
> It's odd that you describe this as a 2.3.1 regression... I couldn't get
> TestNG to work at all in Surefire 2.3.x, hence my work on Surefire 2.4.
>
Using Surefire 2.3.x together with TestNG 5.1 worked for me. Nevertheless,
it's good to have Surefire 2.4 now!
Regards,
Benjamin Bentmann
--
View this message in context: http://www.nabble.com/Test-Suites%2C-Ant%2C-Surefire%2C-and-JunitReport-tp15076378p15093881.html
Sent from the Surefire - Developer mailing list archive at Nabble.com.
Re: Test Suites, Ant, Surefire, and JunitReport
Posted by Dan Fabulich <da...@fabulich.com>.
Benjamin Bentmann wrote:
> For my curiosity: What would be the benefit of setting up a hand-crafted
> test suite? I am a lazy guy and prefer the dumb machine to do the nasty
> things for me so I really like the idea of just dropping a test class into
> src/test/java without bothering to additionally maintain some test suite,
> too.
Typically it's used for slow tests; you can identify a subset of tests
that you want to run and just run those. (You can even pull just one
method from Class X, and another method from Class Y, and so on.)
> My personal concerns are
> a) the console output from Surefire during test execution and
> b) the redirected test output files (*-output.txt)
Surefire's design is to hand off control to the testing framework
(JUnit/TestNG) and then report on the console when we get control back.
For JUnit, we hand off control on a "class" level, so we get control back
after every class. For TestNG, we have to hand it the entire directory,
because any one class may @dependsOn methods in other classes.
Furthermore, TestNG doesn't notify us at the start/end of every class;
indeed, it's easy to configure TestNG so that individual class methods run
out of order (so that some of Class X runs and then some of Class Y, and
then back to Class X, then Class Y finishes, etc.) and that's without
considering parallelized testing.
We could log to the console on individual test methods as they run, though
I'm sure we'd want to make it a configurable option. I've filed
SUREFIRE-437 for the console logging issue.
As for splitting them all up into separate txt files, the only way we
could do it would be to split the tests up into one txt file per METHOD,
which is way too many files IMO.
> With Surefire 2.3.1, I got a result feedback every half minute when a
> test class finished.
It's odd that you describe this as a 2.3.1 regression... I couldn't get
TestNG to work at all in Surefire 2.3.x, hence my work on Surefire 2.4.
-Dan
Re: Test Suites, Ant, Surefire, and JunitReport
Posted by Benjamin Bentmann <be...@udo.edu>.
dfabulich wrote:
>
> First, it seems like most people don't even use test suites, preferring to
> simply run every test in their src/test/java directory.
>
For my curiosity: What would be the benefit of setting up a hand-crafted
test suite? I am a lazy guy and prefer the dumb machine to do the nasty
things for me so I really like the idea of just dropping a test class into
src/test/java without bothering to additionally maintain some test suite,
too.
dfabulich wrote:
>
> What I propose is that, in order to avoid destroying information, Surefire
> should generate XML that looks like Example 7 (all-in-one-file), and not
> try to fake it to look like Example 2 (one-file-per-class).
>
At least for me, I do not care about the structure/contents of the XML
report files (never read nor parse them by myself). Having a central file
with all information sounds like a reasonable approach for me, no problems
so far.
My personal concerns are
a) the console output from Surefire during test execution and
b) the redirected test output files (*-output.txt)
I am the kind of guy who prefers washing machines with a window, so I really
dislike the situation with Surefire 2.4 and TestNG where I see *nothing* on
the console until the suite completes (which takes roughly 5 minutes in one
of our modules). It is in general no good ui design if the user doesn't get
any feedback for long running tasks. With Surefire 2.3.1, I got a result
feedback every half minute when a test class finished.
Our tests use log4j with debug level logging on the console to track the
test execution and to evaluate it in case of failures. Therefore, we have
<redirectTestOutputToFile>true</redirectTestOutputToFile>. Now image a test
suite with say n test classes, every class producing X KB output. If test
class A fails, we go inspect and A-output.txt with an acceptable size of X
KB. This becomes less pleasant when you have to inspect the all-in-one
TestSuite-output.txt which has a total size of n * X KB and n being a
growing factor as the test suite is extended.
I hope these illustrations help to understand the intended use-case.
Regards,
Benjamin Bentmann
--
View this message in context: http://www.nabble.com/Test-Suites%2C-Ant%2C-Surefire%2C-and-JunitReport-tp15076378p15084183.html
Sent from the Surefire - Developer mailing list archive at Nabble.com.