[jira] Updated: (SUREFIRE-457) TestNG should log on the console before/after every test class

["Dan Fabulich (JIRA)" <ji...@codehaus.org>]

     [ http://jira.codehaus.org/browse/SUREFIRE-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dan Fabulich updated SUREFIRE-457:
----------------------------------

    Attachment: testNGListenerTest.zip

Jason Chaffee claims that you can get the right thing with onStart/onFinish notifications.  I've attached a sample project that seems to show that this is false.

http://markmail.org/message/in2iosogjpjev5yk
http://markmail.org/message/yhbzjbhxmcydvqna

I've attached a sample project that simply attaches an ITestListener that logs to system.out whenever it gets called, specifically during onStart and onFinish.  It's got two test classes: Foo and Bar, each of which have one method "foo()" and "bar()" respectively.  So you'd expect:

onStart [Foo]
onTestStart [foo()]
onTestFinish [foo()]
onFinish [Foo]
onStart [Bar]
onTestStart [bar()]
onTestFinish [bar()]
onFinish [Bar]

But here's what you get instead:

onStart
onTestStart [foo()]
onTestFinish [foo()]
onTestStart [bar()]
onTestFinish [bar()]
onFinish

... not too useful, is it?  In fact, onStart is called at the start/end of the "test" which is TestNG's very silly term for a collection of tests that isn't a "group" or a "suite".


> TestNG should log on the console before/after every test class
> --------------------------------------------------------------
>
>                 Key: SUREFIRE-457
>                 URL: http://jira.codehaus.org/browse/SUREFIRE-457
>             Project: Maven Surefire
>          Issue Type: Wish
>          Components: TestNG support
>    Affects Versions: 2.4, 2.4.1, 2.4.2
>            Reporter: Dan Fabulich
>            Priority: Minor
>             Fix For: Future
>
>         Attachments: testNGListenerTest.zip
>
>
> In Surefire 2.3.x, when running the tests in "directory mode" (i.e. without a suite.xml file) we used to run each TestNG class as a separate suite.  As a result, we logged to the console before/after every test class.
> This behavior totally broke tests that had rich interdependencies (which is a big part of the point of TestNG), so in Surefire 2.4 we handed more control over to TestNG.  As a result, we let TestNG run the entire directory at once, and we aren't notified when classes start/end.  Since we aren't notified, we can't log to the console before/after every test class.  But this looks like a regression to those who were used to the 2.3.x behavior.
> (This is trickier than it looks, because TestNG can/will run test methods entirely out of order, running part of class X, and then part of class Y, then back to class X, then a bit of class Z, etc.  And that's not even considering parallelized testing.)
> I argue that to fix this "right" we'd need TestNG to notify us when classes start/end; we should pressure the TestNG team to provide this functionality.  Benjamin has argued that there should be an option to run each test in its own suite; I think that option is confusing and error-prone.
> http://www.nabble.com/Re%3A-Test-Suites%2C-Ant%2C-Surefire%2C-and-JunitReport-p15094586.html
> If you think you want a one-class-per-suite option, I offer this remark: Many TestNG tests aren't safe to run in one-class-per-suite mode.  If your tests are safe to run in that mode, then they're also safe to run in JUnit 4.  (In fact, if your tests are that simple, you probably aren't using any of TestNG's unique features, and you should prefer to use JUnit 4.)  So, as a workaround, don't use TestNG; use JUnit 4 instead.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira