You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@maven.apache.org by "karthik kandasamy (JIRA)" <ji...@codehaus.org> on 2012/02/22 14:36:02 UTC
[jira] (SUREFIRE-841) Incorrect Test Run Count
karthik kandasamy created SUREFIRE-841:
------------------------------------------
Summary: Incorrect Test Run Count
Key: SUREFIRE-841
URL: https://jira.codehaus.org/browse/SUREFIRE-841
Project: Maven Surefire
Issue Type: Bug
Components: Maven Surefire Plugin
Reporter: karthik kandasamy
When a simple Junit test with errors in the @Before() and @After() method are run directly with java or ant's junit task, it reports correctly that the Tests Run = 1 and Errors = 2.
But when the same is run through maven surefire plugin, it reports it as Tests Run = 2 and Errors = 2.
Its the same test in which 2 errors are encountered, so the Tests Run should be 1.
I traced the issue to the org.apache.maven.surefire.report.TestSetRunListener Class - testError() method, where the completed count is also incremented along with the error count irrespective of whether its in the same test the error is encountered.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (SUREFIRE-841) Incorrect Test Run Count
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/SUREFIRE-841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=308589#comment-308589 ]
Kristian Rosenvold commented on SUREFIRE-841:
---------------------------------------------
The ant runner has basically been un-maintained for several years, so I'm not really sure they're anything to go by.
As for "who's who" in the testing business, JUnit usually defines the standard and the others follow suite. AFIK JUnit is still almost 10x the users of TestNG. So I really have no problem with us folowing JUnit4 only ;) Although it would be real easy to adapt to the underlying framework, I'm afraid of the potential for creating confusion....
> Incorrect Test Run Count
> ------------------------
>
> Key: SUREFIRE-841
> URL: https://jira.codehaus.org/browse/SUREFIRE-841
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
> Reporter: karthik kandasamy
> Fix For: 2.13
>
>
> When a simple Junit test with errors in the @Before() and @After() method are run directly with java or ant's junit task, it reports correctly that the Tests Run = 1 and Errors = 2.
> But when the same is run through maven surefire plugin, it reports it as Tests Run = 2 and Errors = 2.
> Its the same test in which 2 errors are encountered, so the Tests Run should be 1.
> I traced the issue to the org.apache.maven.surefire.report.TestSetRunListener Class - testError() method, where the completed count is also incremented along with the error count irrespective of whether its in the same test the error is encountered.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (SUREFIRE-841) Incorrect Test Run Count
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/SUREFIRE-841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kristian Rosenvold updated SUREFIRE-841:
----------------------------------------
Fix Version/s: (was: 2.13)
> Incorrect Test Run Count
> ------------------------
>
> Key: SUREFIRE-841
> URL: https://jira.codehaus.org/browse/SUREFIRE-841
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
> Reporter: karthik kandasamy
>
> When a simple Junit test with errors in the @Before() and @After() method are run directly with java or ant's junit task, it reports correctly that the Tests Run = 1 and Errors = 2.
> But when the same is run through maven surefire plugin, it reports it as Tests Run = 2 and Errors = 2.
> Its the same test in which 2 errors are encountered, so the Tests Run should be 1.
> I traced the issue to the org.apache.maven.surefire.report.TestSetRunListener Class - testError() method, where the completed count is also incremented along with the error count irrespective of whether its in the same test the error is encountered.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (SUREFIRE-841) Incorrect Test Run Count
Posted by "Baptiste MATHUS (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/SUREFIRE-841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=308590#comment-308590 ]
Baptiste MATHUS commented on SUREFIRE-841:
------------------------------------------
I agree about being cautious about creating confusion. But still, surefire didn't create that confusion, from what I see JUnit did by changing its behaviour, isn't it?
I think only a small warning in the doc would prevent many users from being surprised. And if some get surprised, well that's a JUnit-side "issue" in the first place...
> Incorrect Test Run Count
> ------------------------
>
> Key: SUREFIRE-841
> URL: https://jira.codehaus.org/browse/SUREFIRE-841
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
> Reporter: karthik kandasamy
> Fix For: 2.13
>
>
> When a simple Junit test with errors in the @Before() and @After() method are run directly with java or ant's junit task, it reports correctly that the Tests Run = 1 and Errors = 2.
> But when the same is run through maven surefire plugin, it reports it as Tests Run = 2 and Errors = 2.
> Its the same test in which 2 errors are encountered, so the Tests Run should be 1.
> I traced the issue to the org.apache.maven.surefire.report.TestSetRunListener Class - testError() method, where the completed count is also incremented along with the error count irrespective of whether its in the same test the error is encountered.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (SUREFIRE-841) Incorrect Test Run Count
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/SUREFIRE-841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=308587#comment-308587 ]
Kristian Rosenvold edited comment on SUREFIRE-841 at 9/15/12 12:05 PM:
-----------------------------------------------------------------------
The behaviour you expect is compliant with a JUnit3 runner (and TestNG, which up until just recently was only JUnit3 compliant). Surefire's behaviour is consistent with JUnit4. (I am not sure how TestNG behaves when running JUnit4 tests)
Ant is a JUnit3 based test-runner.
I'm still trying to figure out what to do with this issue:
A) Close it as wontfix.
B) Make surefire report JUnit3/TestNG style when using these libraries and JUnit4 style when using junit4.
The problem with B is that surefire will appear to behave inconsistently when users upgrade from 3.x to 4.x.
For those wishing to experiment with this, there is an IT project at:
https://svn.apache.org/repos/asf/maven/surefire/trunk/surefire-integration-tests/src/test/resources/failure-result-counting
This verifies all the different modes of reporting.
was (Author: krosenvold):
The behaviour you expect is compliant with a JUnit3 runner (and TestNG, which up until just recently was only JUnit3 compliant). Surefire's behaviour is consistent with JUnit4.
Ant is a JUnit3 based test-runner.
I'm still trying to figure out what to do with this issue:
A) Close it as wontfix.
B) Make surefire report JUnit3/TestNG style when using these libraries and JUnit4 style when using junit4.
The problem with B is that surefire will appear to behave inconsistently when users upgrade from 3.x to 4.x.
For those wishing to experiment with this, there is an IT project at:
https://svn.apache.org/repos/asf/maven/surefire/trunk/surefire-integration-tests/src/test/resources/failure-result-counting
This verifies all the different modes of reporting.
> Incorrect Test Run Count
> ------------------------
>
> Key: SUREFIRE-841
> URL: https://jira.codehaus.org/browse/SUREFIRE-841
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
> Reporter: karthik kandasamy
> Fix For: 2.13
>
>
> When a simple Junit test with errors in the @Before() and @After() method are run directly with java or ant's junit task, it reports correctly that the Tests Run = 1 and Errors = 2.
> But when the same is run through maven surefire plugin, it reports it as Tests Run = 2 and Errors = 2.
> Its the same test in which 2 errors are encountered, so the Tests Run should be 1.
> I traced the issue to the org.apache.maven.surefire.report.TestSetRunListener Class - testError() method, where the completed count is also incremented along with the error count irrespective of whether its in the same test the error is encountered.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (SUREFIRE-841) Incorrect Test Run Count
Posted by "Baptiste MATHUS (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/SUREFIRE-841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=308588#comment-308588 ]
Baptiste MATHUS commented on SUREFIRE-841:
------------------------------------------
I would go with being consistent with the underlying test "engine" in use. So I guess it's B).
Ant Runner should be fixed for the cases when it runs JUnit 4 or JUnit 3.
By the way, maybe the problem is "larger" than just Surefire: maybe TestNg team and Junit one should try & align their way of thinking if possible. If already done, then we're back to B.
> Incorrect Test Run Count
> ------------------------
>
> Key: SUREFIRE-841
> URL: https://jira.codehaus.org/browse/SUREFIRE-841
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
> Reporter: karthik kandasamy
> Fix For: 2.13
>
>
> When a simple Junit test with errors in the @Before() and @After() method are run directly with java or ant's junit task, it reports correctly that the Tests Run = 1 and Errors = 2.
> But when the same is run through maven surefire plugin, it reports it as Tests Run = 2 and Errors = 2.
> Its the same test in which 2 errors are encountered, so the Tests Run should be 1.
> I traced the issue to the org.apache.maven.surefire.report.TestSetRunListener Class - testError() method, where the completed count is also incremented along with the error count irrespective of whether its in the same test the error is encountered.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (SUREFIRE-841) Incorrect Test Run Count
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/SUREFIRE-841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kristian Rosenvold updated SUREFIRE-841:
----------------------------------------
Fix Version/s: 2.13
> Incorrect Test Run Count
> ------------------------
>
> Key: SUREFIRE-841
> URL: https://jira.codehaus.org/browse/SUREFIRE-841
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
> Reporter: karthik kandasamy
> Fix For: 2.13
>
>
> When a simple Junit test with errors in the @Before() and @After() method are run directly with java or ant's junit task, it reports correctly that the Tests Run = 1 and Errors = 2.
> But when the same is run through maven surefire plugin, it reports it as Tests Run = 2 and Errors = 2.
> Its the same test in which 2 errors are encountered, so the Tests Run should be 1.
> I traced the issue to the org.apache.maven.surefire.report.TestSetRunListener Class - testError() method, where the completed count is also incremented along with the error count irrespective of whether its in the same test the error is encountered.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (SUREFIRE-841) Incorrect Test Run Count
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/SUREFIRE-841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=308587#comment-308587 ]
Kristian Rosenvold commented on SUREFIRE-841:
---------------------------------------------
The behaviour you expect is compliant with a JUnit3 runner (and TestNG, which up until just recently was only JUnit3 compliant). Surefire's behaviour is consistent with JUnit4.
Ant is a JUnit3 based test-runner.
I'm still trying to figure out what to do with this issue:
A) Close it as wontfix.
B) Make surefire report JUnit3/TestNG style when using these libraries and JUnit4 style when using junit4.
The problem with B is that surefire will appear to behave inconsistently when users upgrade from 3.x to 4.x.
For those wishing to experiment with this, there is an IT project at:
https://svn.apache.org/repos/asf/maven/surefire/trunk/surefire-integration-tests/src/test/resources/failure-result-counting
This verifies all the different modes of reporting.
> Incorrect Test Run Count
> ------------------------
>
> Key: SUREFIRE-841
> URL: https://jira.codehaus.org/browse/SUREFIRE-841
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
> Reporter: karthik kandasamy
> Fix For: 2.13
>
>
> When a simple Junit test with errors in the @Before() and @After() method are run directly with java or ant's junit task, it reports correctly that the Tests Run = 1 and Errors = 2.
> But when the same is run through maven surefire plugin, it reports it as Tests Run = 2 and Errors = 2.
> Its the same test in which 2 errors are encountered, so the Tests Run should be 1.
> I traced the issue to the org.apache.maven.surefire.report.TestSetRunListener Class - testError() method, where the completed count is also incremented along with the error count irrespective of whether its in the same test the error is encountered.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (SUREFIRE-841) Incorrect Test Run Count
Posted by "Mikhail Mazursky (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/SUREFIRE-841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=307511#comment-307511 ]
Mikhail Mazursky commented on SUREFIRE-841:
-------------------------------------------
Also encountered this with surefire 2.12.3 and TestNG 6.7. Failure in @BeforeClass increments "Tests run".
{code}
public class RealTest {
@BeforeClass
public void beforeClass() {
assertThat(false).isTrue();
}
@Test
public void test(){}
}
{code}
With failure it's output is
{noformat}
Tests run: 2, Failures: 1, Errors: 0, Skipped: 1
{noformat}
Without:
{noformat}
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
{noformat}
> Incorrect Test Run Count
> ------------------------
>
> Key: SUREFIRE-841
> URL: https://jira.codehaus.org/browse/SUREFIRE-841
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
> Reporter: karthik kandasamy
>
> When a simple Junit test with errors in the @Before() and @After() method are run directly with java or ant's junit task, it reports correctly that the Tests Run = 1 and Errors = 2.
> But when the same is run through maven surefire plugin, it reports it as Tests Run = 2 and Errors = 2.
> Its the same test in which 2 errors are encountered, so the Tests Run should be 1.
> I traced the issue to the org.apache.maven.surefire.report.TestSetRunListener Class - testError() method, where the completed count is also incremented along with the error count irrespective of whether its in the same test the error is encountered.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira