You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@maven.apache.org by "Mag Hoehme (JIRA)" <ji...@codehaus.org> on 2010/04/22 10:59:12 UTC
[jira] Created: (SUREFIRE-613) Wrong number of JUnit4 tests when
running in parallel
Wrong number of JUnit4 tests when running in parallel
-----------------------------------------------------
Key: SUREFIRE-613
URL: http://jira.codehaus.org/browse/SUREFIRE-613
Project: Maven Surefire
Issue Type: Bug
Components: Junit 4.x support
Affects Versions: 2.5
Environment: Cygwin
Maven 2.0.9
Maven Surefire plugin 2.5
JUnit 4.8.1
Reporter: Mag Hoehme
I have a test class with three test methods, out of which two fail. When running sequentially, I get the following output, which is correct in my opinion:
===
Tests run: 3, Failures: 2, Errors 0, Skipped 0, ....
Results:
Failed tests:
a(...)
b(...)
Tests run: 3, Failures: 2, Errors: 0, Skipped: 0
===
Then I switch to parallel execution:
=== pom.xml ===
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.5.</version>
<configuration>
<parallel>classes</parallel>
</configuration>
</plugin>
===============
This gives me the following output from the same test class:
===
Tests run: 2, Failures: 2, Errors, 0, Skipped: 0 ...
Results :
Failed tests:
a(...)
b(...)
Tests run: 2, Failures: 2, Errors: 0, Skipped: 0
===
The same problem occurs with "<parallel>methods</parallel>" and "<parallel>tests</parallel>". When running a bunch of tests, the reported number of tests is lower than the actual number of tests.
Am I missing something, or is this a bug?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (SUREFIRE-613) Wrong number of JUnit4 tests when
running in parallel
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/SUREFIRE-613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=218602#action_218602 ]
Kristian Rosenvold commented on SUREFIRE-613:
---------------------------------------------
The parallel junit provider requires legal junit-tests (as opposed to the classic junit4-provider). Make sure all your test methods are annotated with @Test or they will not be run.
> Wrong number of JUnit4 tests when running in parallel
> -----------------------------------------------------
>
> Key: SUREFIRE-613
> URL: http://jira.codehaus.org/browse/SUREFIRE-613
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.x support
> Affects Versions: 2.5
> Environment: Cygwin
> Maven 2.0.9
> Maven Surefire plugin 2.5
> JUnit 4.8.1
> Reporter: Mag Hoehme
>
> I have a test class with three test methods, out of which two fail. When running sequentially, I get the following output, which is correct in my opinion:
> ===
> Tests run: 3, Failures: 2, Errors 0, Skipped 0, ....
> Results:
> Failed tests:
> a(...)
> b(...)
> Tests run: 3, Failures: 2, Errors: 0, Skipped: 0
> ===
> Then I switch to parallel execution:
> === pom.xml ===
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-surefire-plugin</artifactId>
> <version>2.5.</version>
> <configuration>
> <parallel>classes</parallel>
> </configuration>
> </plugin>
> ===============
> This gives me the following output from the same test class:
> ===
> Tests run: 2, Failures: 2, Errors, 0, Skipped: 0 ...
> Results :
> Failed tests:
> a(...)
> b(...)
> Tests run: 2, Failures: 2, Errors: 0, Skipped: 0
> ===
> The same problem occurs with "<parallel>methods</parallel>" and "<parallel>tests</parallel>". When running a bunch of tests, the reported number of tests is lower than the actual number of tests.
> Am I missing something, or is this a bug?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (SUREFIRE-613) Wrong number of JUnit4 tests when
running in parallel
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/SUREFIRE-613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kristian Rosenvold updated SUREFIRE-613:
----------------------------------------
Component/s: (was: Junit 4.x support)
Junit 4.7+ (parallel) support
> Wrong number of JUnit4 tests when running in parallel
> -----------------------------------------------------
>
> Key: SUREFIRE-613
> URL: http://jira.codehaus.org/browse/SUREFIRE-613
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.7+ (parallel) support
> Affects Versions: 2.5
> Environment: Cygwin
> Maven 2.0.9
> Maven Surefire plugin 2.5
> JUnit 4.8.1
> Reporter: Mag Hoehme
> Assignee: Kristian Rosenvold
> Fix For: 2.6
>
> Attachments: junit4-test.tar.gz
>
>
> I have a test class with three test methods, out of which two fail. When running sequentially, I get the following output, which is correct in my opinion:
> ===
> Tests run: 3, Failures: 2, Errors 0, Skipped 0, ....
> Results:
> Failed tests:
> a(...)
> b(...)
> Tests run: 3, Failures: 2, Errors: 0, Skipped: 0
> ===
> Then I switch to parallel execution:
> === pom.xml ===
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-surefire-plugin</artifactId>
> <version>2.5.</version>
> <configuration>
> <parallel>classes</parallel>
> </configuration>
> </plugin>
> ===============
> This gives me the following output from the same test class:
> ===
> Tests run: 2, Failures: 2, Errors, 0, Skipped: 0 ...
> Results :
> Failed tests:
> a(...)
> b(...)
> Tests run: 2, Failures: 2, Errors: 0, Skipped: 0
> ===
> The same problem occurs with "<parallel>methods</parallel>" and "<parallel>tests</parallel>". When running a bunch of tests, the reported number of tests is lower than the actual number of tests.
> Am I missing something, or is this a bug?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (SUREFIRE-613) Wrong number of JUnit4 tests when
running in parallel
Posted by "Mag Hoehme (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/SUREFIRE-613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=218735#action_218735 ]
Mag Hoehme commented on SUREFIRE-613:
-------------------------------------
Hi Kristian, thank you for your quick response.
The problem seems to be with the surefire plugin 2.5 and only arises when the tests run in parallel. The number of tests is fine when parallel execution is switched off. The tests are correctly annotated. I attached a small demo project that shows the problem.
> Wrong number of JUnit4 tests when running in parallel
> -----------------------------------------------------
>
> Key: SUREFIRE-613
> URL: http://jira.codehaus.org/browse/SUREFIRE-613
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.x support
> Affects Versions: 2.5
> Environment: Cygwin
> Maven 2.0.9
> Maven Surefire plugin 2.5
> JUnit 4.8.1
> Reporter: Mag Hoehme
> Attachments: junit4-test.tar.gz
>
>
> I have a test class with three test methods, out of which two fail. When running sequentially, I get the following output, which is correct in my opinion:
> ===
> Tests run: 3, Failures: 2, Errors 0, Skipped 0, ....
> Results:
> Failed tests:
> a(...)
> b(...)
> Tests run: 3, Failures: 2, Errors: 0, Skipped: 0
> ===
> Then I switch to parallel execution:
> === pom.xml ===
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-surefire-plugin</artifactId>
> <version>2.5.</version>
> <configuration>
> <parallel>classes</parallel>
> </configuration>
> </plugin>
> ===============
> This gives me the following output from the same test class:
> ===
> Tests run: 2, Failures: 2, Errors, 0, Skipped: 0 ...
> Results :
> Failed tests:
> a(...)
> b(...)
> Tests run: 2, Failures: 2, Errors: 0, Skipped: 0
> ===
> The same problem occurs with "<parallel>methods</parallel>" and "<parallel>tests</parallel>". When running a bunch of tests, the reported number of tests is lower than the actual number of tests.
> Am I missing something, or is this a bug?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (SUREFIRE-613) Wrong number of JUnit4 tests when
running in parallel
Posted by "Paul Gier (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/SUREFIRE-613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Gier updated SUREFIRE-613:
-------------------------------
Fix Version/s: (was: 2.5.1)
2.6
> Wrong number of JUnit4 tests when running in parallel
> -----------------------------------------------------
>
> Key: SUREFIRE-613
> URL: http://jira.codehaus.org/browse/SUREFIRE-613
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.x support
> Affects Versions: 2.5
> Environment: Cygwin
> Maven 2.0.9
> Maven Surefire plugin 2.5
> JUnit 4.8.1
> Reporter: Mag Hoehme
> Assignee: Kristian Rosenvold
> Fix For: 2.6
>
> Attachments: junit4-test.tar.gz
>
>
> I have a test class with three test methods, out of which two fail. When running sequentially, I get the following output, which is correct in my opinion:
> ===
> Tests run: 3, Failures: 2, Errors 0, Skipped 0, ....
> Results:
> Failed tests:
> a(...)
> b(...)
> Tests run: 3, Failures: 2, Errors: 0, Skipped: 0
> ===
> Then I switch to parallel execution:
> === pom.xml ===
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-surefire-plugin</artifactId>
> <version>2.5.</version>
> <configuration>
> <parallel>classes</parallel>
> </configuration>
> </plugin>
> ===============
> This gives me the following output from the same test class:
> ===
> Tests run: 2, Failures: 2, Errors, 0, Skipped: 0 ...
> Results :
> Failed tests:
> a(...)
> b(...)
> Tests run: 2, Failures: 2, Errors: 0, Skipped: 0
> ===
> The same problem occurs with "<parallel>methods</parallel>" and "<parallel>tests</parallel>". When running a bunch of tests, the reported number of tests is lower than the actual number of tests.
> Am I missing something, or is this a bug?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Closed: (SUREFIRE-613) Wrong number of JUnit4 tests when
running in parallel
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/SUREFIRE-613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kristian Rosenvold closed SUREFIRE-613.
---------------------------------------
Resolution: Fixed
Fix Version/s: 2.6
Fixed in r940602: Junit 4.7 provider does not count tests correctly in all circumstances.
Updated DemultiplexingRunListener to non-ketchup-bottle variety that logs output smoothly per completed test.
Added IT
> Wrong number of JUnit4 tests when running in parallel
> -----------------------------------------------------
>
> Key: SUREFIRE-613
> URL: http://jira.codehaus.org/browse/SUREFIRE-613
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.x support
> Affects Versions: 2.5
> Environment: Cygwin
> Maven 2.0.9
> Maven Surefire plugin 2.5
> JUnit 4.8.1
> Reporter: Mag Hoehme
> Fix For: 2.6
>
> Attachments: junit4-test.tar.gz
>
>
> I have a test class with three test methods, out of which two fail. When running sequentially, I get the following output, which is correct in my opinion:
> ===
> Tests run: 3, Failures: 2, Errors 0, Skipped 0, ....
> Results:
> Failed tests:
> a(...)
> b(...)
> Tests run: 3, Failures: 2, Errors: 0, Skipped: 0
> ===
> Then I switch to parallel execution:
> === pom.xml ===
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-surefire-plugin</artifactId>
> <version>2.5.</version>
> <configuration>
> <parallel>classes</parallel>
> </configuration>
> </plugin>
> ===============
> This gives me the following output from the same test class:
> ===
> Tests run: 2, Failures: 2, Errors, 0, Skipped: 0 ...
> Results :
> Failed tests:
> a(...)
> b(...)
> Tests run: 2, Failures: 2, Errors: 0, Skipped: 0
> ===
> The same problem occurs with "<parallel>methods</parallel>" and "<parallel>tests</parallel>". When running a bunch of tests, the reported number of tests is lower than the actual number of tests.
> Am I missing something, or is this a bug?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (SUREFIRE-613) Wrong number of JUnit4 tests when
running in parallel
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/SUREFIRE-613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kristian Rosenvold updated SUREFIRE-613:
----------------------------------------
Fix Version/s: (was: 2.6)
2.5.1
> Wrong number of JUnit4 tests when running in parallel
> -----------------------------------------------------
>
> Key: SUREFIRE-613
> URL: http://jira.codehaus.org/browse/SUREFIRE-613
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.x support
> Affects Versions: 2.5
> Environment: Cygwin
> Maven 2.0.9
> Maven Surefire plugin 2.5
> JUnit 4.8.1
> Reporter: Mag Hoehme
> Assignee: Kristian Rosenvold
> Fix For: 2.5.1
>
> Attachments: junit4-test.tar.gz
>
>
> I have a test class with three test methods, out of which two fail. When running sequentially, I get the following output, which is correct in my opinion:
> ===
> Tests run: 3, Failures: 2, Errors 0, Skipped 0, ....
> Results:
> Failed tests:
> a(...)
> b(...)
> Tests run: 3, Failures: 2, Errors: 0, Skipped: 0
> ===
> Then I switch to parallel execution:
> === pom.xml ===
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-surefire-plugin</artifactId>
> <version>2.5.</version>
> <configuration>
> <parallel>classes</parallel>
> </configuration>
> </plugin>
> ===============
> This gives me the following output from the same test class:
> ===
> Tests run: 2, Failures: 2, Errors, 0, Skipped: 0 ...
> Results :
> Failed tests:
> a(...)
> b(...)
> Tests run: 2, Failures: 2, Errors: 0, Skipped: 0
> ===
> The same problem occurs with "<parallel>methods</parallel>" and "<parallel>tests</parallel>". When running a bunch of tests, the reported number of tests is lower than the actual number of tests.
> Am I missing something, or is this a bug?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (SUREFIRE-613) Wrong number of JUnit4 tests when
running in parallel
Posted by "Mag Hoehme (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/SUREFIRE-613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mag Hoehme updated SUREFIRE-613:
--------------------------------
Attachment: junit4-test.tar.gz
Demo project illustrating the wrong number of executed tests for parallel execution.
> Wrong number of JUnit4 tests when running in parallel
> -----------------------------------------------------
>
> Key: SUREFIRE-613
> URL: http://jira.codehaus.org/browse/SUREFIRE-613
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.x support
> Affects Versions: 2.5
> Environment: Cygwin
> Maven 2.0.9
> Maven Surefire plugin 2.5
> JUnit 4.8.1
> Reporter: Mag Hoehme
> Attachments: junit4-test.tar.gz
>
>
> I have a test class with three test methods, out of which two fail. When running sequentially, I get the following output, which is correct in my opinion:
> ===
> Tests run: 3, Failures: 2, Errors 0, Skipped 0, ....
> Results:
> Failed tests:
> a(...)
> b(...)
> Tests run: 3, Failures: 2, Errors: 0, Skipped: 0
> ===
> Then I switch to parallel execution:
> === pom.xml ===
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-surefire-plugin</artifactId>
> <version>2.5.</version>
> <configuration>
> <parallel>classes</parallel>
> </configuration>
> </plugin>
> ===============
> This gives me the following output from the same test class:
> ===
> Tests run: 2, Failures: 2, Errors, 0, Skipped: 0 ...
> Results :
> Failed tests:
> a(...)
> b(...)
> Tests run: 2, Failures: 2, Errors: 0, Skipped: 0
> ===
> The same problem occurs with "<parallel>methods</parallel>" and "<parallel>tests</parallel>". When running a bunch of tests, the reported number of tests is lower than the actual number of tests.
> Am I missing something, or is this a bug?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (SUREFIRE-613) Wrong number of JUnit4 tests when
running in parallel
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/SUREFIRE-613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kristian Rosenvold updated SUREFIRE-613:
----------------------------------------
Assignee: Kristian Rosenvold
> Wrong number of JUnit4 tests when running in parallel
> -----------------------------------------------------
>
> Key: SUREFIRE-613
> URL: http://jira.codehaus.org/browse/SUREFIRE-613
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.x support
> Affects Versions: 2.5
> Environment: Cygwin
> Maven 2.0.9
> Maven Surefire plugin 2.5
> JUnit 4.8.1
> Reporter: Mag Hoehme
> Assignee: Kristian Rosenvold
> Fix For: 2.6
>
> Attachments: junit4-test.tar.gz
>
>
> I have a test class with three test methods, out of which two fail. When running sequentially, I get the following output, which is correct in my opinion:
> ===
> Tests run: 3, Failures: 2, Errors 0, Skipped 0, ....
> Results:
> Failed tests:
> a(...)
> b(...)
> Tests run: 3, Failures: 2, Errors: 0, Skipped: 0
> ===
> Then I switch to parallel execution:
> === pom.xml ===
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-surefire-plugin</artifactId>
> <version>2.5.</version>
> <configuration>
> <parallel>classes</parallel>
> </configuration>
> </plugin>
> ===============
> This gives me the following output from the same test class:
> ===
> Tests run: 2, Failures: 2, Errors, 0, Skipped: 0 ...
> Results :
> Failed tests:
> a(...)
> b(...)
> Tests run: 2, Failures: 2, Errors: 0, Skipped: 0
> ===
> The same problem occurs with "<parallel>methods</parallel>" and "<parallel>tests</parallel>". When running a bunch of tests, the reported number of tests is lower than the actual number of tests.
> Am I missing something, or is this a bug?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira