You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@maven.apache.org by "nkeywal (JIRA)" <ji...@codehaus.org> on 2011/12/27 12:29:02 UTC
[jira] (SUREFIRE-813) Randomly wrong tests count and empty summary
files with JUnitProvider47 when forking the tests.
nkeywal created SUREFIRE-813:
--------------------------------
Summary: Randomly wrong tests count and empty summary files with JUnitProvider47 when forking the tests.
Key: SUREFIRE-813
URL: https://jira.codehaus.org/browse/SUREFIRE-813
Project: Maven Surefire
Issue Type: Bug
Components: Junit 4.7+ (parallel) support
Affects Versions: 2.11, 2.10, 2.12
Environment: test case checked on windows 7 64 bits; issues on Linux.
jdk1.6.0_24
Reporter: nkeywal
Attachments: pom.xml, Test6.java
When running the test suite, with forkMode set and junit provider 47, we can have some test results missing. The test class is executed, but:
- the summary file is empty (exists, but as a zero length file)
- the global test count is not increased.
It's random, and does not occurs all the time. The occurrence seems increased when the machine is loaded, but it's more an impression. In our case, with ~250 test classes, it occurs on average once per run (so it's a real problem). It seems to occur more if we activate 'perthread' parallelization.
I added some log lines, and it seems that ForkClient does not receive all the test events when the problem occurs. It receives:
- ForkingRunListener.BOOTERCODE_TESTSET_STARTING
- ForkingRunListener.BOOTERCODE_TEST_STARTING
But not the final:
- ForkingRunListener.BOOTERCODE_TEST_SUCCEEDED
- ForkingRunListener.BOOTERCODE_TESTSET_COMPLETED
The original issue is on Linux, java 1.6, with a lot of tests running. I wrote a test case, hopefully reproducing the real issue with surefire 2.10 and 2.11. It's still random, but fails 50% of the time on my machine (laptop 2 cores, Windows 7, 64 bits, jdk1.6.0_24).
In this example, we create a thread writing 1000 lines of 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" in stdout in the tear down method.
That's an example of success with surefire 2.11. You see that exactly two lines of 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' has been written to the console. On some other successes it can be zero lines or more.
{noformat}
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Concurrency config is parallel='classes', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
Running Test6
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec
658:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
659:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.724s
[INFO] Finished at: Tue Dec 27 11:46:03 CET 2011
[INFO] Final Memory: 10M/109M
[INFO] ------------------------------------------------------------------------
{noformat}
Error case, with surefire 2.11. In this one there is no 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' in the output, but you can have errors with multiple such lines.
You see that the number of tests run is zero.
{noformat}
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Concurrency config is parallel='classes', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
Running Test6
Results :
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.820s
[INFO] Finished at: Tue Dec 27 11:48:02 CET 2011
[INFO] Final Memory: 10M/109M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.11:test (default-test) on project surefireparametizedbug: No tests were executed! (Set -DfailIfNoTests=false to ignore this error.) -> [Help 1]
[ERROR]
{noformat}
With junit4 instead of junit47 it seems it never fails, but you can still have some AAA lines in the final output:
{noformat}
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running Test6
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.07 sec
18:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
19:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
20:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
21:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
22:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.667s
[INFO] Finished at: Tue Dec 27 11:54:16 CET 2011
[INFO] Final Memory: 10M/114M
[INFO] ------------------------------------------------------------------------
{noformat}
It can fail as well (but less often it seems) if redirectTestOutputToFile is set to false.
It fails as well with forkMode=once (on surefire 2.11 at least).
parallel is set to "classes"; but if fails as well if it's not set (still on surefire 2.11 at least).
Note that I write 1000 lines in the test case, it can fail with much less as well. It's just a simplification of the initially complex issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (SUREFIRE-813) Randomly wrong tests count and empty summary
files with JUnitProvider47 when forking the tests.
Posted by "nkeywal (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/SUREFIRE-813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=286915#comment-286915 ]
nkeywal commented on SUREFIRE-813:
----------------------------------
Yes, it's consistent:we have many lines (~100) logged during the tests tear down, and it's not always in a separate thread.
> Randomly wrong tests count and empty summary files with JUnitProvider47 when forking the tests.
> -----------------------------------------------------------------------------------------------
>
> Key: SUREFIRE-813
> URL: https://jira.codehaus.org/browse/SUREFIRE-813
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.7+ (parallel) support
> Affects Versions: 2.10, 2.11, 2.12
> Environment: test case checked on windows 7 64 bits; issues on Linux.
> jdk1.6.0_24
> Reporter: nkeywal
> Assignee: Kristian Rosenvold
> Attachments: pom.xml, Test6.java
>
>
> When running the test suite, with forkMode set and junit provider 47, we can have some test results missing. The test class is executed, but:
> - the summary file is empty (exists, but as a zero length file)
> - the global test count is not increased.
> It's random, and does not occurs all the time. The occurrence seems increased when the machine is loaded, but it's more an impression. In our case, with ~250 test classes, it occurs on average once per run (so it's a real problem). It seems to occur more if we activate 'perthread' parallelization.
>
> I added some log lines, and it seems that ForkClient does not receive all the test events when the problem occurs. It receives:
> - ForkingRunListener.BOOTERCODE_TESTSET_STARTING
> - ForkingRunListener.BOOTERCODE_TEST_STARTING
>
> But not the final:
> - ForkingRunListener.BOOTERCODE_TEST_SUCCEEDED
> - ForkingRunListener.BOOTERCODE_TESTSET_COMPLETED
>
> The original issue is on Linux, java 1.6, with a lot of tests running. I wrote a test case, hopefully reproducing the real issue with surefire 2.10 and 2.11. It's still random, but fails 50% of the time on my machine (laptop 2 cores, Windows 7, 64 bits, jdk1.6.0_24).
> In this example, we create a thread writing 1000 lines of 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" in stdout in the tear down method.
> That's an example of success with surefire 2.11. You see that exactly two lines of 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' has been written to the console. On some other successes it can be zero lines or more.
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Concurrency config is parallel='classes', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
> Running Test6
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec
> 658:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 659:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Results :
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.724s
> [INFO] Finished at: Tue Dec 27 11:46:03 CET 2011
> [INFO] Final Memory: 10M/109M
> [INFO] ------------------------------------------------------------------------
> {noformat}
> Error case, with surefire 2.11. In this one there is no 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' in the output, but you can have errors with multiple such lines.
> You see that the number of tests run is zero.
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Concurrency config is parallel='classes', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
> Running Test6
> Results :
> Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.820s
> [INFO] Finished at: Tue Dec 27 11:48:02 CET 2011
> [INFO] Final Memory: 10M/109M
> [INFO] ------------------------------------------------------------------------
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.11:test (default-test) on project surefireparametizedbug: No tests were executed! (Set -DfailIfNoTests=false to ignore this error.) -> [Help 1]
> [ERROR]
> {noformat}
> With junit4 instead of junit47 it seems it never fails, but you can still have some AAA lines in the final output:
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running Test6
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.07 sec
> 18:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 19:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 20:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 21:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 22:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Results :
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.667s
> [INFO] Finished at: Tue Dec 27 11:54:16 CET 2011
> [INFO] Final Memory: 10M/114M
> [INFO] ------------------------------------------------------------------------
> {noformat}
> It can fail as well (but less often it seems) if redirectTestOutputToFile is set to false.
> It fails as well with forkMode=once (on surefire 2.11 at least).
> parallel is set to "classes"; but if fails as well if it's not set (still on surefire 2.11 at least).
> Note that I write 1000 lines in the test case, it can fail with much less as well. It's just a simplification of the initially complex issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (SUREFIRE-813) Randomly wrong tests count and empty summary
files with JUnitProvider47 when forking the tests.
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/SUREFIRE-813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on SUREFIRE-813 started by Kristian Rosenvold.
> Randomly wrong tests count and empty summary files with JUnitProvider47 when forking the tests.
> -----------------------------------------------------------------------------------------------
>
> Key: SUREFIRE-813
> URL: https://jira.codehaus.org/browse/SUREFIRE-813
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.7+ (parallel) support
> Affects Versions: 2.10, 2.11, 2.12
> Environment: test case checked on windows 7 64 bits; issues on Linux.
> jdk1.6.0_24
> Reporter: nkeywal
> Assignee: Kristian Rosenvold
> Attachments: pom.xml, Test6.java
>
>
> When running the test suite, with forkMode set and junit provider 47, we can have some test results missing. The test class is executed, but:
> - the summary file is empty (exists, but as a zero length file)
> - the global test count is not increased.
> It's random, and does not occurs all the time. The occurrence seems increased when the machine is loaded, but it's more an impression. In our case, with ~250 test classes, it occurs on average once per run (so it's a real problem). It seems to occur more if we activate 'perthread' parallelization.
>
> I added some log lines, and it seems that ForkClient does not receive all the test events when the problem occurs. It receives:
> - ForkingRunListener.BOOTERCODE_TESTSET_STARTING
> - ForkingRunListener.BOOTERCODE_TEST_STARTING
>
> But not the final:
> - ForkingRunListener.BOOTERCODE_TEST_SUCCEEDED
> - ForkingRunListener.BOOTERCODE_TESTSET_COMPLETED
>
> The original issue is on Linux, java 1.6, with a lot of tests running. I wrote a test case, hopefully reproducing the real issue with surefire 2.10 and 2.11. It's still random, but fails 50% of the time on my machine (laptop 2 cores, Windows 7, 64 bits, jdk1.6.0_24).
> In this example, we create a thread writing 1000 lines of 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" in stdout in the tear down method.
> That's an example of success with surefire 2.11. You see that exactly two lines of 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' has been written to the console. On some other successes it can be zero lines or more.
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Concurrency config is parallel='classes', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
> Running Test6
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec
> 658:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 659:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Results :
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.724s
> [INFO] Finished at: Tue Dec 27 11:46:03 CET 2011
> [INFO] Final Memory: 10M/109M
> [INFO] ------------------------------------------------------------------------
> {noformat}
> Error case, with surefire 2.11. In this one there is no 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' in the output, but you can have errors with multiple such lines.
> You see that the number of tests run is zero.
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Concurrency config is parallel='classes', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
> Running Test6
> Results :
> Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.820s
> [INFO] Finished at: Tue Dec 27 11:48:02 CET 2011
> [INFO] Final Memory: 10M/109M
> [INFO] ------------------------------------------------------------------------
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.11:test (default-test) on project surefireparametizedbug: No tests were executed! (Set -DfailIfNoTests=false to ignore this error.) -> [Help 1]
> [ERROR]
> {noformat}
> With junit4 instead of junit47 it seems it never fails, but you can still have some AAA lines in the final output:
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running Test6
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.07 sec
> 18:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 19:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 20:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 21:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 22:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Results :
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.667s
> [INFO] Finished at: Tue Dec 27 11:54:16 CET 2011
> [INFO] Final Memory: 10M/114M
> [INFO] ------------------------------------------------------------------------
> {noformat}
> It can fail as well (but less often it seems) if redirectTestOutputToFile is set to false.
> It fails as well with forkMode=once (on surefire 2.11 at least).
> parallel is set to "classes"; but if fails as well if it's not set (still on surefire 2.11 at least).
> Note that I write 1000 lines in the test case, it can fail with much less as well. It's just a simplification of the initially complex issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (SUREFIRE-813) Randomly wrong tests count and empty summary
files with JUnitProvider47 when forking the tests.
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/SUREFIRE-813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kristian Rosenvold closed SUREFIRE-813.
---------------------------------------
Resolution: Fixed
Fixed with unit test in r1225784, thanks for the test class !
> Randomly wrong tests count and empty summary files with JUnitProvider47 when forking the tests.
> -----------------------------------------------------------------------------------------------
>
> Key: SUREFIRE-813
> URL: https://jira.codehaus.org/browse/SUREFIRE-813
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.7+ (parallel) support
> Affects Versions: 2.10, 2.11
> Environment: test case checked on windows 7 64 bits; issues on Linux.
> jdk1.6.0_24
> Reporter: nkeywal
> Assignee: Kristian Rosenvold
> Attachments: pom.xml, Test6.java
>
>
> When running the test suite, with forkMode set and junit provider 47, we can have some test results missing. The test class is executed, but:
> - the summary file is empty (exists, but as a zero length file)
> - the global test count is not increased.
> It's random, and does not occurs all the time. The occurrence seems increased when the machine is loaded, but it's more an impression. In our case, with ~250 test classes, it occurs on average once per run (so it's a real problem). It seems to occur more if we activate 'perthread' parallelization.
>
> I added some log lines, and it seems that ForkClient does not receive all the test events when the problem occurs. It receives:
> - ForkingRunListener.BOOTERCODE_TESTSET_STARTING
> - ForkingRunListener.BOOTERCODE_TEST_STARTING
>
> But not the final:
> - ForkingRunListener.BOOTERCODE_TEST_SUCCEEDED
> - ForkingRunListener.BOOTERCODE_TESTSET_COMPLETED
>
> The original issue is on Linux, java 1.6, with a lot of tests running. I wrote a test case, hopefully reproducing the real issue with surefire 2.10 and 2.11. It's still random, but fails 50% of the time on my machine (laptop 2 cores, Windows 7, 64 bits, jdk1.6.0_24).
> In this example, we create a thread writing 1000 lines of 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" in stdout in the tear down method.
> That's an example of success with surefire 2.11. You see that exactly two lines of 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' has been written to the console. On some other successes it can be zero lines or more.
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Concurrency config is parallel='classes', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
> Running Test6
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec
> 658:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 659:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Results :
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.724s
> [INFO] Finished at: Tue Dec 27 11:46:03 CET 2011
> [INFO] Final Memory: 10M/109M
> [INFO] ------------------------------------------------------------------------
> {noformat}
> Error case, with surefire 2.11. In this one there is no 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' in the output, but you can have errors with multiple such lines.
> You see that the number of tests run is zero.
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Concurrency config is parallel='classes', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
> Running Test6
> Results :
> Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.820s
> [INFO] Finished at: Tue Dec 27 11:48:02 CET 2011
> [INFO] Final Memory: 10M/109M
> [INFO] ------------------------------------------------------------------------
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.11:test (default-test) on project surefireparametizedbug: No tests were executed! (Set -DfailIfNoTests=false to ignore this error.) -> [Help 1]
> [ERROR]
> {noformat}
> With junit4 instead of junit47 it seems it never fails, but you can still have some AAA lines in the final output:
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running Test6
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.07 sec
> 18:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 19:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 20:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 21:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 22:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Results :
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.667s
> [INFO] Finished at: Tue Dec 27 11:54:16 CET 2011
> [INFO] Final Memory: 10M/114M
> [INFO] ------------------------------------------------------------------------
> {noformat}
> It can fail as well (but less often it seems) if redirectTestOutputToFile is set to false.
> It fails as well with forkMode=once (on surefire 2.11 at least).
> parallel is set to "classes"; but if fails as well if it's not set (still on surefire 2.11 at least).
> Note that I write 1000 lines in the test case, it can fail with much less as well. It's just a simplification of the initially complex issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (SUREFIRE-813) Randomly wrong tests count and empty summary
files with JUnitProvider47 when forking the tests.
Posted by "nkeywal (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/SUREFIRE-813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=286925#comment-286925 ]
nkeywal commented on SUREFIRE-813:
----------------------------------
What you mean for LogicalStream line 81 is that it's possible to add a synchronized to the method to avoid concurrent modifications?
In our case, we have:
- daemon threads that will execute tasks every seconds or so
- shutdown hooks
- finalizers looking for unclosed resources and logging alerts if it's the case.
So deciding that the output is finished is not easy :-)
But as we fork with forkMode = always (and soon with perthread), it could be attributed to the right test class.
> Randomly wrong tests count and empty summary files with JUnitProvider47 when forking the tests.
> -----------------------------------------------------------------------------------------------
>
> Key: SUREFIRE-813
> URL: https://jira.codehaus.org/browse/SUREFIRE-813
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.7+ (parallel) support
> Affects Versions: 2.10, 2.11, 2.12
> Environment: test case checked on windows 7 64 bits; issues on Linux.
> jdk1.6.0_24
> Reporter: nkeywal
> Assignee: Kristian Rosenvold
> Attachments: pom.xml, Test6.java
>
>
> When running the test suite, with forkMode set and junit provider 47, we can have some test results missing. The test class is executed, but:
> - the summary file is empty (exists, but as a zero length file)
> - the global test count is not increased.
> It's random, and does not occurs all the time. The occurrence seems increased when the machine is loaded, but it's more an impression. In our case, with ~250 test classes, it occurs on average once per run (so it's a real problem). It seems to occur more if we activate 'perthread' parallelization.
>
> I added some log lines, and it seems that ForkClient does not receive all the test events when the problem occurs. It receives:
> - ForkingRunListener.BOOTERCODE_TESTSET_STARTING
> - ForkingRunListener.BOOTERCODE_TEST_STARTING
>
> But not the final:
> - ForkingRunListener.BOOTERCODE_TEST_SUCCEEDED
> - ForkingRunListener.BOOTERCODE_TESTSET_COMPLETED
>
> The original issue is on Linux, java 1.6, with a lot of tests running. I wrote a test case, hopefully reproducing the real issue with surefire 2.10 and 2.11. It's still random, but fails 50% of the time on my machine (laptop 2 cores, Windows 7, 64 bits, jdk1.6.0_24).
> In this example, we create a thread writing 1000 lines of 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" in stdout in the tear down method.
> That's an example of success with surefire 2.11. You see that exactly two lines of 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' has been written to the console. On some other successes it can be zero lines or more.
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Concurrency config is parallel='classes', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
> Running Test6
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec
> 658:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 659:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Results :
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.724s
> [INFO] Finished at: Tue Dec 27 11:46:03 CET 2011
> [INFO] Final Memory: 10M/109M
> [INFO] ------------------------------------------------------------------------
> {noformat}
> Error case, with surefire 2.11. In this one there is no 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' in the output, but you can have errors with multiple such lines.
> You see that the number of tests run is zero.
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Concurrency config is parallel='classes', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
> Running Test6
> Results :
> Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.820s
> [INFO] Finished at: Tue Dec 27 11:48:02 CET 2011
> [INFO] Final Memory: 10M/109M
> [INFO] ------------------------------------------------------------------------
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.11:test (default-test) on project surefireparametizedbug: No tests were executed! (Set -DfailIfNoTests=false to ignore this error.) -> [Help 1]
> [ERROR]
> {noformat}
> With junit4 instead of junit47 it seems it never fails, but you can still have some AAA lines in the final output:
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running Test6
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.07 sec
> 18:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 19:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 20:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 21:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 22:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Results :
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.667s
> [INFO] Finished at: Tue Dec 27 11:54:16 CET 2011
> [INFO] Final Memory: 10M/114M
> [INFO] ------------------------------------------------------------------------
> {noformat}
> It can fail as well (but less often it seems) if redirectTestOutputToFile is set to false.
> It fails as well with forkMode=once (on surefire 2.11 at least).
> parallel is set to "classes"; but if fails as well if it's not set (still on surefire 2.11 at least).
> Note that I write 1000 lines in the test case, it can fail with much less as well. It's just a simplification of the initially complex issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (SUREFIRE-813) Randomly wrong tests count and empty summary
files with JUnitProvider47 when forking the tests.
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/SUREFIRE-813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=286919#comment-286919 ]
Kristian Rosenvold commented on SUREFIRE-813:
---------------------------------------------
The root cause of this is a ConcurrentModificationException in LogicalStream line 81. While avoding the exception itself is simple, we again have somewhat of a halting problem, since there is no way to determined when any forked threads are finished. At this moment the only "solution" I can come up with is to detach further output from the test-cases in question and just mix it in with the single global stream.
Additionally, we could iterate all the streams on termination and maybe append "left over" output. This stuff isn't too easy to get right, so any suggestions are welcome ;)
> Randomly wrong tests count and empty summary files with JUnitProvider47 when forking the tests.
> -----------------------------------------------------------------------------------------------
>
> Key: SUREFIRE-813
> URL: https://jira.codehaus.org/browse/SUREFIRE-813
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.7+ (parallel) support
> Affects Versions: 2.10, 2.11, 2.12
> Environment: test case checked on windows 7 64 bits; issues on Linux.
> jdk1.6.0_24
> Reporter: nkeywal
> Assignee: Kristian Rosenvold
> Attachments: pom.xml, Test6.java
>
>
> When running the test suite, with forkMode set and junit provider 47, we can have some test results missing. The test class is executed, but:
> - the summary file is empty (exists, but as a zero length file)
> - the global test count is not increased.
> It's random, and does not occurs all the time. The occurrence seems increased when the machine is loaded, but it's more an impression. In our case, with ~250 test classes, it occurs on average once per run (so it's a real problem). It seems to occur more if we activate 'perthread' parallelization.
>
> I added some log lines, and it seems that ForkClient does not receive all the test events when the problem occurs. It receives:
> - ForkingRunListener.BOOTERCODE_TESTSET_STARTING
> - ForkingRunListener.BOOTERCODE_TEST_STARTING
>
> But not the final:
> - ForkingRunListener.BOOTERCODE_TEST_SUCCEEDED
> - ForkingRunListener.BOOTERCODE_TESTSET_COMPLETED
>
> The original issue is on Linux, java 1.6, with a lot of tests running. I wrote a test case, hopefully reproducing the real issue with surefire 2.10 and 2.11. It's still random, but fails 50% of the time on my machine (laptop 2 cores, Windows 7, 64 bits, jdk1.6.0_24).
> In this example, we create a thread writing 1000 lines of 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" in stdout in the tear down method.
> That's an example of success with surefire 2.11. You see that exactly two lines of 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' has been written to the console. On some other successes it can be zero lines or more.
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Concurrency config is parallel='classes', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
> Running Test6
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec
> 658:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 659:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Results :
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.724s
> [INFO] Finished at: Tue Dec 27 11:46:03 CET 2011
> [INFO] Final Memory: 10M/109M
> [INFO] ------------------------------------------------------------------------
> {noformat}
> Error case, with surefire 2.11. In this one there is no 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' in the output, but you can have errors with multiple such lines.
> You see that the number of tests run is zero.
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Concurrency config is parallel='classes', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
> Running Test6
> Results :
> Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.820s
> [INFO] Finished at: Tue Dec 27 11:48:02 CET 2011
> [INFO] Final Memory: 10M/109M
> [INFO] ------------------------------------------------------------------------
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.11:test (default-test) on project surefireparametizedbug: No tests were executed! (Set -DfailIfNoTests=false to ignore this error.) -> [Help 1]
> [ERROR]
> {noformat}
> With junit4 instead of junit47 it seems it never fails, but you can still have some AAA lines in the final output:
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running Test6
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.07 sec
> 18:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 19:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 20:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 21:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 22:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Results :
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.667s
> [INFO] Finished at: Tue Dec 27 11:54:16 CET 2011
> [INFO] Final Memory: 10M/114M
> [INFO] ------------------------------------------------------------------------
> {noformat}
> It can fail as well (but less often it seems) if redirectTestOutputToFile is set to false.
> It fails as well with forkMode=once (on surefire 2.11 at least).
> parallel is set to "classes"; but if fails as well if it's not set (still on surefire 2.11 at least).
> Note that I write 1000 lines in the test case, it can fail with much less as well. It's just a simplification of the initially complex issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (SUREFIRE-813) Randomly wrong tests count and empty summary
files with JUnitProvider47 when forking the tests.
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/SUREFIRE-813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=286914#comment-286914 ]
Kristian Rosenvold commented on SUREFIRE-813:
---------------------------------------------
Your testcase seems to demonstrate that output from the java process that happens *after* termination of the test itself seems to create trouble, and I can reproduce that; with or without any threading involved. Would that seem to be consistent with what you're experiencing ?
> Randomly wrong tests count and empty summary files with JUnitProvider47 when forking the tests.
> -----------------------------------------------------------------------------------------------
>
> Key: SUREFIRE-813
> URL: https://jira.codehaus.org/browse/SUREFIRE-813
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.7+ (parallel) support
> Affects Versions: 2.10, 2.11, 2.12
> Environment: test case checked on windows 7 64 bits; issues on Linux.
> jdk1.6.0_24
> Reporter: nkeywal
> Assignee: Kristian Rosenvold
> Attachments: pom.xml, Test6.java
>
>
> When running the test suite, with forkMode set and junit provider 47, we can have some test results missing. The test class is executed, but:
> - the summary file is empty (exists, but as a zero length file)
> - the global test count is not increased.
> It's random, and does not occurs all the time. The occurrence seems increased when the machine is loaded, but it's more an impression. In our case, with ~250 test classes, it occurs on average once per run (so it's a real problem). It seems to occur more if we activate 'perthread' parallelization.
>
> I added some log lines, and it seems that ForkClient does not receive all the test events when the problem occurs. It receives:
> - ForkingRunListener.BOOTERCODE_TESTSET_STARTING
> - ForkingRunListener.BOOTERCODE_TEST_STARTING
>
> But not the final:
> - ForkingRunListener.BOOTERCODE_TEST_SUCCEEDED
> - ForkingRunListener.BOOTERCODE_TESTSET_COMPLETED
>
> The original issue is on Linux, java 1.6, with a lot of tests running. I wrote a test case, hopefully reproducing the real issue with surefire 2.10 and 2.11. It's still random, but fails 50% of the time on my machine (laptop 2 cores, Windows 7, 64 bits, jdk1.6.0_24).
> In this example, we create a thread writing 1000 lines of 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" in stdout in the tear down method.
> That's an example of success with surefire 2.11. You see that exactly two lines of 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' has been written to the console. On some other successes it can be zero lines or more.
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Concurrency config is parallel='classes', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
> Running Test6
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec
> 658:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 659:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Results :
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.724s
> [INFO] Finished at: Tue Dec 27 11:46:03 CET 2011
> [INFO] Final Memory: 10M/109M
> [INFO] ------------------------------------------------------------------------
> {noformat}
> Error case, with surefire 2.11. In this one there is no 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' in the output, but you can have errors with multiple such lines.
> You see that the number of tests run is zero.
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Concurrency config is parallel='classes', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
> Running Test6
> Results :
> Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.820s
> [INFO] Finished at: Tue Dec 27 11:48:02 CET 2011
> [INFO] Final Memory: 10M/109M
> [INFO] ------------------------------------------------------------------------
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.11:test (default-test) on project surefireparametizedbug: No tests were executed! (Set -DfailIfNoTests=false to ignore this error.) -> [Help 1]
> [ERROR]
> {noformat}
> With junit4 instead of junit47 it seems it never fails, but you can still have some AAA lines in the final output:
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running Test6
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.07 sec
> 18:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 19:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 20:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 21:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 22:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Results :
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.667s
> [INFO] Finished at: Tue Dec 27 11:54:16 CET 2011
> [INFO] Final Memory: 10M/114M
> [INFO] ------------------------------------------------------------------------
> {noformat}
> It can fail as well (but less often it seems) if redirectTestOutputToFile is set to false.
> It fails as well with forkMode=once (on surefire 2.11 at least).
> parallel is set to "classes"; but if fails as well if it's not set (still on surefire 2.11 at least).
> Note that I write 1000 lines in the test case, it can fail with much less as well. It's just a simplification of the initially complex issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (SUREFIRE-813) Randomly wrong tests count and empty summary
files with JUnitProvider47 when forking the tests.
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/SUREFIRE-813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=286928#comment-286928 ]
Kristian Rosenvold commented on SUREFIRE-813:
---------------------------------------------
Btw; this is mostly a problem with the concurrent reporter, it should be a lot better with non-concurrent reporter and forkMode=perthread
> Randomly wrong tests count and empty summary files with JUnitProvider47 when forking the tests.
> -----------------------------------------------------------------------------------------------
>
> Key: SUREFIRE-813
> URL: https://jira.codehaus.org/browse/SUREFIRE-813
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.7+ (parallel) support
> Affects Versions: 2.10, 2.11, 2.12
> Environment: test case checked on windows 7 64 bits; issues on Linux.
> jdk1.6.0_24
> Reporter: nkeywal
> Assignee: Kristian Rosenvold
> Attachments: pom.xml, Test6.java
>
>
> When running the test suite, with forkMode set and junit provider 47, we can have some test results missing. The test class is executed, but:
> - the summary file is empty (exists, but as a zero length file)
> - the global test count is not increased.
> It's random, and does not occurs all the time. The occurrence seems increased when the machine is loaded, but it's more an impression. In our case, with ~250 test classes, it occurs on average once per run (so it's a real problem). It seems to occur more if we activate 'perthread' parallelization.
>
> I added some log lines, and it seems that ForkClient does not receive all the test events when the problem occurs. It receives:
> - ForkingRunListener.BOOTERCODE_TESTSET_STARTING
> - ForkingRunListener.BOOTERCODE_TEST_STARTING
>
> But not the final:
> - ForkingRunListener.BOOTERCODE_TEST_SUCCEEDED
> - ForkingRunListener.BOOTERCODE_TESTSET_COMPLETED
>
> The original issue is on Linux, java 1.6, with a lot of tests running. I wrote a test case, hopefully reproducing the real issue with surefire 2.10 and 2.11. It's still random, but fails 50% of the time on my machine (laptop 2 cores, Windows 7, 64 bits, jdk1.6.0_24).
> In this example, we create a thread writing 1000 lines of 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" in stdout in the tear down method.
> That's an example of success with surefire 2.11. You see that exactly two lines of 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' has been written to the console. On some other successes it can be zero lines or more.
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Concurrency config is parallel='classes', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
> Running Test6
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec
> 658:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 659:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Results :
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.724s
> [INFO] Finished at: Tue Dec 27 11:46:03 CET 2011
> [INFO] Final Memory: 10M/109M
> [INFO] ------------------------------------------------------------------------
> {noformat}
> Error case, with surefire 2.11. In this one there is no 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' in the output, but you can have errors with multiple such lines.
> You see that the number of tests run is zero.
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Concurrency config is parallel='classes', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
> Running Test6
> Results :
> Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.820s
> [INFO] Finished at: Tue Dec 27 11:48:02 CET 2011
> [INFO] Final Memory: 10M/109M
> [INFO] ------------------------------------------------------------------------
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.11:test (default-test) on project surefireparametizedbug: No tests were executed! (Set -DfailIfNoTests=false to ignore this error.) -> [Help 1]
> [ERROR]
> {noformat}
> With junit4 instead of junit47 it seems it never fails, but you can still have some AAA lines in the final output:
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running Test6
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.07 sec
> 18:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 19:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 20:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 21:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 22:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Results :
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.667s
> [INFO] Finished at: Tue Dec 27 11:54:16 CET 2011
> [INFO] Final Memory: 10M/114M
> [INFO] ------------------------------------------------------------------------
> {noformat}
> It can fail as well (but less often it seems) if redirectTestOutputToFile is set to false.
> It fails as well with forkMode=once (on surefire 2.11 at least).
> parallel is set to "classes"; but if fails as well if it's not set (still on surefire 2.11 at least).
> Note that I write 1000 lines in the test case, it can fail with much less as well. It's just a simplification of the initially complex issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (SUREFIRE-813) Randomly wrong tests count and empty summary
files with JUnitProvider47 when forking the tests.
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/SUREFIRE-813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=286927#comment-286927 ]
Kristian Rosenvold commented on SUREFIRE-813:
---------------------------------------------
simple synchronization would be a stopgap that would relieve the critical part of the problem, but a good solution would have to cover what to to with output receiced "after" the LogicalStream has been reported (in other words, after "writeDetails" has been invoked)
> Randomly wrong tests count and empty summary files with JUnitProvider47 when forking the tests.
> -----------------------------------------------------------------------------------------------
>
> Key: SUREFIRE-813
> URL: https://jira.codehaus.org/browse/SUREFIRE-813
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.7+ (parallel) support
> Affects Versions: 2.10, 2.11, 2.12
> Environment: test case checked on windows 7 64 bits; issues on Linux.
> jdk1.6.0_24
> Reporter: nkeywal
> Assignee: Kristian Rosenvold
> Attachments: pom.xml, Test6.java
>
>
> When running the test suite, with forkMode set and junit provider 47, we can have some test results missing. The test class is executed, but:
> - the summary file is empty (exists, but as a zero length file)
> - the global test count is not increased.
> It's random, and does not occurs all the time. The occurrence seems increased when the machine is loaded, but it's more an impression. In our case, with ~250 test classes, it occurs on average once per run (so it's a real problem). It seems to occur more if we activate 'perthread' parallelization.
>
> I added some log lines, and it seems that ForkClient does not receive all the test events when the problem occurs. It receives:
> - ForkingRunListener.BOOTERCODE_TESTSET_STARTING
> - ForkingRunListener.BOOTERCODE_TEST_STARTING
>
> But not the final:
> - ForkingRunListener.BOOTERCODE_TEST_SUCCEEDED
> - ForkingRunListener.BOOTERCODE_TESTSET_COMPLETED
>
> The original issue is on Linux, java 1.6, with a lot of tests running. I wrote a test case, hopefully reproducing the real issue with surefire 2.10 and 2.11. It's still random, but fails 50% of the time on my machine (laptop 2 cores, Windows 7, 64 bits, jdk1.6.0_24).
> In this example, we create a thread writing 1000 lines of 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" in stdout in the tear down method.
> That's an example of success with surefire 2.11. You see that exactly two lines of 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' has been written to the console. On some other successes it can be zero lines or more.
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Concurrency config is parallel='classes', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
> Running Test6
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec
> 658:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 659:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Results :
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.724s
> [INFO] Finished at: Tue Dec 27 11:46:03 CET 2011
> [INFO] Final Memory: 10M/109M
> [INFO] ------------------------------------------------------------------------
> {noformat}
> Error case, with surefire 2.11. In this one there is no 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' in the output, but you can have errors with multiple such lines.
> You see that the number of tests run is zero.
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Concurrency config is parallel='classes', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
> Running Test6
> Results :
> Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.820s
> [INFO] Finished at: Tue Dec 27 11:48:02 CET 2011
> [INFO] Final Memory: 10M/109M
> [INFO] ------------------------------------------------------------------------
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.11:test (default-test) on project surefireparametizedbug: No tests were executed! (Set -DfailIfNoTests=false to ignore this error.) -> [Help 1]
> [ERROR]
> {noformat}
> With junit4 instead of junit47 it seems it never fails, but you can still have some AAA lines in the final output:
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running Test6
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.07 sec
> 18:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 19:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 20:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 21:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 22:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Results :
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.667s
> [INFO] Finished at: Tue Dec 27 11:54:16 CET 2011
> [INFO] Final Memory: 10M/114M
> [INFO] ------------------------------------------------------------------------
> {noformat}
> It can fail as well (but less often it seems) if redirectTestOutputToFile is set to false.
> It fails as well with forkMode=once (on surefire 2.11 at least).
> parallel is set to "classes"; but if fails as well if it's not set (still on surefire 2.11 at least).
> Note that I write 1000 lines in the test case, it can fail with much less as well. It's just a simplification of the initially complex issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (SUREFIRE-813) Randomly wrong tests count and empty summary
files with JUnitProvider47 when forking the tests.
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/SUREFIRE-813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kristian Rosenvold updated SUREFIRE-813:
----------------------------------------
Affects Version/s: (was: 2.12)
Fix Version/s: 2.12
> Randomly wrong tests count and empty summary files with JUnitProvider47 when forking the tests.
> -----------------------------------------------------------------------------------------------
>
> Key: SUREFIRE-813
> URL: https://jira.codehaus.org/browse/SUREFIRE-813
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.7+ (parallel) support
> Affects Versions: 2.10, 2.11
> Environment: test case checked on windows 7 64 bits; issues on Linux.
> jdk1.6.0_24
> Reporter: nkeywal
> Assignee: Kristian Rosenvold
> Fix For: 2.12
>
> Attachments: pom.xml, Test6.java
>
>
> When running the test suite, with forkMode set and junit provider 47, we can have some test results missing. The test class is executed, but:
> - the summary file is empty (exists, but as a zero length file)
> - the global test count is not increased.
> It's random, and does not occurs all the time. The occurrence seems increased when the machine is loaded, but it's more an impression. In our case, with ~250 test classes, it occurs on average once per run (so it's a real problem). It seems to occur more if we activate 'perthread' parallelization.
>
> I added some log lines, and it seems that ForkClient does not receive all the test events when the problem occurs. It receives:
> - ForkingRunListener.BOOTERCODE_TESTSET_STARTING
> - ForkingRunListener.BOOTERCODE_TEST_STARTING
>
> But not the final:
> - ForkingRunListener.BOOTERCODE_TEST_SUCCEEDED
> - ForkingRunListener.BOOTERCODE_TESTSET_COMPLETED
>
> The original issue is on Linux, java 1.6, with a lot of tests running. I wrote a test case, hopefully reproducing the real issue with surefire 2.10 and 2.11. It's still random, but fails 50% of the time on my machine (laptop 2 cores, Windows 7, 64 bits, jdk1.6.0_24).
> In this example, we create a thread writing 1000 lines of 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" in stdout in the tear down method.
> That's an example of success with surefire 2.11. You see that exactly two lines of 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' has been written to the console. On some other successes it can be zero lines or more.
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Concurrency config is parallel='classes', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
> Running Test6
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec
> 658:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 659:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Results :
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.724s
> [INFO] Finished at: Tue Dec 27 11:46:03 CET 2011
> [INFO] Final Memory: 10M/109M
> [INFO] ------------------------------------------------------------------------
> {noformat}
> Error case, with surefire 2.11. In this one there is no 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' in the output, but you can have errors with multiple such lines.
> You see that the number of tests run is zero.
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Concurrency config is parallel='classes', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
> Running Test6
> Results :
> Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.820s
> [INFO] Finished at: Tue Dec 27 11:48:02 CET 2011
> [INFO] Final Memory: 10M/109M
> [INFO] ------------------------------------------------------------------------
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.11:test (default-test) on project surefireparametizedbug: No tests were executed! (Set -DfailIfNoTests=false to ignore this error.) -> [Help 1]
> [ERROR]
> {noformat}
> With junit4 instead of junit47 it seems it never fails, but you can still have some AAA lines in the final output:
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running Test6
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.07 sec
> 18:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 19:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 20:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 21:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 22:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Results :
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.667s
> [INFO] Finished at: Tue Dec 27 11:54:16 CET 2011
> [INFO] Final Memory: 10M/114M
> [INFO] ------------------------------------------------------------------------
> {noformat}
> It can fail as well (but less often it seems) if redirectTestOutputToFile is set to false.
> It fails as well with forkMode=once (on surefire 2.11 at least).
> parallel is set to "classes"; but if fails as well if it's not set (still on surefire 2.11 at least).
> Note that I write 1000 lines in the test case, it can fail with much less as well. It's just a simplification of the initially complex issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (SUREFIRE-813) Randomly wrong tests count and empty summary
files with JUnitProvider47 when forking the tests.
Posted by "nkeywal (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/SUREFIRE-813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=286939#comment-286939 ]
nkeywal commented on SUREFIRE-813:
----------------------------------
After more tests, I confirm it works well with the synchronized method. I've done as well an unrelated modification to close the fork client when there is an exception. The change is there: https://github.com/nkeywal/maven-surefire/commit/be9c95b4acdf2ebddb1ee5e0564d411c6eae6f3a
> Randomly wrong tests count and empty summary files with JUnitProvider47 when forking the tests.
> -----------------------------------------------------------------------------------------------
>
> Key: SUREFIRE-813
> URL: https://jira.codehaus.org/browse/SUREFIRE-813
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.7+ (parallel) support
> Affects Versions: 2.10, 2.11, 2.12
> Environment: test case checked on windows 7 64 bits; issues on Linux.
> jdk1.6.0_24
> Reporter: nkeywal
> Assignee: Kristian Rosenvold
> Attachments: pom.xml, Test6.java
>
>
> When running the test suite, with forkMode set and junit provider 47, we can have some test results missing. The test class is executed, but:
> - the summary file is empty (exists, but as a zero length file)
> - the global test count is not increased.
> It's random, and does not occurs all the time. The occurrence seems increased when the machine is loaded, but it's more an impression. In our case, with ~250 test classes, it occurs on average once per run (so it's a real problem). It seems to occur more if we activate 'perthread' parallelization.
>
> I added some log lines, and it seems that ForkClient does not receive all the test events when the problem occurs. It receives:
> - ForkingRunListener.BOOTERCODE_TESTSET_STARTING
> - ForkingRunListener.BOOTERCODE_TEST_STARTING
>
> But not the final:
> - ForkingRunListener.BOOTERCODE_TEST_SUCCEEDED
> - ForkingRunListener.BOOTERCODE_TESTSET_COMPLETED
>
> The original issue is on Linux, java 1.6, with a lot of tests running. I wrote a test case, hopefully reproducing the real issue with surefire 2.10 and 2.11. It's still random, but fails 50% of the time on my machine (laptop 2 cores, Windows 7, 64 bits, jdk1.6.0_24).
> In this example, we create a thread writing 1000 lines of 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" in stdout in the tear down method.
> That's an example of success with surefire 2.11. You see that exactly two lines of 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' has been written to the console. On some other successes it can be zero lines or more.
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Concurrency config is parallel='classes', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
> Running Test6
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec
> 658:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 659:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Results :
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.724s
> [INFO] Finished at: Tue Dec 27 11:46:03 CET 2011
> [INFO] Final Memory: 10M/109M
> [INFO] ------------------------------------------------------------------------
> {noformat}
> Error case, with surefire 2.11. In this one there is no 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' in the output, but you can have errors with multiple such lines.
> You see that the number of tests run is zero.
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Concurrency config is parallel='classes', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
> Running Test6
> Results :
> Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.820s
> [INFO] Finished at: Tue Dec 27 11:48:02 CET 2011
> [INFO] Final Memory: 10M/109M
> [INFO] ------------------------------------------------------------------------
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.11:test (default-test) on project surefireparametizedbug: No tests were executed! (Set -DfailIfNoTests=false to ignore this error.) -> [Help 1]
> [ERROR]
> {noformat}
> With junit4 instead of junit47 it seems it never fails, but you can still have some AAA lines in the final output:
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running Test6
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.07 sec
> 18:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 19:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 20:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 21:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 22:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Results :
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.667s
> [INFO] Finished at: Tue Dec 27 11:54:16 CET 2011
> [INFO] Final Memory: 10M/114M
> [INFO] ------------------------------------------------------------------------
> {noformat}
> It can fail as well (but less often it seems) if redirectTestOutputToFile is set to false.
> It fails as well with forkMode=once (on surefire 2.11 at least).
> parallel is set to "classes"; but if fails as well if it's not set (still on surefire 2.11 at least).
> Note that I write 1000 lines in the test case, it can fail with much less as well. It's just a simplification of the initially complex issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (SUREFIRE-813) Randomly wrong tests count and empty summary
files with JUnitProvider47 when forking the tests.
Posted by "nkeywal (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/SUREFIRE-813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=286930#comment-286930 ]
nkeywal commented on SUREFIRE-813:
----------------------------------
Yep, with the synchronized all the events are received by ForkClient. However there is something fishy afterward I need to look at.
There is a non concurrent reporter? By default with categories/JUnit47 all the reporters are concurrent?
> Randomly wrong tests count and empty summary files with JUnitProvider47 when forking the tests.
> -----------------------------------------------------------------------------------------------
>
> Key: SUREFIRE-813
> URL: https://jira.codehaus.org/browse/SUREFIRE-813
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.7+ (parallel) support
> Affects Versions: 2.10, 2.11, 2.12
> Environment: test case checked on windows 7 64 bits; issues on Linux.
> jdk1.6.0_24
> Reporter: nkeywal
> Assignee: Kristian Rosenvold
> Attachments: pom.xml, Test6.java
>
>
> When running the test suite, with forkMode set and junit provider 47, we can have some test results missing. The test class is executed, but:
> - the summary file is empty (exists, but as a zero length file)
> - the global test count is not increased.
> It's random, and does not occurs all the time. The occurrence seems increased when the machine is loaded, but it's more an impression. In our case, with ~250 test classes, it occurs on average once per run (so it's a real problem). It seems to occur more if we activate 'perthread' parallelization.
>
> I added some log lines, and it seems that ForkClient does not receive all the test events when the problem occurs. It receives:
> - ForkingRunListener.BOOTERCODE_TESTSET_STARTING
> - ForkingRunListener.BOOTERCODE_TEST_STARTING
>
> But not the final:
> - ForkingRunListener.BOOTERCODE_TEST_SUCCEEDED
> - ForkingRunListener.BOOTERCODE_TESTSET_COMPLETED
>
> The original issue is on Linux, java 1.6, with a lot of tests running. I wrote a test case, hopefully reproducing the real issue with surefire 2.10 and 2.11. It's still random, but fails 50% of the time on my machine (laptop 2 cores, Windows 7, 64 bits, jdk1.6.0_24).
> In this example, we create a thread writing 1000 lines of 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" in stdout in the tear down method.
> That's an example of success with surefire 2.11. You see that exactly two lines of 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' has been written to the console. On some other successes it can be zero lines or more.
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Concurrency config is parallel='classes', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
> Running Test6
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec
> 658:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 659:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Results :
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.724s
> [INFO] Finished at: Tue Dec 27 11:46:03 CET 2011
> [INFO] Final Memory: 10M/109M
> [INFO] ------------------------------------------------------------------------
> {noformat}
> Error case, with surefire 2.11. In this one there is no 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' in the output, but you can have errors with multiple such lines.
> You see that the number of tests run is zero.
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Concurrency config is parallel='classes', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
> Running Test6
> Results :
> Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.820s
> [INFO] Finished at: Tue Dec 27 11:48:02 CET 2011
> [INFO] Final Memory: 10M/109M
> [INFO] ------------------------------------------------------------------------
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.11:test (default-test) on project surefireparametizedbug: No tests were executed! (Set -DfailIfNoTests=false to ignore this error.) -> [Help 1]
> [ERROR]
> {noformat}
> With junit4 instead of junit47 it seems it never fails, but you can still have some AAA lines in the final output:
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running Test6
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.07 sec
> 18:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 19:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 20:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 21:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> 22:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Results :
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.667s
> [INFO] Finished at: Tue Dec 27 11:54:16 CET 2011
> [INFO] Final Memory: 10M/114M
> [INFO] ------------------------------------------------------------------------
> {noformat}
> It can fail as well (but less often it seems) if redirectTestOutputToFile is set to false.
> It fails as well with forkMode=once (on surefire 2.11 at least).
> parallel is set to "classes"; but if fails as well if it's not set (still on surefire 2.11 at least).
> Note that I write 1000 lines in the test case, it can fail with much less as well. It's just a simplification of the initially complex issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira