You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@maven.apache.org by "Jean-Luc Derrien (JIRA)" <ji...@apache.org> on 2016/08/16 16:16:20 UTC
[jira] [Comment Edited] (SUREFIRE-1264) Some tests can be lost when
running in parallel with parameterized tests
[ https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422967#comment-15422967 ]
Jean-Luc Derrien edited comment on SUREFIRE-1264 at 8/16/16 4:16 PM:
---------------------------------------------------------------------
[~tibor17], I've modified my branch [simple-1|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] accordingly.
I still reproduce the problem with {{ConcurrentLinkedQueue}}. It seems to be a reporting error since all the 12 files are created even when the output reports 11 tests:
{noformat}
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running [p2]
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - sleeptime = 2 => start
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - sleeptime = 5 => start
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - sleeptime = 1 => start
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - in [p2]
Running [p2]
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - in [p2]
Results :
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 7.315 s
[INFO] Finished at: 2016-08-16T18:06:26+02:00
[INFO] Final Memory: 16M/168M
[INFO] ------------------------------------------------------------------------
{noformat}
{noformat}
$ ls target/*.log
target/com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0].log
target/com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1].log
target/com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2].log
target/com.appnexus.viewability.core.surefireJunitTests.ATest.methodA2[p0].log
target/com.appnexus.viewability.core.surefireJunitTests.ATest.methodA2[p1].log
target/com.appnexus.viewability.core.surefireJunitTests.ATest.methodA2[p2].log
target/com.appnexus.viewability.core.surefireJunitTests.BTest.methodB1[p0].log
target/com.appnexus.viewability.core.surefireJunitTests.BTest.methodB1[p1].log
target/com.appnexus.viewability.core.surefireJunitTests.BTest.methodB1[p2].log
target/com.appnexus.viewability.core.surefireJunitTests.BTest.methodB2[p0].log
target/com.appnexus.viewability.core.surefireJunitTests.BTest.methodB2[p1].log
target/com.appnexus.viewability.core.surefireJunitTests.BTest.methodB2[p2].log
{noformat}
Note: I also reproduce the problem on Linux with different version of Java and Maven:
{noformat}
$ mvn --version
Warning: JAVA_HOME environment variable is not set.
Apache Maven 2.2.1 (r801777; 2009-08-06 21:16:01+0200)
Java version: 1.7.0_95
Java home: /usr/lib/jvm/java-7-openjdk-amd64/jre
Default locale: en_GB, platform encoding: UTF-8
OS name: "linux" version: "3.14.32-xxxx-std-ipv6-64" arch: "amd64" Family: "unix"
{noformat}
In this case, the output is slightly different (10 tests only, and the output shows the {{=> stop}} logs correctly when the problem occurs, which is not the case with my other setup):
{noformat}
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running [p2]
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec - in [p2]
Running [p2]
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - sleeptime = 2 => start
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - sleeptime = 2 => stop
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - sleeptime = 5 => start
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - sleeptime = 5 => stop
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - sleeptime = 1 => start
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - sleeptime = 1 => stop
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.002 sec - in [p2]
Results :
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 7 seconds
[INFO] Finished at: Tue Aug 16 17:32:18 CEST 2016
[INFO] Final Memory: 30M/478M
[INFO] ------------------------------------------------------------------------
{noformat}
was (Author: jderrien):
[~tibor17], I've modified my branch [simple-1|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] accordingly.
I still reproduce the problem with {{ConcurrentLinkedQueue}}. It seems to be a reporting error since all the 12 files are created even when the output reports 11 tests:
{noformat}
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running [p2]
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - sleeptime = 2 => start
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - sleeptime = 5 => start
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - sleeptime = 1 => start
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - in [p2]
Running [p2]
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - in [p2]
Results :
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 7.315 s
[INFO] Finished at: 2016-08-16T18:06:26+02:00
[INFO] Final Memory: 16M/168M
[INFO] ------------------------------------------------------------------------
{noformat}
{noformat}
$ ls target/*.log
target/com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0].log
target/com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1].log
target/com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2].log
target/com.appnexus.viewability.core.surefireJunitTests.ATest.methodA2[p0].log
target/com.appnexus.viewability.core.surefireJunitTests.ATest.methodA2[p1].log
target/com.appnexus.viewability.core.surefireJunitTests.ATest.methodA2[p2].log
target/com.appnexus.viewability.core.surefireJunitTests.BTest.methodB1[p0].log
target/com.appnexus.viewability.core.surefireJunitTests.BTest.methodB1[p1].log
target/com.appnexus.viewability.core.surefireJunitTests.BTest.methodB1[p2].log
target/com.appnexus.viewability.core.surefireJunitTests.BTest.methodB2[p0].log
target/com.appnexus.viewability.core.surefireJunitTests.BTest.methodB2[p1].log
target/com.appnexus.viewability.core.surefireJunitTests.BTest.methodB2[p2].log
{noformat}
Note: I also reproduce the problem on Linux with different version of Java and Maven:
{noformat}
aiantis@s30:~/tmp/surefire-junit-tests$ mvn --version
Warning: JAVA_HOME environment variable is not set.
Apache Maven 2.2.1 (r801777; 2009-08-06 21:16:01+0200)
Java version: 1.7.0_95
Java home: /usr/lib/jvm/java-7-openjdk-amd64/jre
Default locale: en_GB, platform encoding: UTF-8
OS name: "linux" version: "3.14.32-xxxx-std-ipv6-64" arch: "amd64" Family: "unix"
{noformat}
In this case, the output is slightly different (10 tests only, and the output shows the {{=> stop}} logs correctly when the problem occurs, which is not the case with my other setup):
{noformat}
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running [p2]
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec - in [p2]
Running [p2]
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - sleeptime = 2 => start
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - sleeptime = 2 => stop
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - sleeptime = 5 => start
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - sleeptime = 5 => stop
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - sleeptime = 1 => start
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - sleeptime = 1 => stop
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.002 sec - in [p2]
Results :
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 7 seconds
[INFO] Finished at: Tue Aug 16 17:32:18 CEST 2016
[INFO] Final Memory: 30M/478M
[INFO] ------------------------------------------------------------------------
{noformat}
> Some tests can be lost when running in parallel with parameterized tests
> ------------------------------------------------------------------------
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.7+ (parallel) support
> Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
> Reporter: Jean-Luc Derrien
>
> Hello,
> It appears some tests can be lost when using the parallel mode with parameterized tests. Here is a [small project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's even more frequent/easy to reproduce.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)