You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@maven.apache.org by Stevo Slavić <ss...@gmail.com> on 2013/08/29 12:52:57 UTC
[surefire] Combining test, runOrder and forkCount
Hello Apache Maven community,
To reproduce random failing test issue, I'm trying to run two tests in
parallel, but start running them in specific order. Problem is that
runOrder (added via SUREFIRE-614) doesn't seem to be respected, at least
not when forkCount and/or test is specified as well.
When trying to run two tests from Apache Mahout core module (trunk), I
cannot get surefire to run them in reverse alphabetical order.
Regardless if I run:
mvn -Dtest=*KMeansTest -DrunOrder=reversealphabetical test
or
mvn -Dtest=*KMeansTest -DrunOrder=alphabetical test
the two tests which match the pattern always get run in alphabetical order:
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.mahout.clustering.streaming.cluster.BallKMeansTest
Running org.apache.mahout.clustering.streaming.cluster.StreamingKMeansTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.706 sec
- in org.apache.mahout.clustering.streaming.cluster.BallKMeansTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 82.15 sec -
in org.apache.mahout.clustering.streaming.cluster.StreamingKMeansTest
Results :
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
Surefire 2.15 is used, and configuration is:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<forkCount>1.5C</forkCount>
<reuseForks>false</reuseForks>
<threadCount>1</threadCount>
<perCoreThreadCount>false</perCoreThreadCount>
<parallel>classes</parallel>
<argLine>-Xmx512m</argLine>
<testFailureIgnore>false</testFailureIgnore>
<redirectTestOutputToFile>true</redirectTestOutputToFile>
<systemPropertyVariables>
<mahout.test.directory>${project.build.directory}</mahout.test.directory>
</systemPropertyVariables>
</configuration>
</plugin>
When I use latest surefire 2.16, outcome is even more odd - output that any
of the tests is running takes longer to render, it seems to be rendered
together with info that test has completed, so by looking at build log
output it may appear as if tests were not running in parallel even though
they did:
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.mahout.clustering.streaming.cluster.BallKMeansTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 59.347 sec
- in org.apache.mahout.clustering.streaming.cluster.BallKMeansTest
Running org.apache.mahout.clustering.streaming.cluster.StreamingKMeansTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 74.766 sec
- in org.apache.mahout.clustering.streaming.cluster.StreamingKMeansTest
Results :
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
Here is my "mvn --version" output:
[sslavic@laptop core]$ mvn --version
Apache Maven 3.1.0 (893ca28a1da9d5f51ac03827af98bb730128f9f2; 2013-06-28
04:15:32+0200)
Maven home: /home/sslavic/work/tools/apache-maven
Java version: 1.7.0_25, vendor: Oracle Corporation
Java home: /usr/java/jdk1.7.0_25/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.9-200.fc19.x86_64", arch: "amd64", family:
"unix"
Am I doing something wrong or is this behavior bug (or two bugs) in
surefire?
Kind regards,
Stevo Slavic.
Re: [surefire] Combining test, runOrder and forkCount
Posted by Andreas Gudian <an...@gmail.com>.
Am Freitag, 30. August 2013 schrieb Stevo Slavić :
> Thanks Andreas,
>
> Removing parallel=classes did the trick for timely console reporting.
> Having parallel=classes didn't bother m-surefire-p in 2.15 and earlier.
In surefire 2.16, the whole Junit parallel stuff was rewritten, so some
changed behavior is to be expected.
> With forkCount > 1 and reuseForks = false, parallel=classes should just be
> ignored. But, as long as there is solution, I'm fine.
>
> IMO specifying test parameter shouldn't disable the ordering (runOrder),
> since specifying test parameter (by docs) is equivalent to specifying
> includes patterns in pom, and includes should be used only to filter
> scanned test classes, ordering should be applied after scanning, and before
> executing test classes.
>
> I understand that there are no guarantees on order when only two test
> classes are run in parallel with forkCount > 1.
> Still if runOrder is applied before forking VMs and forked VMs as you
> describe poll for next test class to execute, and I run a dozen of tests
> (specified with test parameter) each of which takes different number of
> more than few seconds to finish, and I have forkCount set to 2, I'd expect
> runOrder to be somewhat respected - for the first two tests which get run
> it would be random, but for the rest it should be pretty much in specified
> runOrder, until two finish at roughly same time and there is (again as on
> beginning) concurrency which test class will run next. Do you agree?
Totally. Would you please create a bug report for that?
Thanks,
Andreas
> I tried that (test specifying dozen of test classes, forkCount at 2,
> reuseForks at false, and different runOrder settings), and still I don't
> see runOrder being respected.
>
> Btw my original suspicion, that the two of Apache Mahout tests do not run
> well together, was wrong - failing test had a bug, it makes use of random
> number generator, but didn't set test seed to make execution deterministic.
>
> Kind regards,
> Stevo Slavic.
>
>
> On Fri, Aug 30, 2013 at 8:06 AM, Andreas Gudian <andreas.gudian@gmail.com<javascript:;>
> >wrote:
>
> > I'll have to check the code, but I'm pretty sure that the order of the
> test
> > classes to be handed over to the forked VMs is supposed to be what was is
> > specified with runOrder.
> >
> > However, using -Dtest= might disable the ordering - but again, I'd have
> to
> > check the code.
> >
> > Bit there's more: when having two classes to test and two forked VMs,
> there
> > is no real way to tell which test comes first. Both VMs are started right
> > away and only when the are up and running and ready to execute a test,
> they
> > ask the main process for one test class. Which VM will be the first to
> get
> > a class is quite random.
> >
> > For the delayed reporting to console, try removing parallel=classes from
> > the config. IThe classes-setting has no effect when forking to more than
> > one fork anyway. If the output is still delayed then, create a
> bug-report,
> > please.
> >
> >
> > Am Donnerstag, 29. August 2013 schrieb Stevo Slavić :
> >
> > > Hello Apache Maven community,
> > >
> > > To reproduce random failing test issue, I'm trying to run two tests in
> > > parallel, but start running them in specific order. Problem is that
> > > runOrder (added via SUREFIRE-614) doesn't seem to be respected, at
> least
> > > not when forkCount and/or test is specified as well.
> > >
> > > When trying to run two tests from Apache Mahout core module (trunk), I
> > > cannot get surefire to run them in reverse alphabetical order.
> > >
> > > Regardless if I run:
> > > mvn -Dtest=*KMeansTest -DrunOrder=reversealphabetical test
> > >
> > > or
> > > mvn -Dtest=*KMeansTest -DrunOrder=alphabetical test
> > >
> > > the two tests which match the pattern always get run in alphabetical
> > order:
> > >
> > > -------------------------------------------------------
> > > T E S T S
> > > -------------------------------------------------------
> > > Running org.apache.mahout.clustering.streaming.cluster.BallKMeansTest
> > > Running
> > org.apache.mahout.clustering.streaming.cluster.StreamingKMeansTest
> > > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.706
> > sec
> > > - in org.apache.mahout.clustering.streaming.cluster.BallKMeansTest
> > > Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 82.15
> > sec -
> > > in org.apache.mahout.clustering.streaming.cluster.StreamingKMeansTest
> > >
> > > Results :
> > >
> > > Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> > >
> > >
> > >
> > >
> > > Surefire 2.15 is used, and configuration is:
> > >
> > >
> > > <plugin>
> > > <groupId>org.apache.maven.plugins</groupId>
> > > <artifactId>maven-surefire-plugin</artifactId>
> > > <configuration>
> > > <forkCount>1.5C</forkCount>
> > > <reuseForks>false</reuseForks>
> > > <threadCount>1</threadCount>
> > > <perCoreThreadCount>false</perCoreThreadCount>
> > > <parallel>classes</parallel>
> > > <argLine>-Xmx512m</argLine>
> > > <testFailureIgnore>false</testFailureIgnore>
> > > <redirectTestOutputToFile>true</redirectTestOutputToFile>
> > > <systemPropertyVariables>
> > >
> > >
> <mahout.test.directory>${project.build.directory}</mahout.test.directory>
> > > </systemPropertyVariables>
> > > </configuration>
> > > </plugin>
> > >
> > >
> > > When I use latest surefire 2.16, outcome is even more odd - output that
> > any
> > > of the tests is running takes longer to render, it seems to be rendered
> > > together with info that test has completed, so by looking at build log
> > > output it may appear as if tests were not running in parallel even
> though
>
Re: [surefire] Combining test, runOrder and forkCount
Posted by Stevo Slavić <ss...@gmail.com>.
Thanks Andreas,
Removing parallel=classes did the trick for timely console reporting.
Having parallel=classes didn't bother m-surefire-p in 2.15 and earlier.
With forkCount > 1 and reuseForks = false, parallel=classes should just be
ignored. But, as long as there is solution, I'm fine.
IMO specifying test parameter shouldn't disable the ordering (runOrder),
since specifying test parameter (by docs) is equivalent to specifying
includes patterns in pom, and includes should be used only to filter
scanned test classes, ordering should be applied after scanning, and before
executing test classes.
I understand that there are no guarantees on order when only two test
classes are run in parallel with forkCount > 1.
Still if runOrder is applied before forking VMs and forked VMs as you
describe poll for next test class to execute, and I run a dozen of tests
(specified with test parameter) each of which takes different number of
more than few seconds to finish, and I have forkCount set to 2, I'd expect
runOrder to be somewhat respected - for the first two tests which get run
it would be random, but for the rest it should be pretty much in specified
runOrder, until two finish at roughly same time and there is (again as on
beginning) concurrency which test class will run next. Do you agree?
I tried that (test specifying dozen of test classes, forkCount at 2,
reuseForks at false, and different runOrder settings), and still I don't
see runOrder being respected.
Btw my original suspicion, that the two of Apache Mahout tests do not run
well together, was wrong - failing test had a bug, it makes use of random
number generator, but didn't set test seed to make execution deterministic.
Kind regards,
Stevo Slavic.
On Fri, Aug 30, 2013 at 8:06 AM, Andreas Gudian <an...@gmail.com>wrote:
> I'll have to check the code, but I'm pretty sure that the order of the test
> classes to be handed over to the forked VMs is supposed to be what was is
> specified with runOrder.
>
> However, using -Dtest= might disable the ordering - but again, I'd have to
> check the code.
>
> Bit there's more: when having two classes to test and two forked VMs, there
> is no real way to tell which test comes first. Both VMs are started right
> away and only when the are up and running and ready to execute a test, they
> ask the main process for one test class. Which VM will be the first to get
> a class is quite random.
>
> For the delayed reporting to console, try removing parallel=classes from
> the config. IThe classes-setting has no effect when forking to more than
> one fork anyway. If the output is still delayed then, create a bug-report,
> please.
>
>
> Am Donnerstag, 29. August 2013 schrieb Stevo Slavić :
>
> > Hello Apache Maven community,
> >
> > To reproduce random failing test issue, I'm trying to run two tests in
> > parallel, but start running them in specific order. Problem is that
> > runOrder (added via SUREFIRE-614) doesn't seem to be respected, at least
> > not when forkCount and/or test is specified as well.
> >
> > When trying to run two tests from Apache Mahout core module (trunk), I
> > cannot get surefire to run them in reverse alphabetical order.
> >
> > Regardless if I run:
> > mvn -Dtest=*KMeansTest -DrunOrder=reversealphabetical test
> >
> > or
> > mvn -Dtest=*KMeansTest -DrunOrder=alphabetical test
> >
> > the two tests which match the pattern always get run in alphabetical
> order:
> >
> > -------------------------------------------------------
> > T E S T S
> > -------------------------------------------------------
> > Running org.apache.mahout.clustering.streaming.cluster.BallKMeansTest
> > Running
> org.apache.mahout.clustering.streaming.cluster.StreamingKMeansTest
> > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.706
> sec
> > - in org.apache.mahout.clustering.streaming.cluster.BallKMeansTest
> > Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 82.15
> sec -
> > in org.apache.mahout.clustering.streaming.cluster.StreamingKMeansTest
> >
> > Results :
> >
> > Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> >
> >
> >
> >
> > Surefire 2.15 is used, and configuration is:
> >
> >
> > <plugin>
> > <groupId>org.apache.maven.plugins</groupId>
> > <artifactId>maven-surefire-plugin</artifactId>
> > <configuration>
> > <forkCount>1.5C</forkCount>
> > <reuseForks>false</reuseForks>
> > <threadCount>1</threadCount>
> > <perCoreThreadCount>false</perCoreThreadCount>
> > <parallel>classes</parallel>
> > <argLine>-Xmx512m</argLine>
> > <testFailureIgnore>false</testFailureIgnore>
> > <redirectTestOutputToFile>true</redirectTestOutputToFile>
> > <systemPropertyVariables>
> >
> > <mahout.test.directory>${project.build.directory}</mahout.test.directory>
> > </systemPropertyVariables>
> > </configuration>
> > </plugin>
> >
> >
> > When I use latest surefire 2.16, outcome is even more odd - output that
> any
> > of the tests is running takes longer to render, it seems to be rendered
> > together with info that test has completed, so by looking at build log
> > output it may appear as if tests were not running in parallel even though
> > they did:
> >
> > -------------------------------------------------------
> > T E S T S
> > -------------------------------------------------------
> > Running org.apache.mahout.clustering.streaming.cluster.BallKMeansTest
> > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 59.347
> sec
> > - in org.apache.mahout.clustering.streaming.cluster.BallKMeansTest
> > Running
> org.apache.mahout.clustering.streaming.cluster.StreamingKMeansTest
> > Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 74.766
> sec
> > - in org.apache.mahout.clustering.streaming.cluster.StreamingKMeansTest
> >
> > Results :
> >
> > Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> >
> >
> > Here is my "mvn --version" output:
> >
> > [sslavic@laptop core]$ mvn --version
> > Apache Maven 3.1.0 (893ca28a1da9d5f51ac03827af98bb730128f9f2; 2013-06-28
> > 04:15:32+0200)
> > Maven home: /home/sslavic/work/tools/apache-maven
> > Java version: 1.7.0_25, vendor: Oracle Corporation
> > Java home: /usr/java/jdk1.7.0_25/jre
> > Default locale: en_US, platform encoding: UTF-8
> > OS name: "linux", version: "3.10.9-200.fc19.x86_64", arch: "amd64",
> family:
> > "unix"
> >
> >
> >
> > Am I doing something wrong or is this behavior bug (or two bugs) in
> > surefire?
> >
> > Kind regards,
> > Stevo Slavic.
> >
>
Re: [surefire] Combining test, runOrder and forkCount
Posted by Andreas Gudian <an...@gmail.com>.
I'll have to check the code, but I'm pretty sure that the order of the test
classes to be handed over to the forked VMs is supposed to be what was is
specified with runOrder.
However, using -Dtest= might disable the ordering - but again, I'd have to
check the code.
Bit there's more: when having two classes to test and two forked VMs, there
is no real way to tell which test comes first. Both VMs are started right
away and only when the are up and running and ready to execute a test, they
ask the main process for one test class. Which VM will be the first to get
a class is quite random.
For the delayed reporting to console, try removing parallel=classes from
the config. IThe classes-setting has no effect when forking to more than
one fork anyway. If the output is still delayed then, create a bug-report,
please.
Am Donnerstag, 29. August 2013 schrieb Stevo Slavić :
> Hello Apache Maven community,
>
> To reproduce random failing test issue, I'm trying to run two tests in
> parallel, but start running them in specific order. Problem is that
> runOrder (added via SUREFIRE-614) doesn't seem to be respected, at least
> not when forkCount and/or test is specified as well.
>
> When trying to run two tests from Apache Mahout core module (trunk), I
> cannot get surefire to run them in reverse alphabetical order.
>
> Regardless if I run:
> mvn -Dtest=*KMeansTest -DrunOrder=reversealphabetical test
>
> or
> mvn -Dtest=*KMeansTest -DrunOrder=alphabetical test
>
> the two tests which match the pattern always get run in alphabetical order:
>
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running org.apache.mahout.clustering.streaming.cluster.BallKMeansTest
> Running org.apache.mahout.clustering.streaming.cluster.StreamingKMeansTest
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.706 sec
> - in org.apache.mahout.clustering.streaming.cluster.BallKMeansTest
> Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 82.15 sec -
> in org.apache.mahout.clustering.streaming.cluster.StreamingKMeansTest
>
> Results :
>
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
>
>
>
>
> Surefire 2.15 is used, and configuration is:
>
>
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-surefire-plugin</artifactId>
> <configuration>
> <forkCount>1.5C</forkCount>
> <reuseForks>false</reuseForks>
> <threadCount>1</threadCount>
> <perCoreThreadCount>false</perCoreThreadCount>
> <parallel>classes</parallel>
> <argLine>-Xmx512m</argLine>
> <testFailureIgnore>false</testFailureIgnore>
> <redirectTestOutputToFile>true</redirectTestOutputToFile>
> <systemPropertyVariables>
>
> <mahout.test.directory>${project.build.directory}</mahout.test.directory>
> </systemPropertyVariables>
> </configuration>
> </plugin>
>
>
> When I use latest surefire 2.16, outcome is even more odd - output that any
> of the tests is running takes longer to render, it seems to be rendered
> together with info that test has completed, so by looking at build log
> output it may appear as if tests were not running in parallel even though
> they did:
>
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running org.apache.mahout.clustering.streaming.cluster.BallKMeansTest
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 59.347 sec
> - in org.apache.mahout.clustering.streaming.cluster.BallKMeansTest
> Running org.apache.mahout.clustering.streaming.cluster.StreamingKMeansTest
> Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 74.766 sec
> - in org.apache.mahout.clustering.streaming.cluster.StreamingKMeansTest
>
> Results :
>
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
>
>
> Here is my "mvn --version" output:
>
> [sslavic@laptop core]$ mvn --version
> Apache Maven 3.1.0 (893ca28a1da9d5f51ac03827af98bb730128f9f2; 2013-06-28
> 04:15:32+0200)
> Maven home: /home/sslavic/work/tools/apache-maven
> Java version: 1.7.0_25, vendor: Oracle Corporation
> Java home: /usr/java/jdk1.7.0_25/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "3.10.9-200.fc19.x86_64", arch: "amd64", family:
> "unix"
>
>
>
> Am I doing something wrong or is this behavior bug (or two bugs) in
> surefire?
>
> Kind regards,
> Stevo Slavic.
>