You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by Kristian Rosenvold <kr...@gmail.com> on 2012/01/29 18:56:56 UTC
Surefire 2.11/2.12 and runOrder=balanced
It was suggested that I was less than clear about the
runOrder=balanced parameter, so here goes:
Based on data from pervious runs, this setting optimizes the runorder
of the tests to minimize total run-time.
Given for tests:
A 7 minutes
B 5 minutes
C 1 minute
D 1 minute
Previously you'd be *required* to have 4 threads to get the minimum
guaranteed run-time, which is 7 minutes.
Both 2 and 3 threads had a worst-case run-time behaviour of 12 minutes.
With surefire 2.11 you'd be guaranteed 7 minutes total run-time for
threads=2 (parallel=classes).
For 2.12 you get the same guarantee with forkMode=perthread and threadCount=2.
Hope that makes it clear ;)
Kristian
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Surefire 2.11/2.12 and runOrder=balanced
Posted by Kristian Rosenvold <kr...@gmail.com>.
Definitely on-topic, I hope you'll find the time some day to make a patch.
Kristian
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Surefire 2.11/2.12 and runOrder=balanced
Posted by Dawid Weiss <da...@gmail.com>.
That piece of code was implemented for ANT and only later ported into
a Maven plugin/ mojo. I have no doubt that taking the code directly
won't work because of architectural differences between the projects
and like most people I too have too much work and too little time so
delving into surefire's internals is not currently possible for me. I
only mentioned what I know from experience because I thought it would
be relevant and on-topic.
Dawid
On Sun, Jan 29, 2012 at 8:52 PM, Kristian Rosenvold
<kr...@gmail.com> wrote:
> If you want to submit any of your stuff as patches I can assure you it
> will be reviewed,
> patches w/tests in surefire get processed quickly.
>
> Kristian
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Surefire 2.11/2.12 and runOrder=balanced
Posted by Kristian Rosenvold <kr...@gmail.com>.
If you want to submit any of your stuff as patches I can assure you it
will be reviewed,
patches w/tests in surefire get processed quickly.
Kristian
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Surefire 2.11/2.12 and runOrder=balanced
Posted by Dawid Weiss <da...@gmail.com>.
> We don't have the two-way comms channel between the plugin and the fork in
> surefire which means it's all precalculated.
Yes, I know.
> There are certainly benefits/drawbacks
> to both approaches. My personal use-case involves extremely long-running tests and of
> extreme variation in run-time, which seems to fit better with the implementation
> in surefire at the moment.
I will disagree with you here. The project I mentioned was originally
prepared for Apache Lucene and the tests in there are randomized and
component-randomized so their execution time can vary by order of
magnitude. I know for a fact that static statistics (even calculated
for many runs) are not so good at scheduling even deadline for all
slave JVMs because I observed this in practice. These stats are very
useful for bootstrapping the process (scheduling initially longest
tests first, for example) but they are not working for static
scheduling in a dynamically changing situation.
Don't get me wrong, I still think it's great for surefire to have
static balancing, but it would be nice if dynamic balancing appeared
some time in the future. :)
> It's a fairly simple patch to write up, just needs a lot of tests ;)
Yes, it does take a fair bit of testing, especially jvm crashing is
tough because jvms emit logs to communication channels regardless of
the content of System.out/err. We switch communication channels
depending on jvm type detected at runtime, actually. The tests are in
the project as well.
Dawid
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Surefire 2.11/2.12 and runOrder=balanced
Posted by Kristian Rosenvold <kr...@gmail.com>.
We don't have the two-way comms channel between the plugin and the fork in
surefire which means it's all precalculated. There are certainly
benefits/drawbacks
to both approaches. My personal use-case involves extremely
long-running tests and of
extreme variation in run-time, which seems to fit better with the implementation
in surefire at the moment.
The code to make the comms channel 2 way is already written, and making reusable
forks is one of the few remaining things I'd like to add to surefire.
But I'm not sure I'm
going to be prioritizing that much further, there's so many other things that
need to be done ;)
It's a fairly simple patch to write up, just needs a lot of tests ;)
Kristian
2012/1/29 Dawid Weiss <da...@gmail.com>:
>> Based on data from pervious runs, this setting optimizes the runorder
>> of the tests to minimize total run-time.
>
> This is a digression, but I think may be inspirational.
>
> Kristian's description matches how I implemented per-jvm balancing in
> our JUnit4 runner initially and I have a suggestion on how to make it
> better. In practice a similar near-optimal balancing can be achieved
> by doing initial balancing of, say, 50% of all suites, followed by
> serving any of the remaining test suites to any idle slave JVM (much
> like job stealing in fork-join). While it may happen that the longest
> test will be scheduled last, it is unlikely in practice (given a lot
> of tests and their permutations). Also, if history of runs is
> available, the above behavior will still work and even provide a
> shield against tests that have a large variance on execution time
> (when static scheduling may be incorrect).
>
> We did implement a Maven plugin for running JUnit (4.x) tests that
> complements (or replaces, if one wishes to do so) Surefire so you can
> compare both approaches. The code to do the above trick does add some
> complexity because it requires master-slave communication to take
> place, but this is relatively easy to do and provides other benefits
> (and we needed it anyway to provide consistent reporting over slave
> JVMs). Grab anything you want, the project is in our labs and it is
> Apache licensed:
>
> http://labs.carrotsearch.com/randomizedtesting.html
>
> Load balancing is just part of what the project is about; the code I
> mentioned above is mostly around here:
> https://github.com/carrotsearch/randomizedtesting/blob/master/integration-ant/ant-junit4/src/main/java/com/carrotsearch/ant/tasks/junit4/JUnit4.java#L472
>
> Maven integration can be seen as an integration test (with scarce
> documentation yet) here:
> https://github.com/carrotsearch/randomizedtesting/blob/master/integration-maven/junit4-maven-plugin-tests/src/it/01-basic-test/pom.xml
>
> I've used it in another project, so a cleaner example of use from
> within a POM is here (you should disable surefire or your tests will
> run twice):
> https://github.com/carrotsearch/hppc/blob/master/hppc-core/pom.xml#L217
>
> Dawid
>
>>
>> Given for tests:
>>
>> A 7 minutes
>> B 5 minutes
>> C 1 minute
>> D 1 minute
>>
>> Previously you'd be *required* to have 4 threads to get the minimum
>> guaranteed run-time, which is 7 minutes.
>> Both 2 and 3 threads had a worst-case run-time behaviour of 12 minutes.
>>
>> With surefire 2.11 you'd be guaranteed 7 minutes total run-time for
>> threads=2 (parallel=classes).
>> For 2.12 you get the same guarantee with forkMode=perthread and threadCount=2.
>>
>> Hope that makes it clear ;)
>>
>> Kristian
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Surefire 2.11/2.12 and runOrder=balanced
Posted by Dawid Weiss <da...@gmail.com>.
> Based on data from pervious runs, this setting optimizes the runorder
> of the tests to minimize total run-time.
This is a digression, but I think may be inspirational.
Kristian's description matches how I implemented per-jvm balancing in
our JUnit4 runner initially and I have a suggestion on how to make it
better. In practice a similar near-optimal balancing can be achieved
by doing initial balancing of, say, 50% of all suites, followed by
serving any of the remaining test suites to any idle slave JVM (much
like job stealing in fork-join). While it may happen that the longest
test will be scheduled last, it is unlikely in practice (given a lot
of tests and their permutations). Also, if history of runs is
available, the above behavior will still work and even provide a
shield against tests that have a large variance on execution time
(when static scheduling may be incorrect).
We did implement a Maven plugin for running JUnit (4.x) tests that
complements (or replaces, if one wishes to do so) Surefire so you can
compare both approaches. The code to do the above trick does add some
complexity because it requires master-slave communication to take
place, but this is relatively easy to do and provides other benefits
(and we needed it anyway to provide consistent reporting over slave
JVMs). Grab anything you want, the project is in our labs and it is
Apache licensed:
http://labs.carrotsearch.com/randomizedtesting.html
Load balancing is just part of what the project is about; the code I
mentioned above is mostly around here:
https://github.com/carrotsearch/randomizedtesting/blob/master/integration-ant/ant-junit4/src/main/java/com/carrotsearch/ant/tasks/junit4/JUnit4.java#L472
Maven integration can be seen as an integration test (with scarce
documentation yet) here:
https://github.com/carrotsearch/randomizedtesting/blob/master/integration-maven/junit4-maven-plugin-tests/src/it/01-basic-test/pom.xml
I've used it in another project, so a cleaner example of use from
within a POM is here (you should disable surefire or your tests will
run twice):
https://github.com/carrotsearch/hppc/blob/master/hppc-core/pom.xml#L217
Dawid
>
> Given for tests:
>
> A 7 minutes
> B 5 minutes
> C 1 minute
> D 1 minute
>
> Previously you'd be *required* to have 4 threads to get the minimum
> guaranteed run-time, which is 7 minutes.
> Both 2 and 3 threads had a worst-case run-time behaviour of 12 minutes.
>
> With surefire 2.11 you'd be guaranteed 7 minutes total run-time for
> threads=2 (parallel=classes).
> For 2.12 you get the same guarantee with forkMode=perthread and threadCount=2.
>
> Hope that makes it clear ;)
>
> Kristian
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org