How are tests divided among multiple JVMs?

[Shawn Heisey <so...@elyograg.org>]

I was running Lucene/Solr tests on my dev server, including nightly and 
weekly.  For my hardware, the build system automatically decides that 3 
JVMs should be used.

I noticed that for one bunch of tests (probably lucene/core), J1 was the 
only JVM that was actually running tests - J0 and J2 were completely 
silent, not even outputting heartbeats.  After a while, I guessed that 
what happened was that all the test suites were divided among the JVMs 
at the beginning, J0 and J2 had executed all their tests, and J1 was not 
finished making its way through its list of tests.

Is this how it works, and if so, is it a major undertaking to have it 
assign tests to JVMs as each suite is completed, rather than at the 
beginning?

Thanks,
Shawn

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: How are tests divided among multiple JVMs?

[Dawid Weiss <da...@cs.put.poznan.pl>]

I filed this issue so that the test execution statistics from a local
machine could be reused:

https://issues.apache.org/jira/browse/LUCENE-4654

Dawid

On Thu, Jan 3, 2013 at 8:45 AM, Dawid Weiss
<da...@cs.put.poznan.pl> wrote:
>> Is this how it works, and if so,
>
> Yes and no.
>
> First, test classes are scattered pseudo-randomly (depending on the
> master seed) across forked JVMs. This is as to ensure repeatability in
> case of test-suite-interdependencies (like one class leaving something
> in a static field and another class depending on it) this order needs
> to be preserved between runs.
>
> Second, the allocation of test classes to forked JVMs is not done at
> random; there is a greedy heuristic that tries to level the load on
> all forked JVMs. These statistics, however, are computed and cached in
> advance so they'll be more and more off target as time passes,
> depending on the different (random) configuration of selected
> components, nightly/ weekly switches etc, particular hardware
> configuration, etc. You could try to recompute up-to-date stats for
> your machine:
>
> # Update global (versioned!) execution times cache (top level).
> ant clean test
> ant -f lucene/build.xml test-updatecache
>
> and then the balancing should improve. Read on.
>
>> is it a major undertaking to have it assign
>> tests to JVMs as each suite is completed, rather than at the beginning?
>
> This feature is actually already in the code, but (for reasons above)
> not active by default. If you want you can run in job-stealing mode
> where suite classes are dispatched to forked JVMs as they become
> available. When you run 'ant test-help' from top-level Solr/Lucene
> checkout there is this hint which is of relevance:
>
> # Use repeatable suite order on slave JVMs (disables job stealing).
> ant -Dtests.dynamicAssignmentRatio=0 test
>
> this means "fully preallocated assignment". To have exactly the
> opposite (no preallocated assignment) you'd do:
>
> ant -Dtests.dynamicAssignmentRatio=1 test
>
> Anything in between is partially preallocated queue with a tail
> allocated on demand. Knowing this, if you inspect Lucene build file
> sources you'll notice that the value of this parameter is:
>
>         <property name="tests.dynamicAssignmentRatio" value="0.50" />
> <!-- 50% of suites -->
>
> So what I said at the beginning is not true -- we actually preallocate
> only 50% of the suite classes to forked JVMs, the rest is dispatched
> on demand. Like I said though, the allocation stats may be so off
> target that in your case this is not enough. Just  set the assignment
> ratio to '1'. Note that even then the last JVM may be given a long
> suite so the execution time won't be "even".
>
> Final note. If you're an algorithmic freak and think the allocation
> model should use an optimal heuristic (greedy allocation of large
> tasks first) then I can tell you in advance that this is a wrong
> approach here. The first reason is that we want to have a uniform
> sample from the set of  permutations of all test suites running on
> forked JVMs so that we can detect suite-dependencies early. The second
> reason is more mundane --
> in practice if we feed 4 large, heavy test suites to forked JVMs at
> the same time they start to compete for resources (i/o, memory bus)
> and in the end the execution lasts longer compared to the random
> version.
>
> Dawid

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: How are tests divided among multiple JVMs?

[Dawid Weiss <da...@cs.put.poznan.pl>]

> Is this how it works, and if so,

Yes and no.

First, test classes are scattered pseudo-randomly (depending on the
master seed) across forked JVMs. This is as to ensure repeatability in
case of test-suite-interdependencies (like one class leaving something
in a static field and another class depending on it) this order needs
to be preserved between runs.

Second, the allocation of test classes to forked JVMs is not done at
random; there is a greedy heuristic that tries to level the load on
all forked JVMs. These statistics, however, are computed and cached in
advance so they'll be more and more off target as time passes,
depending on the different (random) configuration of selected
components, nightly/ weekly switches etc, particular hardware
configuration, etc. You could try to recompute up-to-date stats for
your machine:

# Update global (versioned!) execution times cache (top level).
ant clean test
ant -f lucene/build.xml test-updatecache

and then the balancing should improve. Read on.

> is it a major undertaking to have it assign
> tests to JVMs as each suite is completed, rather than at the beginning?

This feature is actually already in the code, but (for reasons above)
not active by default. If you want you can run in job-stealing mode
where suite classes are dispatched to forked JVMs as they become
available. When you run 'ant test-help' from top-level Solr/Lucene
checkout there is this hint which is of relevance:

# Use repeatable suite order on slave JVMs (disables job stealing).
ant -Dtests.dynamicAssignmentRatio=0 test

this means "fully preallocated assignment". To have exactly the
opposite (no preallocated assignment) you'd do:

ant -Dtests.dynamicAssignmentRatio=1 test

Anything in between is partially preallocated queue with a tail
allocated on demand. Knowing this, if you inspect Lucene build file
sources you'll notice that the value of this parameter is:

        <property name="tests.dynamicAssignmentRatio" value="0.50" />
<!-- 50% of suites -->

So what I said at the beginning is not true -- we actually preallocate
only 50% of the suite classes to forked JVMs, the rest is dispatched
on demand. Like I said though, the allocation stats may be so off
target that in your case this is not enough. Just  set the assignment
ratio to '1'. Note that even then the last JVM may be given a long
suite so the execution time won't be "even".

Final note. If you're an algorithmic freak and think the allocation
model should use an optimal heuristic (greedy allocation of large
tasks first) then I can tell you in advance that this is a wrong
approach here. The first reason is that we want to have a uniform
sample from the set of  permutations of all test suites running on
forked JVMs so that we can detect suite-dependencies early. The second
reason is more mundane --
in practice if we feed 4 large, heavy test suites to forked JVMs at
the same time they start to compete for resources (i/o, memory bus)
and in the end the execution lasts longer compared to the random
version.

Dawid

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: How are tests divided among multiple JVMs?

[Robert Muir <rc...@gmail.com>]

On Wed, Jan 2, 2013 at 6:28 PM, Shawn Heisey <so...@elyograg.org> wrote:
> I was running Lucene/Solr tests on my dev server, including nightly and
> weekly.

if you do this, lucene tests will be unbalanced. because when you pass
things like nightly then lots of really slow tests (Test2BPostings,
Test4GBStoredFields, ...) run. Also other tests like TestFSTs that
normally take a few seconds will take minutes (on purpose).

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org