squeezing some multi-processing out of the nightly test runs?

[Mike Matrigali <mi...@sbcglobal.net>]

I would like to get the nightly tests to run faster on machines that
have more than one cpu - or even hyperthreaded on a single cpu.  Since 
most of the tests are single threaded I think this means somehow running 
more than one test at a time.  For me
it might even help if I was just able to run tests on 2 different
codelines on the same machine at the same time.
I think going forward this will be more and
more common for developers.  You can find reasonably priced laptops
nowadays that come dual core already.

I know what outstanding issue is that the network tests default to the
same port.  I believe at least for the junit tests one can set a 
different port number.  Does anyone know if this fixes all the issues
for the junit tests - I think I have seen some tests try different ports
rather than the default one.

Also what is the situation with the derbyall tests.  How much work to 
get the rest of the network server tests converted to junit.

On a separate note how hard would it be to get a test runner to execute
more than one test (fixture? not sure of the junit terminology),
at a time.  I don't think we could execute them in the same db as most
seem to assume they can use any name space and then clean up the whole
db when they are done.  From those knowledgeable about junit, should I
look for a test runner with these features or is this just a programming
job for derby specific tests?  Maybe just a different set of suites?

Re: squeezing some multi-processing out of the nightly test runs?

[Knut Anders Hatlen <Kn...@Sun.COM>]

Kristian Waagan <Kr...@Sun.COM> writes:

> Knut Anders Hatlen wrote:
>> Rick Hillegas <Ri...@Sun.COM> writes:
>>
>>> Mike Matrigali wrote:
>>>> I would like to get the nightly tests to run faster on machines that
>>>> have more than one cpu - or even hyperthreaded on a single cpu.
>>>> Since most of the tests are single threaded I think this means
>>>> somehow running more than one test at a time.  For me
>>>> it might even help if I was just able to run tests on 2 different
>>>> codelines on the same machine at the same time.
>>>> I think going forward this will be more and
>>>> more common for developers.  You can find reasonably priced laptops
>>>> nowadays that come dual core already.
>>>>
>>>> I know what outstanding issue is that the network tests default to the
>>>> same port.  I believe at least for the junit tests one can set a
>>>> different port number.  Does anyone know if this fixes all the
>>>> issues
>>>> for the junit tests - I think I have seen some tests try different ports
>>>> rather than the default one.
>>> Hi Mike,
>>>
>>> I don't know how much work is needed to make the junit tests
>>> independent of the port number. I can't find a JIRA which lists all of
>>> the problem cases. However, the following JIRAs may be helpful:
>>> DERBY-2440 and DERBY-2404.
>>
>> It is possible to change the port for all tests in a JUnit suite if we
>> run the suite like this:
>>
>>   java -Dport=XXX -DjmxPort=YYY ...
>
> Hi Knut Anders,
>
> Have you tested this?
> The last time I looked at it, the relevant code was only invoked for
> test runs in the DERBY_HARNESS_CONFIG configuration, which I believe
> is some kind of backward compatibility mode with respect to the old
> harness.
> It seems this mode can be triggered by various things, for instance if
> derby.system.home is specified, so people might see different behavior
> depending on how they start the tests.
>
> I also see that that some methods are using the constant DEFAULT_PORT
> directly, which suggests we are not quite there yet...

Yes, I tested it, but only after I sent the mail. :)

I saw the same behaviour as you describe. Both -Dport=XXX and
-DjmxPort=YYY only work if the JUnit test are run from the old harness.

It also looks like the replication tests have port 1527 hard-coded.

> I find this code, or at least the naming and the comments, a tad
> confusing. I would like to remove the notion of the old harness, and
> can do another pass and see if we can remove it now.

I don't think we have any JUnit tests in the old harness any more
(except the compatibility test, but that one doesn't use
TestConfiguration), so removing the notion of the old harness sounds
like a good idea to me.

-- 
Knut Anders

Re: squeezing some multi-processing out of the nightly test runs?

[Kristian Waagan <Kr...@Sun.COM>]

Knut Anders Hatlen wrote:
> Rick Hillegas <Ri...@Sun.COM> writes:
> 
>> Mike Matrigali wrote:
>>> I would like to get the nightly tests to run faster on machines that
>>> have more than one cpu - or even hyperthreaded on a single cpu.
>>> Since most of the tests are single threaded I think this means
>>> somehow running more than one test at a time.  For me
>>> it might even help if I was just able to run tests on 2 different
>>> codelines on the same machine at the same time.
>>> I think going forward this will be more and
>>> more common for developers.  You can find reasonably priced laptops
>>> nowadays that come dual core already.
>>>
>>> I know what outstanding issue is that the network tests default to the
>>> same port.  I believe at least for the junit tests one can set a
>>> different port number.  Does anyone know if this fixes all the
>>> issues
>>> for the junit tests - I think I have seen some tests try different ports
>>> rather than the default one.
>> Hi Mike,
>>
>> I don't know how much work is needed to make the junit tests
>> independent of the port number. I can't find a JIRA which lists all of
>> the problem cases. However, the following JIRAs may be helpful:
>> DERBY-2440 and DERBY-2404.
> 
> It is possible to change the port for all tests in a JUnit suite if we
> run the suite like this:
> 
>   java -Dport=XXX -DjmxPort=YYY ...

Hi Knut Anders,

Have you tested this?
The last time I looked at it, the relevant code was only invoked for 
test runs in the DERBY_HARNESS_CONFIG configuration, which I believe is 
some kind of backward compatibility mode with respect to the old harness.
It seems this mode can be triggered by various things, for instance if 
derby.system.home is specified, so people might see different behavior 
depending on how they start the tests.

I also see that that some methods are using the constant DEFAULT_PORT 
directly, which suggests we are not quite there yet...

I find this code, or at least the naming and the comments, a tad 
confusing. I would like to remove the notion of the old harness, and can 
do another pass and see if we can remove it now.



-- 
Kristian

[ snip - parallelization ideas ]

Re: squeezing some multi-processing out of the nightly test runs?

[Knut Anders Hatlen <Kn...@Sun.COM>]

Rick Hillegas <Ri...@Sun.COM> writes:

> Mike Matrigali wrote:
>> I would like to get the nightly tests to run faster on machines that
>> have more than one cpu - or even hyperthreaded on a single cpu.
>> Since most of the tests are single threaded I think this means
>> somehow running more than one test at a time.  For me
>> it might even help if I was just able to run tests on 2 different
>> codelines on the same machine at the same time.
>> I think going forward this will be more and
>> more common for developers.  You can find reasonably priced laptops
>> nowadays that come dual core already.
>>
>> I know what outstanding issue is that the network tests default to the
>> same port.  I believe at least for the junit tests one can set a
>> different port number.  Does anyone know if this fixes all the
>> issues
>> for the junit tests - I think I have seen some tests try different ports
>> rather than the default one.
> Hi Mike,
>
> I don't know how much work is needed to make the junit tests
> independent of the port number. I can't find a JIRA which lists all of
> the problem cases. However, the following JIRAs may be helpful:
> DERBY-2440 and DERBY-2404.

It is possible to change the port for all tests in a JUnit suite if we
run the suite like this:

  java -Dport=XXX -DjmxPort=YYY ...

Note that some tests also use port XXX+1, so you shouldn't allocate
ports to close to each other if you run many JUnit processes in
parallel.

One thing I've considered, is to have a shell script that runs each
top-level test suite in a separate process using different ports. To
make such a script more maintainable (for instance, I don't like the
thought of having to update the script each time a new suite is added) I
was thinking that we could change the AllPackages class so that it
prints the list of test suites if we execute the class from the command
line. Then we could have a shell script which did something like this:

port=1527
for i in $(java org.apache.derbyTesting.functionTests.suites.AllPackages)
do
  java junit.textui.TestRunner -Dport=$port -DjmxPort=$((port+5)) $i &
  ((port += 10))
done

>> Also what is the situation with the derbyall tests.  How much work
>> to get the rest of the network server tests converted to junit.
>>
>> On a separate note how hard would it be to get a test runner to execute
>> more than one test (fixture? not sure of the junit terminology),
>> at a time.  I don't think we could execute them in the same db as most
>> seem to assume they can use any name space and then clean up the whole
>> db when they are done.  From those knowledgeable about junit, should I
>> look for a test runner with these features or is this just a programming
>> job for derby specific tests?  Maybe just a different set of suites?
> I haven't experimented with this myself, but the following link might
> be useful: https://parallel-junit.dev.java.net/

That looks interesting! We could perhaps have a helper method that
creates a ParallelTestSuite, if that class is available, or an ordinary
TestSuite if the class isn't available. As a first step to increase the
parallelism, we could make TestConfiguration.defaultSuite() run the
embedded test suite in parallel with the client/server test suite. Then
we don't need to worry about port collisions, we only need to make sure
the two threads use different databases.

Many of the existing tests can probably be parallelized even further
with that library without too much effort, but there are also many that
need to be executed sequentially because they work on the same set of
tables and require that the tearDown() method of the preceding test case
has done the proper clean-up. Some of the tests also reboot the
database, which could affect tests running in parallel against the same
database.

-- 
Knut Anders

Re: squeezing some multi-processing out of the nightly test runs?

["Dag H. Wanvik" <Da...@Sun.COM>]

Rick Hillegas <Ri...@Sun.COM> writes:

> I don't know how much work is needed to make the junit tests
> independent of the port number. I can't find a JIRA which lists all of
> the problem cases. However, the following JIRAs may be helpful:
> DERBY-2440 and DERBY-2404.

When I last looked at this (DERBY-2440), what was missing for
suites.All was just a way to specify at runtime which default port to
use (instead of using the default in
TestConfiguration#DEFAULT_PORT. But new tests have been added since
then, and those may or may not heed TestConfiguration#DEFAULT_PORT.

Re: squeezing some multi-processing out of the nightly test runs?

[Rick Hillegas <Ri...@Sun.COM>]

Mike Matrigali wrote:
> I would like to get the nightly tests to run faster on machines that
> have more than one cpu - or even hyperthreaded on a single cpu.  Since 
> most of the tests are single threaded I think this means somehow 
> running more than one test at a time.  For me
> it might even help if I was just able to run tests on 2 different
> codelines on the same machine at the same time.
> I think going forward this will be more and
> more common for developers.  You can find reasonably priced laptops
> nowadays that come dual core already.
>
> I know what outstanding issue is that the network tests default to the
> same port.  I believe at least for the junit tests one can set a 
> different port number.  Does anyone know if this fixes all the issues
> for the junit tests - I think I have seen some tests try different ports
> rather than the default one.
Hi Mike,

I don't know how much work is needed to make the junit tests independent 
of the port number. I can't find a JIRA which lists all of the problem 
cases. However, the following JIRAs may be helpful: DERBY-2440 and 
DERBY-2404.
>
> Also what is the situation with the derbyall tests.  How much work to 
> get the rest of the network server tests converted to junit.
>
> On a separate note how hard would it be to get a test runner to execute
> more than one test (fixture? not sure of the junit terminology),
> at a time.  I don't think we could execute them in the same db as most
> seem to assume they can use any name space and then clean up the whole
> db when they are done.  From those knowledgeable about junit, should I
> look for a test runner with these features or is this just a programming
> job for derby specific tests?  Maybe just a different set of suites?
I haven't experimented with this myself, but the following link might be 
useful: https://parallel-junit.dev.java.net/

Thanks for looking into this,
-Rick