You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by Keith Wall <ke...@gmail.com> on 2011/07/04 12:44:42 UTC
Ideas to rationalise the Java test profiles.
Robbie and I are presently working on QPID-2811 and QPID-2815. These Jiras
will rationalise the transport mechanism in Qpid and to allow client/broker
to be tested together within the same JVM regardless of transport.
It seems to me that now is a good time to review the test-profiles. I see
the following problems:
1) 'default' profile being 0-8 makes little sense as most of the project's
development focus is on 0-10, and with the removal of the old .NET client,
we have no client that defaults to 0-8.
2) The time taken to run some test profiles is excessive and this is a
disincentive to developer good practice. For instance, running java-0.10 on
my machine takes 30mins. This is harming productivity.
Current profiles:
default.testprofile - tests 0-8 using in-VM, broker/client share JVM,
Memory Message Store
java.testprofile - tests 0-9-1, using tcp, separate JVMs, Memory Message
Store
java-derby.testprofile - tests 0-9-1, using tcp, separate JVMs, Derby
Message Store
java.0.10.testprofile - tests 0-10, using tcp, separate JVMs,, Memory
Message Store
java-derby.0.10.testprofile -- tests 0-10, using tcp, separate JVMs, Derby
Message Store
My proposal:
Four profiles intended for regular developer use. These profiles will
exploit the feature for client/broker to share the same VM. This saves the
overhead of each testcase spawning a new broker as a separate process, and
brings the length of time to run the profile down considerably, assisting
developer productivity.
(mms=memory message store, dby = Apache Derby)
java-mms.0-10
java-dby.0-10
java-mms.0-9-1
java-dby.0-9-1
Running client/broker in separate JVMs remains valuable, but is probably not
something most developers will want to do most of the time (because of the
time overhead). The intention would be we'd schedule these to run on CI
(using Jenkin's matrix feature) and let Jenkins do the heavy lifting.
java-mms-spawn.0-10
java-dby-spawn.0-10
java-mms-spawn.0-9-1
java-dby-spawn.0-9-1
The idea of the default testprofile would disappear and it would become an
error to run -Dprofile=default. If the developer omits -Dprofile from the
Ant command line, the system will run java-mms.0-10, which I think will be
useful fallback to most developers.
I think we also need a "Java Test Howto" page on the website, explaining the
purposes behind the profiles, the differences between system and non-system
tests and giving advice on appropriate use of both.
Any thoughts?
Keith.
Re: Ideas to rationalise the Java test profiles.
Posted by "Weston M. Price" <wp...@redhat.com>.
On Jul 5, 2011, at 5:22 PM, Keith Wall wrote:
> Hi Rajith
>
>> 1. Coverage
>> We currently have close to 1200+ test cases and my impression is that we
> still have quite a few gaps. All though it's painful maybe we should try to
> get an inventory of our existing test cases.
>
I have started work on this as one of my tasks and would be more than happy to work with someone.
> I think the best form of inventory would be the tests themselves. The test
> methods need clear meaningful names and, if the test has unavoidably
> complexity, supporting method level Javadoc. I'd worry that a separate
> test inventory would get quickly out of date.
>
> I agree we have many gaps. I'd like to get coverage reports turned on
> Jenkins, perhaps on a nightly schedule. I realise coverage alone is no
> silver bullet, but it can be a useful pointer to areas of weakness.
>
+1 to both comments.
>> 2. Manageability/Maintainability of test suites As Robbie pointed out we
> have a lot of low utility tests. these tests just bloat our test suite but
> add very little value. We need to get out of the habit of adding a new test
> case for every commit. I personally think we have too many tests.
>
> I'd say our main problem is an over-reliance on system tests. We have cases
> where we have exhaustively exercised _every_ feature of a use-case from a
> system test. This approach makes for a slow test suite that offers the
> developer little support when refactoring (the test failure is often to
> granular to be useful). I think a better approach is to rely on unit
> test(s) to drive out of the behaviour, then back these up with a smaller
> number of judicious system tests.
>
+1
>> 3. System tests We need to make a distinction between client tests and
> broker tests
>
> Yes, agree. I'm suggesting we add a Java Testing Howto to the website. This
> can help make the distinction between unit/system tests and help our
> contributors submit useful test cases with their patches.
>
>> and a developer should have the flexibility of running either or both.
>
> I think we already have this. The -Dtest=xx operand can be an Ant glob.
>
> For instance:
> ant -f build.xml test -Dtest='org/apache/qpid/server/**/*'
> will run all the tests in package org.apache.qpid.server downwards,
> regardless of depth
> Of course, if the package structure in the test tree made sense, it would be
> even more useful.
>
> kind regards Keith.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: Ideas to rationalise the Java test profiles.
Posted by Keith Wall <ke...@gmail.com>.
Hi Rajith
> 1. Coverage
> We currently have close to 1200+ test cases and my impression is that we
still have quite a few gaps. All though it's painful maybe we should try to
get an inventory of our existing test cases.
I think the best form of inventory would be the tests themselves. The test
methods need clear meaningful names and, if the test has unavoidably
complexity, supporting method level Javadoc. I'd worry that a separate
test inventory would get quickly out of date.
I agree we have many gaps. I'd like to get coverage reports turned on
Jenkins, perhaps on a nightly schedule. I realise coverage alone is no
silver bullet, but it can be a useful pointer to areas of weakness.
> 2. Manageability/Maintainability of test suites As Robbie pointed out we
have a lot of low utility tests. these tests just bloat our test suite but
add very little value. We need to get out of the habit of adding a new test
case for every commit. I personally think we have too many tests.
I'd say our main problem is an over-reliance on system tests. We have cases
where we have exhaustively exercised _every_ feature of a use-case from a
system test. This approach makes for a slow test suite that offers the
developer little support when refactoring (the test failure is often to
granular to be useful). I think a better approach is to rely on unit
test(s) to drive out of the behaviour, then back these up with a smaller
number of judicious system tests.
> 3. System tests We need to make a distinction between client tests and
broker tests
Yes, agree. I'm suggesting we add a Java Testing Howto to the website. This
can help make the distinction between unit/system tests and help our
contributors submit useful test cases with their patches.
> and a developer should have the flexibility of running either or both.
I think we already have this. The -Dtest=xx operand can be an Ant glob.
For instance:
ant -f build.xml test -Dtest='org/apache/qpid/server/**/*'
will run all the tests in package org.apache.qpid.server downwards,
regardless of depth
Of course, if the package structure in the test tree made sense, it would be
even more useful.
kind regards Keith.
Re: Ideas to rationalise the Java test profiles.
Posted by Rajith Attapattu <ra...@gmail.com>.
Hi Robbie,
I couldn't agree more with your comments !
We definitely need a set of system tests that covers the common
messaging use cases that developers can run before checking in
changes.
We could then rely on a CI instance to cover the various test profiles.
Now that we are on the subject let me take the opportunity to share
some thoughts that I've been meaning to put forth for some time.
1. Coverage
We currently have close to 1200+ test cases and my impression is
that we still have quite a few gaps. All though it's painful maybe we
should try to get an inventory of our existing test cases. I started
this a few times and gave up :(. This will be a lot less painful if we
share this task among several folks :)
2. Manageability/Maintainability of test suites
As Robbie pointed out we have a lot of low utility tests. these
tests just bloat our test suite but add very little value.
We need to get out of the habit of adding a new test case for every
commit. I personally think we have too many tests.
As Robbie mentioned we need to look at building a suite of
meaningful tests that covers real world messaging use cases.
3. System tests
We need to make a distinction between client tests and broker
tests and a developer should have the flexibility of running either or
both.
Broker tests
-----------------
I think going forward we should have a common test suite for the
brokers (both C++ and Java) rather than duplicating and spreading
ourselves think
JMS tests
--------------
Quite a few of our tests are written using implementation specific
AMQ* classes. This is a major hurdle in refactoring the client as we
automatically loose a fair percentage of the tests when we write a new
implementation. If we write highly configurable pure JMS based tests
then we can easily reuse them irrespective of the underlying
implementation strategy.
Qpid API tests
------------------
If and when we come up with the Qpid API equivalent in Java we need
to have tests there as well. But probably we could benefit from a
common test suite across all languages for this. This is easier said
than done, but if we do get it right, it will be a huge win.
Regards,
Rajith.
On Mon, Jul 4, 2011 at 11:45 AM, Robbie Gemmell
<ro...@gmail.com> wrote:
> Hi Rajith,
>
> The changes we are proposing to the Java test profiles only relate to
> the Java broker used during test runs, nothing else will change, and
> as always changes should be tested against the C++ broker.
>
> Going forward, the 30-40min test profiles will pretty much be a thing
> of the past as far as humans are concerned, and we will leave those up
> to the Jenkins box 99% of the time, all the profiles we'd expect
> people to run will complete significantly faster.
>
> The obvious way to improve up our testing situation is to stop using
> system tests where unit tests are more appropriate, i.e the vast
> majority of cases. A lot of our current system tests are of low
> utility and would be better implemented as unit tests. We should
> probably also look towards implementing a suite of system tests that
> exercise common messaging use cases, and going forward attempt to
> transition as much of the rest into well thought through and
> specifically targeted unit tests.
>
> Robbie
>
> On 4 July 2011 15:39, Rajith Attapattu <ra...@gmail.com> wrote:
>> Just to add to it, any changes to the Java client should also be
>> tested with the cpp test profile.
>> Any changes to the client failover mechanism should also be tested
>> with the cpp.cluster profile.
>>
>> I know it's a pain to run all these profiles, especially considering
>> each run takes about 30-40 mins.
>> But atm there is no better alternative. I think long term we need to
>> evaluate our testing strategy to see if we can come up with something
>> more efficient.
>>
>> Regards,
>>
>> Rajith
>>
>> On Mon, Jul 4, 2011 at 7:11 AM, Robert Godfrey <ro...@gmail.com> wrote:
>>> On 4 July 2011 12:44, Keith Wall <ke...@gmail.com> wrote:
>>>
>>>> Robbie and I are presently working on QPID-2811 and QPID-2815. These Jiras
>>>> will rationalise the transport mechanism in Qpid and to allow client/broker
>>>> to be tested together within the same JVM regardless of transport.
>>>>
>>>> It seems to me that now is a good time to review the test-profiles. I see
>>>> the following problems:
>>>>
>>>> 1) 'default' profile being 0-8 makes little sense as most of the project's
>>>> development focus is on 0-10, and with the removal of the old .NET client,
>>>> we have no client that defaults to 0-8.
>>>> 2) The time taken to run some test profiles is excessive and this is a
>>>> disincentive to developer good practice. For instance, running java-0.10
>>>> on
>>>> my machine takes 30mins. This is harming productivity.
>>>>
>>>> Current profiles:
>>>>
>>>> default.testprofile - tests 0-8 using in-VM, broker/client share JVM,
>>>> Memory Message Store
>>>> java.testprofile - tests 0-9-1, using tcp, separate JVMs, Memory Message
>>>> Store
>>>> java-derby.testprofile - tests 0-9-1, using tcp, separate JVMs, Derby
>>>> Message Store
>>>>
>>>> java.0.10.testprofile - tests 0-10, using tcp, separate JVMs,, Memory
>>>> Message Store
>>>> java-derby.0.10.testprofile -- tests 0-10, using tcp, separate JVMs, Derby
>>>> Message Store
>>>>
>>>> My proposal:
>>>>
>>>> Four profiles intended for regular developer use. These profiles will
>>>> exploit the feature for client/broker to share the same VM. This saves the
>>>> overhead of each testcase spawning a new broker as a separate process, and
>>>> brings the length of time to run the profile down considerably, assisting
>>>> developer productivity.
>>>>
>>>> (mms=memory message store, dby = Apache Derby)
>>>>
>>>> java-mms.0-10
>>>> java-dby.0-10
>>>>
>>>> java-mms.0-9-1
>>>> java-dby.0-9-1
>>>>
>>>> Running client/broker in separate JVMs remains valuable, but is probably
>>>> not
>>>> something most developers will want to do most of the time (because of the
>>>> time overhead). The intention would be we'd schedule these to run on CI
>>>> (using Jenkin's matrix feature) and let Jenkins do the heavy lifting.
>>>>
>>>> java-mms-spawn.0-10
>>>> java-dby-spawn.0-10
>>>>
>>>> java-mms-spawn.0-9-1
>>>> java-dby-spawn.0-9-1
>>>>
>>>> The idea of the default testprofile would disappear and it would become an
>>>> error to run -Dprofile=default. If the developer omits -Dprofile from the
>>>> Ant command line, the system will run java-mms.0-10, which I think will be
>>>> useful fallback to most developers.
>>>>
>>>> I think we also need a "Java Test Howto" page on the website, explaining
>>>> the
>>>> purposes behind the profiles, the differences between system and non-system
>>>> tests and giving advice on appropriate use of both.
>>>>
>>>> Any thoughts?
>>>>
>>>> Keith.
>>>>
>>>
>>>
>>> All sounds exceedingly sensible to me.
>>>
>>> -- Rob
>>>
>>
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project: http://qpid.apache.org
>> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: Ideas to rationalise the Java test profiles.
Posted by Robbie Gemmell <ro...@gmail.com>.
Hi Rajith,
The changes we are proposing to the Java test profiles only relate to
the Java broker used during test runs, nothing else will change, and
as always changes should be tested against the C++ broker.
Going forward, the 30-40min test profiles will pretty much be a thing
of the past as far as humans are concerned, and we will leave those up
to the Jenkins box 99% of the time, all the profiles we'd expect
people to run will complete significantly faster.
The obvious way to improve up our testing situation is to stop using
system tests where unit tests are more appropriate, i.e the vast
majority of cases. A lot of our current system tests are of low
utility and would be better implemented as unit tests. We should
probably also look towards implementing a suite of system tests that
exercise common messaging use cases, and going forward attempt to
transition as much of the rest into well thought through and
specifically targeted unit tests.
Robbie
On 4 July 2011 15:39, Rajith Attapattu <ra...@gmail.com> wrote:
> Just to add to it, any changes to the Java client should also be
> tested with the cpp test profile.
> Any changes to the client failover mechanism should also be tested
> with the cpp.cluster profile.
>
> I know it's a pain to run all these profiles, especially considering
> each run takes about 30-40 mins.
> But atm there is no better alternative. I think long term we need to
> evaluate our testing strategy to see if we can come up with something
> more efficient.
>
> Regards,
>
> Rajith
>
> On Mon, Jul 4, 2011 at 7:11 AM, Robert Godfrey <ro...@gmail.com> wrote:
>> On 4 July 2011 12:44, Keith Wall <ke...@gmail.com> wrote:
>>
>>> Robbie and I are presently working on QPID-2811 and QPID-2815. These Jiras
>>> will rationalise the transport mechanism in Qpid and to allow client/broker
>>> to be tested together within the same JVM regardless of transport.
>>>
>>> It seems to me that now is a good time to review the test-profiles. I see
>>> the following problems:
>>>
>>> 1) 'default' profile being 0-8 makes little sense as most of the project's
>>> development focus is on 0-10, and with the removal of the old .NET client,
>>> we have no client that defaults to 0-8.
>>> 2) The time taken to run some test profiles is excessive and this is a
>>> disincentive to developer good practice. For instance, running java-0.10
>>> on
>>> my machine takes 30mins. This is harming productivity.
>>>
>>> Current profiles:
>>>
>>> default.testprofile - tests 0-8 using in-VM, broker/client share JVM,
>>> Memory Message Store
>>> java.testprofile - tests 0-9-1, using tcp, separate JVMs, Memory Message
>>> Store
>>> java-derby.testprofile - tests 0-9-1, using tcp, separate JVMs, Derby
>>> Message Store
>>>
>>> java.0.10.testprofile - tests 0-10, using tcp, separate JVMs,, Memory
>>> Message Store
>>> java-derby.0.10.testprofile -- tests 0-10, using tcp, separate JVMs, Derby
>>> Message Store
>>>
>>> My proposal:
>>>
>>> Four profiles intended for regular developer use. These profiles will
>>> exploit the feature for client/broker to share the same VM. This saves the
>>> overhead of each testcase spawning a new broker as a separate process, and
>>> brings the length of time to run the profile down considerably, assisting
>>> developer productivity.
>>>
>>> (mms=memory message store, dby = Apache Derby)
>>>
>>> java-mms.0-10
>>> java-dby.0-10
>>>
>>> java-mms.0-9-1
>>> java-dby.0-9-1
>>>
>>> Running client/broker in separate JVMs remains valuable, but is probably
>>> not
>>> something most developers will want to do most of the time (because of the
>>> time overhead). The intention would be we'd schedule these to run on CI
>>> (using Jenkin's matrix feature) and let Jenkins do the heavy lifting.
>>>
>>> java-mms-spawn.0-10
>>> java-dby-spawn.0-10
>>>
>>> java-mms-spawn.0-9-1
>>> java-dby-spawn.0-9-1
>>>
>>> The idea of the default testprofile would disappear and it would become an
>>> error to run -Dprofile=default. If the developer omits -Dprofile from the
>>> Ant command line, the system will run java-mms.0-10, which I think will be
>>> useful fallback to most developers.
>>>
>>> I think we also need a "Java Test Howto" page on the website, explaining
>>> the
>>> purposes behind the profiles, the differences between system and non-system
>>> tests and giving advice on appropriate use of both.
>>>
>>> Any thoughts?
>>>
>>> Keith.
>>>
>>
>>
>> All sounds exceedingly sensible to me.
>>
>> -- Rob
>>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: Ideas to rationalise the Java test profiles.
Posted by Robbie Gemmell <ro...@gmail.com>.
On 4 July 2011 15:48, Gordon Sim <gs...@redhat.com> wrote:
> On 07/04/2011 03:39 PM, Rajith Attapattu wrote:
>>
>> Just to add to it, any changes to the Java client should also be
>> tested with the cpp test profile.
>> Any changes to the client failover mechanism should also be tested
>> with the cpp.cluster profile.
>>
>> I know it's a pain to run all these profiles, especially considering
>> each run takes about 30-40 mins.
>> But atm there is no better alternative.
>
> I think Keith's suggestion is a very reasonable alternative. It would mean
> that the client's 0-10 codepath is always exercised, which I think is the
> critical thing. If that were done reliably before checkin I think it would
> be an improvement, and by minimising the running time I think that becomes
> more realistic.
>
> Of course the handling of subtle differences between the java- and c++-
> brokers on 0-10 is something we want to test, but having that done even
> nightly by an automated test infrastructure would go a long way there
> (provided we actually stay on top of failure reports!).
>
> What I would like is a simple way to run the tests against an externally
> started broker. I.e. I can give a port (and optionally hostname and perhaps
> authentication details), and then run through the test suite against a
> broker I started myself.
>
It would be good if we could get the C++ broker/cluster tests running
on the ASF Jenkins infrastructure, although I know Andrew Kennedy did
start down that route previously and found doing so would involve a
significant amount of setup to work with the Ubuntu instances
available.
It is currently possible to run tests against an existing broker by
overriding the 'broker' property on the command line and specifying it
as 'external', however any tests in the suite which expect to be able
to restart the broker or pass it test-specific configuration will fail
in this scenario.
Robbie
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: Ideas to rationalise the Java test profiles.
Posted by Gordon Sim <gs...@redhat.com>.
On 07/04/2011 03:39 PM, Rajith Attapattu wrote:
> Just to add to it, any changes to the Java client should also be
> tested with the cpp test profile.
> Any changes to the client failover mechanism should also be tested
> with the cpp.cluster profile.
>
> I know it's a pain to run all these profiles, especially considering
> each run takes about 30-40 mins.
> But atm there is no better alternative.
I think Keith's suggestion is a very reasonable alternative. It would
mean that the client's 0-10 codepath is always exercised, which I think
is the critical thing. If that were done reliably before checkin I think
it would be an improvement, and by minimising the running time I think
that becomes more realistic.
Of course the handling of subtle differences between the java- and c++-
brokers on 0-10 is something we want to test, but having that done even
nightly by an automated test infrastructure would go a long way there
(provided we actually stay on top of failure reports!).
What I would like is a simple way to run the tests against an externally
started broker. I.e. I can give a port (and optionally hostname and
perhaps authentication details), and then run through the test suite
against a broker I started myself.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: Ideas to rationalise the Java test profiles.
Posted by Rajith Attapattu <ra...@gmail.com>.
Just to add to it, any changes to the Java client should also be
tested with the cpp test profile.
Any changes to the client failover mechanism should also be tested
with the cpp.cluster profile.
I know it's a pain to run all these profiles, especially considering
each run takes about 30-40 mins.
But atm there is no better alternative. I think long term we need to
evaluate our testing strategy to see if we can come up with something
more efficient.
Regards,
Rajith
On Mon, Jul 4, 2011 at 7:11 AM, Robert Godfrey <ro...@gmail.com> wrote:
> On 4 July 2011 12:44, Keith Wall <ke...@gmail.com> wrote:
>
>> Robbie and I are presently working on QPID-2811 and QPID-2815. These Jiras
>> will rationalise the transport mechanism in Qpid and to allow client/broker
>> to be tested together within the same JVM regardless of transport.
>>
>> It seems to me that now is a good time to review the test-profiles. I see
>> the following problems:
>>
>> 1) 'default' profile being 0-8 makes little sense as most of the project's
>> development focus is on 0-10, and with the removal of the old .NET client,
>> we have no client that defaults to 0-8.
>> 2) The time taken to run some test profiles is excessive and this is a
>> disincentive to developer good practice. For instance, running java-0.10
>> on
>> my machine takes 30mins. This is harming productivity.
>>
>> Current profiles:
>>
>> default.testprofile - tests 0-8 using in-VM, broker/client share JVM,
>> Memory Message Store
>> java.testprofile - tests 0-9-1, using tcp, separate JVMs, Memory Message
>> Store
>> java-derby.testprofile - tests 0-9-1, using tcp, separate JVMs, Derby
>> Message Store
>>
>> java.0.10.testprofile - tests 0-10, using tcp, separate JVMs,, Memory
>> Message Store
>> java-derby.0.10.testprofile -- tests 0-10, using tcp, separate JVMs, Derby
>> Message Store
>>
>> My proposal:
>>
>> Four profiles intended for regular developer use. These profiles will
>> exploit the feature for client/broker to share the same VM. This saves the
>> overhead of each testcase spawning a new broker as a separate process, and
>> brings the length of time to run the profile down considerably, assisting
>> developer productivity.
>>
>> (mms=memory message store, dby = Apache Derby)
>>
>> java-mms.0-10
>> java-dby.0-10
>>
>> java-mms.0-9-1
>> java-dby.0-9-1
>>
>> Running client/broker in separate JVMs remains valuable, but is probably
>> not
>> something most developers will want to do most of the time (because of the
>> time overhead). The intention would be we'd schedule these to run on CI
>> (using Jenkin's matrix feature) and let Jenkins do the heavy lifting.
>>
>> java-mms-spawn.0-10
>> java-dby-spawn.0-10
>>
>> java-mms-spawn.0-9-1
>> java-dby-spawn.0-9-1
>>
>> The idea of the default testprofile would disappear and it would become an
>> error to run -Dprofile=default. If the developer omits -Dprofile from the
>> Ant command line, the system will run java-mms.0-10, which I think will be
>> useful fallback to most developers.
>>
>> I think we also need a "Java Test Howto" page on the website, explaining
>> the
>> purposes behind the profiles, the differences between system and non-system
>> tests and giving advice on appropriate use of both.
>>
>> Any thoughts?
>>
>> Keith.
>>
>
>
> All sounds exceedingly sensible to me.
>
> -- Rob
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: Ideas to rationalise the Java test profiles.
Posted by Robert Godfrey <ro...@gmail.com>.
On 4 July 2011 12:44, Keith Wall <ke...@gmail.com> wrote:
> Robbie and I are presently working on QPID-2811 and QPID-2815. These Jiras
> will rationalise the transport mechanism in Qpid and to allow client/broker
> to be tested together within the same JVM regardless of transport.
>
> It seems to me that now is a good time to review the test-profiles. I see
> the following problems:
>
> 1) 'default' profile being 0-8 makes little sense as most of the project's
> development focus is on 0-10, and with the removal of the old .NET client,
> we have no client that defaults to 0-8.
> 2) The time taken to run some test profiles is excessive and this is a
> disincentive to developer good practice. For instance, running java-0.10
> on
> my machine takes 30mins. This is harming productivity.
>
> Current profiles:
>
> default.testprofile - tests 0-8 using in-VM, broker/client share JVM,
> Memory Message Store
> java.testprofile - tests 0-9-1, using tcp, separate JVMs, Memory Message
> Store
> java-derby.testprofile - tests 0-9-1, using tcp, separate JVMs, Derby
> Message Store
>
> java.0.10.testprofile - tests 0-10, using tcp, separate JVMs,, Memory
> Message Store
> java-derby.0.10.testprofile -- tests 0-10, using tcp, separate JVMs, Derby
> Message Store
>
> My proposal:
>
> Four profiles intended for regular developer use. These profiles will
> exploit the feature for client/broker to share the same VM. This saves the
> overhead of each testcase spawning a new broker as a separate process, and
> brings the length of time to run the profile down considerably, assisting
> developer productivity.
>
> (mms=memory message store, dby = Apache Derby)
>
> java-mms.0-10
> java-dby.0-10
>
> java-mms.0-9-1
> java-dby.0-9-1
>
> Running client/broker in separate JVMs remains valuable, but is probably
> not
> something most developers will want to do most of the time (because of the
> time overhead). The intention would be we'd schedule these to run on CI
> (using Jenkin's matrix feature) and let Jenkins do the heavy lifting.
>
> java-mms-spawn.0-10
> java-dby-spawn.0-10
>
> java-mms-spawn.0-9-1
> java-dby-spawn.0-9-1
>
> The idea of the default testprofile would disappear and it would become an
> error to run -Dprofile=default. If the developer omits -Dprofile from the
> Ant command line, the system will run java-mms.0-10, which I think will be
> useful fallback to most developers.
>
> I think we also need a "Java Test Howto" page on the website, explaining
> the
> purposes behind the profiles, the differences between system and non-system
> tests and giving advice on appropriate use of both.
>
> Any thoughts?
>
> Keith.
>
All sounds exceedingly sensible to me.
-- Rob