You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by artnaseef <ar...@artnaseef.com> on 2015/02/01 02:25:04 UTC
[DISCUSS] Improving ActiveMQ Tests
*Background*
As we all know, ActiveMQ tests take a very long time to execute, and a
number of the tests have been unreliable. Personally, it was ActiveMQ that
lead me to learn maven's "skip test" features early on because I would not
wait hours to get a small change into a build during development, nor did I
want to spend time (as a user of ActiveMQ - this is before I was a
committer) trying to resolve test failures. This contradicts industry
best-practices.
A large, world-class product such as ActiveMQ relies, wisely, on automated
tests to ensure regression bugs are not introduced with new feature
additions and bug fixes.
ActiveMQ unit tests really contains a variety of test types, such as:
Use Case tests
Load tests
Bug Reproduction tests
Unit Tests that target a single class (i.e. True Unit Tests)*
* I personally haven't seen any of these in ActiveMQ, but I suspect/hope
that some exist.
*Goals*
Improve the automated testing with the following result in mind:
Developer cycle time: reduce the amount of time to execute tests when making
changes to ActiveMQ code so that a developer gets a reasonable expectation
that changes are valid without drawing the development cycle out into
excessive time periods
Continued quality: continue to cover all of the test scenarios currently
covered
Release validation: continue to provide very thorough validation for
releases at the expense of significant, additional test time
Reliability: improve reliability of tests for the release process so that
it is reasonable to expect a good build every time for quality releases
Verify units: improve overall quality of the system through more thorough
testing of each unit (class) so that small bugs in units do not turn into
hard-to-find bugs in the system (I found one of these just walking through
code yesterday - it will get fixed soon)
*Ideas*
Identify junit tests that are really integration or load tests and separate
as such
Create an artifact for integration tests, move integration tests there and
run them during the verify phase
Create an artifact for load tests, move load tests there and also run
during the verify phase (interim)
Stand up a true load-test environment and move load tests into that
environment and out of the main build process
Create true unit tests with high code coverage
Move tests into separate artifacts to reduce the number of tests, and
therefore the amount of time necessary to resume tests from a failure point
(e.g. split activemq-unit-test into activemq-unit-test01, ...02, etc)
Review problematic tests and determine ways to resolve (e.g. replace with
new tests using different approaches, use the load test environment, etc)
*Wrap-Up*
The steps above will move ActiveMQ testing toward a solid, reliable test
framework without creating gaps in quality.
Unless alternative plans are defined, I am going to enter Jira tickets for
these and start moving in this direction. Volunteers are welcome and
greatly appreciated.
--
View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-Improving-ActiveMQ-Tests-tp4690766.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
Re: [DISCUSS] Improving ActiveMQ Tests
Posted by artnaseef <ar...@artnaseef.com>.
Thanks Mark!
--
View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-Improving-ActiveMQ-Tests-tp4690766p4690785.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
Re: [DISCUSS] Improving ActiveMQ Tests
Posted by artnaseef <ar...@artnaseef.com>.
Clebert - please start a separate thread to discuss the move to AMQ 6 vs AMQ
5 .
I appreciate the desire to take advantage of AMQ 6 tests, but this is not
the thread on which to argue any move away from AMQ 5.
--
View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-Improving-ActiveMQ-Tests-tp4690766p4690790.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
Re: [DISCUSS] Improving ActiveMQ Tests
Posted by Clebert <cl...@gmail.com>.
Not yet .... I'm just suggesting To work together towards making a better broker.
-- Clebert Suconic typing on the iPhone.
> On Feb 1, 2015, at 16:34, artnaseef <ar...@artnaseef.com> wrote:
>
> Clebert - I'm not ready to declare AMQ 5 ready for the attic, especially
> since AMQ 6 isn't really an upgrade of the "traditional" line of ActiveMQ
> and it does not operate as a drop-in replacement.
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-Improving-ActiveMQ-Tests-tp4690766p4690784.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
Re: [DISCUSS] Improving ActiveMQ Tests
Posted by artnaseef <ar...@artnaseef.com>.
Clebert - I'm not ready to declare AMQ 5 ready for the attic, especially
since AMQ 6 isn't really an upgrade of the "traditional" line of ActiveMQ
and it does not operate as a drop-in replacement.
--
View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-Improving-ActiveMQ-Tests-tp4690766p4690784.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
Re: [DISCUSS] Improving ActiveMQ Tests
Posted by "Jamie G." <ja...@gmail.com>.
+1 Art,
I had some other ideas on another thread, reading the general concept
it sounds like a possible way forward to making AMQ testing easier.
Cheers,
J
On Sun, Feb 1, 2015 at 12:34 AM, Clebert Suconic
<cl...@gmail.com> wrote:
> We have been through that exercise already on activemq-6. Perhaps it
> would be a better use of resources if we worked towards activemq-6?
>
> With hornetq (now activemq-6) we differentiated unit-tests,
> integration tests, soak tests and performance tests.
>
> When you do a build you won't do the entire testsuite to validate a
> simple build.. it's just a matter of minutes and simple unit tests.
>
> With integration tests (we could find a better name for it) we do more
> extensive tests and even then we limit certain things as good
> practices. (like using more latches and less Sleeps.. unless you
> really have to. It's impossible to run all the profiles on every
> time.. Like you can't force someone to run a 10Hours soak test to make
> a simple build... we have profiles for that.
>
> Best practices is the key on keeping the testsuite clean. As I said on
> my first paragraph, maybe we should concentrate efforts on activemq-6,
> Improving what we have on activemq-6 repo is a better bet IMO.
>
> We can keep this discussion open if you need help on any specifics on
> coding and running the profiles we have.
>
>
> On Sat, Jan 31, 2015 at 8:35 PM, Mark Frazier <mm...@me.com> wrote:
>> +1, this seems like a great idea Art.
>>
>> I'll be glad to help with this.
>>
>>
>>> On Jan 31, 2015, at 5:25 PM, artnaseef <ar...@artnaseef.com> wrote:
>>>
>>> *Background*
>>> As we all know, ActiveMQ tests take a very long time to execute, and a
>>> number of the tests have been unreliable. Personally, it was ActiveMQ that
>>> lead me to learn maven's "skip test" features early on because I would not
>>> wait hours to get a small change into a build during development, nor did I
>>> want to spend time (as a user of ActiveMQ - this is before I was a
>>> committer) trying to resolve test failures. This contradicts industry
>>> best-practices.
>>> A large, world-class product such as ActiveMQ relies, wisely, on automated
>>> tests to ensure regression bugs are not introduced with new feature
>>> additions and bug fixes.
>>> ActiveMQ unit tests really contains a variety of test types, such as:
>>> Use Case tests
>>> Load tests
>>> Bug Reproduction tests
>>> Unit Tests that target a single class (i.e. True Unit Tests)*
>>> * I personally haven't seen any of these in ActiveMQ, but I suspect/hope
>>> that some exist.
>>>
>>> *Goals*
>>> Improve the automated testing with the following result in mind:
>>> Developer cycle time: reduce the amount of time to execute tests when making
>>> changes to ActiveMQ code so that a developer gets a reasonable expectation
>>> that changes are valid without drawing the development cycle out into
>>> excessive time periods
>>> Continued quality: continue to cover all of the test scenarios currently
>>> covered
>>> Release validation: continue to provide very thorough validation for
>>> releases at the expense of significant, additional test time
>>> Reliability: improve reliability of tests for the release process so that
>>> it is reasonable to expect a good build every time for quality releases
>>> Verify units: improve overall quality of the system through more thorough
>>> testing of each unit (class) so that small bugs in units do not turn into
>>> hard-to-find bugs in the system (I found one of these just walking through
>>> code yesterday - it will get fixed soon)
>>>
>>> *Ideas*
>>> Identify junit tests that are really integration or load tests and separate
>>> as such
>>>
>>> Create an artifact for integration tests, move integration tests there and
>>> run them during the verify phase
>>> Create an artifact for load tests, move load tests there and also run
>>> during the verify phase (interim)
>>> Stand up a true load-test environment and move load tests into that
>>> environment and out of the main build process
>>> Create true unit tests with high code coverage
>>> Move tests into separate artifacts to reduce the number of tests, and
>>> therefore the amount of time necessary to resume tests from a failure point
>>> (e.g. split activemq-unit-test into activemq-unit-test01, ...02, etc)
>>> Review problematic tests and determine ways to resolve (e.g. replace with
>>> new tests using different approaches, use the load test environment, etc)
>>>
>>> *Wrap-Up*
>>> The steps above will move ActiveMQ testing toward a solid, reliable test
>>> framework without creating gaps in quality.
>>> Unless alternative plans are defined, I am going to enter Jira tickets for
>>> these and start moving in this direction. Volunteers are welcome and
>>> greatly appreciated.
>>>
>>>
>>>
>>> --
>>> View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-Improving-ActiveMQ-Tests-tp4690766.html
>>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>
>
>
> --
> Clebert Suconic
> http://community.jboss.org/people/clebert.suconic@jboss.com
> http://clebertsuconic.blogspot.com
Re: [DISCUSS] Improving ActiveMQ Tests
Posted by Clebert Suconic <cl...@gmail.com>.
We have been through that exercise already on activemq-6. Perhaps it
would be a better use of resources if we worked towards activemq-6?
With hornetq (now activemq-6) we differentiated unit-tests,
integration tests, soak tests and performance tests.
When you do a build you won't do the entire testsuite to validate a
simple build.. it's just a matter of minutes and simple unit tests.
With integration tests (we could find a better name for it) we do more
extensive tests and even then we limit certain things as good
practices. (like using more latches and less Sleeps.. unless you
really have to. It's impossible to run all the profiles on every
time.. Like you can't force someone to run a 10Hours soak test to make
a simple build... we have profiles for that.
Best practices is the key on keeping the testsuite clean. As I said on
my first paragraph, maybe we should concentrate efforts on activemq-6,
Improving what we have on activemq-6 repo is a better bet IMO.
We can keep this discussion open if you need help on any specifics on
coding and running the profiles we have.
On Sat, Jan 31, 2015 at 8:35 PM, Mark Frazier <mm...@me.com> wrote:
> +1, this seems like a great idea Art.
>
> I'll be glad to help with this.
>
>
>> On Jan 31, 2015, at 5:25 PM, artnaseef <ar...@artnaseef.com> wrote:
>>
>> *Background*
>> As we all know, ActiveMQ tests take a very long time to execute, and a
>> number of the tests have been unreliable. Personally, it was ActiveMQ that
>> lead me to learn maven's "skip test" features early on because I would not
>> wait hours to get a small change into a build during development, nor did I
>> want to spend time (as a user of ActiveMQ - this is before I was a
>> committer) trying to resolve test failures. This contradicts industry
>> best-practices.
>> A large, world-class product such as ActiveMQ relies, wisely, on automated
>> tests to ensure regression bugs are not introduced with new feature
>> additions and bug fixes.
>> ActiveMQ unit tests really contains a variety of test types, such as:
>> Use Case tests
>> Load tests
>> Bug Reproduction tests
>> Unit Tests that target a single class (i.e. True Unit Tests)*
>> * I personally haven't seen any of these in ActiveMQ, but I suspect/hope
>> that some exist.
>>
>> *Goals*
>> Improve the automated testing with the following result in mind:
>> Developer cycle time: reduce the amount of time to execute tests when making
>> changes to ActiveMQ code so that a developer gets a reasonable expectation
>> that changes are valid without drawing the development cycle out into
>> excessive time periods
>> Continued quality: continue to cover all of the test scenarios currently
>> covered
>> Release validation: continue to provide very thorough validation for
>> releases at the expense of significant, additional test time
>> Reliability: improve reliability of tests for the release process so that
>> it is reasonable to expect a good build every time for quality releases
>> Verify units: improve overall quality of the system through more thorough
>> testing of each unit (class) so that small bugs in units do not turn into
>> hard-to-find bugs in the system (I found one of these just walking through
>> code yesterday - it will get fixed soon)
>>
>> *Ideas*
>> Identify junit tests that are really integration or load tests and separate
>> as such
>>
>> Create an artifact for integration tests, move integration tests there and
>> run them during the verify phase
>> Create an artifact for load tests, move load tests there and also run
>> during the verify phase (interim)
>> Stand up a true load-test environment and move load tests into that
>> environment and out of the main build process
>> Create true unit tests with high code coverage
>> Move tests into separate artifacts to reduce the number of tests, and
>> therefore the amount of time necessary to resume tests from a failure point
>> (e.g. split activemq-unit-test into activemq-unit-test01, ...02, etc)
>> Review problematic tests and determine ways to resolve (e.g. replace with
>> new tests using different approaches, use the load test environment, etc)
>>
>> *Wrap-Up*
>> The steps above will move ActiveMQ testing toward a solid, reliable test
>> framework without creating gaps in quality.
>> Unless alternative plans are defined, I am going to enter Jira tickets for
>> these and start moving in this direction. Volunteers are welcome and
>> greatly appreciated.
>>
>>
>>
>> --
>> View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-Improving-ActiveMQ-Tests-tp4690766.html
>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
--
Clebert Suconic
http://community.jboss.org/people/clebert.suconic@jboss.com
http://clebertsuconic.blogspot.com
Re: [DISCUSS] Improving ActiveMQ Tests
Posted by Mark Frazier <mm...@me.com>.
+1, this seems like a great idea Art.
I'll be glad to help with this.
> On Jan 31, 2015, at 5:25 PM, artnaseef <ar...@artnaseef.com> wrote:
>
> *Background*
> As we all know, ActiveMQ tests take a very long time to execute, and a
> number of the tests have been unreliable. Personally, it was ActiveMQ that
> lead me to learn maven's "skip test" features early on because I would not
> wait hours to get a small change into a build during development, nor did I
> want to spend time (as a user of ActiveMQ - this is before I was a
> committer) trying to resolve test failures. This contradicts industry
> best-practices.
> A large, world-class product such as ActiveMQ relies, wisely, on automated
> tests to ensure regression bugs are not introduced with new feature
> additions and bug fixes.
> ActiveMQ unit tests really contains a variety of test types, such as:
> Use Case tests
> Load tests
> Bug Reproduction tests
> Unit Tests that target a single class (i.e. True Unit Tests)*
> * I personally haven't seen any of these in ActiveMQ, but I suspect/hope
> that some exist.
>
> *Goals*
> Improve the automated testing with the following result in mind:
> Developer cycle time: reduce the amount of time to execute tests when making
> changes to ActiveMQ code so that a developer gets a reasonable expectation
> that changes are valid without drawing the development cycle out into
> excessive time periods
> Continued quality: continue to cover all of the test scenarios currently
> covered
> Release validation: continue to provide very thorough validation for
> releases at the expense of significant, additional test time
> Reliability: improve reliability of tests for the release process so that
> it is reasonable to expect a good build every time for quality releases
> Verify units: improve overall quality of the system through more thorough
> testing of each unit (class) so that small bugs in units do not turn into
> hard-to-find bugs in the system (I found one of these just walking through
> code yesterday - it will get fixed soon)
>
> *Ideas*
> Identify junit tests that are really integration or load tests and separate
> as such
>
> Create an artifact for integration tests, move integration tests there and
> run them during the verify phase
> Create an artifact for load tests, move load tests there and also run
> during the verify phase (interim)
> Stand up a true load-test environment and move load tests into that
> environment and out of the main build process
> Create true unit tests with high code coverage
> Move tests into separate artifacts to reduce the number of tests, and
> therefore the amount of time necessary to resume tests from a failure point
> (e.g. split activemq-unit-test into activemq-unit-test01, ...02, etc)
> Review problematic tests and determine ways to resolve (e.g. replace with
> new tests using different approaches, use the load test environment, etc)
>
> *Wrap-Up*
> The steps above will move ActiveMQ testing toward a solid, reliable test
> framework without creating gaps in quality.
> Unless alternative plans are defined, I am going to enter Jira tickets for
> these and start moving in this direction. Volunteers are welcome and
> greatly appreciated.
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-Improving-ActiveMQ-Tests-tp4690766.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.