You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geronimo.apache.org by Prasad Kashyap <go...@gmail.com> on 2006/09/28 05:40:12 UTC
Testsuite and maven-maven-plugin
Let me try to articulate my vision of the multi-module setup under
testsuite. If my understanding is off base, please feel free to show
me the bigger picture.
- testsuite/
|
| - - - console-testsuite/
| |
| | - - - pom.xml
| |
| | - - - src/test/java/...
|
| - - - webcontainer-testsuite/
| |
| | - - - pom.xml
| |
| | - - - src/test/java/...
|
| - - - ejbcontainer-testsuite/
| |
| | - - - pom.xml
| |
| | - - - src/test/java/...
Custom packaging or lifecycle phase bindings
---------------------------------------------------------------------
The geronimo-maven-plugin (g-m-p) goals like start/stop server,
deploy/undeploy modules will be executed in each of these pom.xmls.
The start-server and deploy-module goals will be bound to some early
phase in the lifecycle, say "validate" (example only) using the
@phase. The stop-server and undeploy-module goals will be bound to a
later phase, say "install". The tests will be executed in the "test"
phase in between the modules deploy and undeploy.
In each *-testsuite, multiple modules can be deployed and a subset of
tests can be run against them. Eg: ATest runs on module1, BTest runs
on module2 and module3, CTest runs on module3 etc.
Exception handling
----------------------------
Only the exceptions from start-server and stop-server goals should
stop the test run completely. Failures from deploy-modules and
undeploy-modules should be logged and the test should continue.
Failures from TestNG testcases should be logged and ignored. Tests
should continue.
Cleanup will done by the stop-server goal at the end. The start-server
configuration in each *-testsuite should have the <refresh> set to
true. This will ensure that each suite will start with a clean slate.
Reporting
--------------
The goals in g-m-p already support multiple reporting mechanisms
(SurefireReporter already implemented). The TestNG testcases will be
logged and reported by surefire anyway. So we have one reporting story
in place. Similar to the way g-m-p goals handles other reporter
implementations, we have to write a goal that will handle the TestNG
test exceptions for other reporters too.
I'd appreciate your comments.
Cheers
Prasad
Re: Testsuite and maven-maven-plugin
Posted by Jason Dillon <ja...@planet57.com>.
*-testsuite modules are intended to be self contained/stand-alone...
which is why I was putting the build ticket there. But if we design
the reporting bits right then we could have linked tickets, so that a
top-level testsuite could have child suites with their own tickets.
But if the ticket only happens in the testsuite, then it makes it
hard to simply run one suite at a time and collect reports.
--jason
On Oct 2, 2006, at 6:39 AM, Prasad Kashyap wrote:
> Thanks Jason for showing me the bigger picture. I agree with you now,
> by and large.
>
> However in your build flow for each *-testsuite, you begin with
> generating a build ticket. I wonder if we should generate the build
> ticket at the geronimo/testsuite level and not one level down.
>
> Yes, we need some real tests now. I'll start working on one so that it
> could serve as an example for others to write their own.
>
> Cheers
> Prasad
>
> On 9/28/06, Jason Dillon <ja...@planet57.com> wrote:
>> On Sep 27, 2006, at 8:40 PM, Prasad Kashyap wrote:
>>
>> Build flow (for a given *-testsuite), probably ends up something
>> like:
>>
>> generate build ticket (used to track all child module reports)
>> do pre-integration-test (start-server, blah)
>> for each module {
>> run tests
>> run plugins (like pmd, clover, etc)
>> collect reports
>> }
>> do post-integration-test
>> publish reports
>>
>> Published reports for a given run will all be indexed together by the
>> build ticket, so it will be easy to see what results are for which
>> build. In this example "generate build ticket", "collect reports"
>> and "publish reports" would all be handled by the reporting plugin I
>> am talking about, the rest is the existing testsuite bits.
>>
>> I think this will work out quite well... giving us flexibility w/o
>> making our exiting plugins more complicated or waiting for other
>> plugin developers to make modifications for us (which is not very
>> likely to happen anyways).
>>
>> Anyways...
>>
>> I think that the reporters we have now (in g-m-p) is fine for a while
>> though... but as soon as we need to start adding similar reporter-
>> like functionality to other plugins, then we should really consider
>> implementing a more generic reporting system that can function on
>> plugin outputs alone (or using a maven reporting api if/when it
>> becomes available).
>>
>> * * *
>>
>> Overall I think the testsuite work of late is positive movement for
>> our community to be able to start building a more comprehensive set
>> of integration and system tests... NOW.
>>
>> I believe that we do need to start working on making some real
>> tests... so that we can validate that the framework we have now is
>> sufficient, and if not refactor it now before it sits for too long.
>>
>> --jason
>>
Re: Testsuite and maven-maven-plugin
Posted by Prasad Kashyap <go...@gmail.com>.
Thanks Jason for showing me the bigger picture. I agree with you now,
by and large.
However in your build flow for each *-testsuite, you begin with
generating a build ticket. I wonder if we should generate the build
ticket at the geronimo/testsuite level and not one level down.
Yes, we need some real tests now. I'll start working on one so that it
could serve as an example for others to write their own.
Cheers
Prasad
On 9/28/06, Jason Dillon <ja...@planet57.com> wrote:
> On Sep 27, 2006, at 8:40 PM, Prasad Kashyap wrote:
>
> Build flow (for a given *-testsuite), probably ends up something like:
>
> generate build ticket (used to track all child module reports)
> do pre-integration-test (start-server, blah)
> for each module {
> run tests
> run plugins (like pmd, clover, etc)
> collect reports
> }
> do post-integration-test
> publish reports
>
> Published reports for a given run will all be indexed together by the
> build ticket, so it will be easy to see what results are for which
> build. In this example "generate build ticket", "collect reports"
> and "publish reports" would all be handled by the reporting plugin I
> am talking about, the rest is the existing testsuite bits.
>
> I think this will work out quite well... giving us flexibility w/o
> making our exiting plugins more complicated or waiting for other
> plugin developers to make modifications for us (which is not very
> likely to happen anyways).
>
> Anyways...
>
> I think that the reporters we have now (in g-m-p) is fine for a while
> though... but as soon as we need to start adding similar reporter-
> like functionality to other plugins, then we should really consider
> implementing a more generic reporting system that can function on
> plugin outputs alone (or using a maven reporting api if/when it
> becomes available).
>
> * * *
>
> Overall I think the testsuite work of late is positive movement for
> our community to be able to start building a more comprehensive set
> of integration and system tests... NOW.
>
> I believe that we do need to start working on making some real
> tests... so that we can validate that the framework we have now is
> sufficient, and if not refactor it now before it sits for too long.
>
> --jason
>
Re: Testsuite and maven-maven-plugin
Posted by Jason Dillon <ja...@planet57.com>.
Both passed fine for me. Perhaps you did not have an up to date
build or a sync'd testsuite with the latest changes?
--jason
On Sep 28, 2006, at 6:59 PM, Prasad Kashyap wrote:
> The first test timed out in 30 mins. The 2nd test seems to be just
> blocked.
>
> [INFO] -------------------------------------------------------
> [INFO] T E S T S
> [INFO] -------------------------------------------------------
> [INFO] Running org.apache.geronimo.testsuite.deployment.Ear13Test
> [INFO] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time
> elapsed: 1,800.63
> sec <<< FAILURE!
> [INFO] Running org.apache.geronimo.testsuite.deployment.Ear14Test
>
> Cheers
> Prasad
>
>
> On 9/28/06, Jason Dillon <ja...@planet57.com> wrote:
>> Sounds like its going to timeout hitting the war pages...
>>
>> --jason
>>
>>
>> On Sep 28, 2006, at 3:01 PM, Prasad Kashyap wrote:
>>
>> > The selenium tests on windows blocks. Firefox does not open.
>> >
>> > I now have 5 java processes running.
>> >
>> > [INFO] -------------------------------------------------------
>> > [INFO] T E S T S
>> > [INFO] -------------------------------------------------------
>> > [INFO] Running org.apache.geronimo.testsuite.deployment.Ear13Test
>> >
>> > Cheers
>> > Prasad.
>> >
>> > On 9/28/06, Jason Dillon <ja...@planet57.com> wrote:
>> >> On Sep 28, 2006, at 2:28 PM, Prasad Kashyap wrote:
>> >> > Thanx for the response Jason.
>> >> >
>> >> > Please see if the attached components.xml is good for our new
>> >> > "integration-test" custom packaging.
>> >> > http://rifers.org/paste/show/1855
>> >>
>> >> Hi Prasad, I have already committed the new packaging and
>> hooked up
>> >> the deployment-testsuite to use it.
>> >>
>> >>
>> >> > If we used the m-m-p to invoke child poms, won't they inherit
>> the
>> >> > goals from their parents ? If they do, then won't start-
>> server and
>> >> > stop-server be also executed ?
>> >>
>> >> You will notice there is is a "child" profile in each *-testsuite
>> >> module and that the m-m-p sets this profile to be active on
>> executed
>> >> child modules. This was done specifically to avoid children
>> picking
>> >> up the start-server/stop-server configuration from their parent.
>> >>
>> >> --jason
>> >>
>> >>
>> >>
>>
>>
Re: Testsuite and maven-maven-plugin
Posted by Prasad Kashyap <go...@gmail.com>.
The first test timed out in 30 mins. The 2nd test seems to be just blocked.
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.geronimo.testsuite.deployment.Ear13Test
[INFO] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1,800.63
sec <<< FAILURE!
[INFO] Running org.apache.geronimo.testsuite.deployment.Ear14Test
Cheers
Prasad
On 9/28/06, Jason Dillon <ja...@planet57.com> wrote:
> Sounds like its going to timeout hitting the war pages...
>
> --jason
>
>
> On Sep 28, 2006, at 3:01 PM, Prasad Kashyap wrote:
>
> > The selenium tests on windows blocks. Firefox does not open.
> >
> > I now have 5 java processes running.
> >
> > [INFO] -------------------------------------------------------
> > [INFO] T E S T S
> > [INFO] -------------------------------------------------------
> > [INFO] Running org.apache.geronimo.testsuite.deployment.Ear13Test
> >
> > Cheers
> > Prasad.
> >
> > On 9/28/06, Jason Dillon <ja...@planet57.com> wrote:
> >> On Sep 28, 2006, at 2:28 PM, Prasad Kashyap wrote:
> >> > Thanx for the response Jason.
> >> >
> >> > Please see if the attached components.xml is good for our new
> >> > "integration-test" custom packaging.
> >> > http://rifers.org/paste/show/1855
> >>
> >> Hi Prasad, I have already committed the new packaging and hooked up
> >> the deployment-testsuite to use it.
> >>
> >>
> >> > If we used the m-m-p to invoke child poms, won't they inherit the
> >> > goals from their parents ? If they do, then won't start-server and
> >> > stop-server be also executed ?
> >>
> >> You will notice there is is a "child" profile in each *-testsuite
> >> module and that the m-m-p sets this profile to be active on executed
> >> child modules. This was done specifically to avoid children picking
> >> up the start-server/stop-server configuration from their parent.
> >>
> >> --jason
> >>
> >>
> >>
>
>
Re: Testsuite and maven-maven-plugin
Posted by Jason Dillon <ja...@planet57.com>.
Sounds like its going to timeout hitting the war pages...
--jason
On Sep 28, 2006, at 3:01 PM, Prasad Kashyap wrote:
> The selenium tests on windows blocks. Firefox does not open.
>
> I now have 5 java processes running.
>
> [INFO] -------------------------------------------------------
> [INFO] T E S T S
> [INFO] -------------------------------------------------------
> [INFO] Running org.apache.geronimo.testsuite.deployment.Ear13Test
>
> Cheers
> Prasad.
>
> On 9/28/06, Jason Dillon <ja...@planet57.com> wrote:
>> On Sep 28, 2006, at 2:28 PM, Prasad Kashyap wrote:
>> > Thanx for the response Jason.
>> >
>> > Please see if the attached components.xml is good for our new
>> > "integration-test" custom packaging.
>> > http://rifers.org/paste/show/1855
>>
>> Hi Prasad, I have already committed the new packaging and hooked up
>> the deployment-testsuite to use it.
>>
>>
>> > If we used the m-m-p to invoke child poms, won't they inherit the
>> > goals from their parents ? If they do, then won't start-server and
>> > stop-server be also executed ?
>>
>> You will notice there is is a "child" profile in each *-testsuite
>> module and that the m-m-p sets this profile to be active on executed
>> child modules. This was done specifically to avoid children picking
>> up the start-server/stop-server configuration from their parent.
>>
>> --jason
>>
>>
>>
Re: Testsuite and maven-maven-plugin
Posted by Prasad Kashyap <go...@gmail.com>.
The selenium tests on windows blocks. Firefox does not open.
I now have 5 java processes running.
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.geronimo.testsuite.deployment.Ear13Test
Cheers
Prasad.
On 9/28/06, Jason Dillon <ja...@planet57.com> wrote:
> On Sep 28, 2006, at 2:28 PM, Prasad Kashyap wrote:
> > Thanx for the response Jason.
> >
> > Please see if the attached components.xml is good for our new
> > "integration-test" custom packaging.
> > http://rifers.org/paste/show/1855
>
> Hi Prasad, I have already committed the new packaging and hooked up
> the deployment-testsuite to use it.
>
>
> > If we used the m-m-p to invoke child poms, won't they inherit the
> > goals from their parents ? If they do, then won't start-server and
> > stop-server be also executed ?
>
> You will notice there is is a "child" profile in each *-testsuite
> module and that the m-m-p sets this profile to be active on executed
> child modules. This was done specifically to avoid children picking
> up the start-server/stop-server configuration from their parent.
>
> --jason
>
>
>
Re: Testsuite and maven-maven-plugin
Posted by Jason Dillon <ja...@planet57.com>.
On Sep 28, 2006, at 2:28 PM, Prasad Kashyap wrote:
> Thanx for the response Jason.
>
> Please see if the attached components.xml is good for our new
> "integration-test" custom packaging.
> http://rifers.org/paste/show/1855
Hi Prasad, I have already committed the new packaging and hooked up
the deployment-testsuite to use it.
> If we used the m-m-p to invoke child poms, won't they inherit the
> goals from their parents ? If they do, then won't start-server and
> stop-server be also executed ?
You will notice there is is a "child" profile in each *-testsuite
module and that the m-m-p sets this profile to be active on executed
child modules. This was done specifically to avoid children picking
up the start-server/stop-server configuration from their parent.
--jason
Re: Testsuite and maven-maven-plugin
Posted by Prasad Kashyap <go...@gmail.com>.
Thanx for the response Jason.
Please see if the attached components.xml is good for our new
"integration-test" custom packaging.
http://rifers.org/paste/show/1855
If we used the m-m-p to invoke child poms, won't they inherit the
goals from their parents ? If they do, then won't start-server and
stop-server be also executed ?
Cheers
Prasad
On 9/28/06, Jason Dillon <ja...@planet57.com> wrote:
> On Sep 27, 2006, at 8:40 PM, Prasad Kashyap wrote:
> > Custom packaging or lifecycle phase bindings
> > ---------------------------------------------------------------------
> > The geronimo-maven-plugin (g-m-p) goals like start/stop server,
> > deploy/undeploy modules will be executed in each of these pom.xmls.
> > The start-server and deploy-module goals will be bound to some early
> > phase in the lifecycle, say "validate" (example only) using the
> > @phase. The stop-server and undeploy-module goals will be bound to a
> > later phase, say "install". The tests will be executed in the "test"
> > phase in between the modules deploy and undeploy.
>
> I think it is inappropriate to overload phases intended for a jar
> package to be used for integration tests. As I have mentioned a few
> times before, it is easy enough to create a new packaging which has
> an appropriate set of phases, probably something like:
>
> # validate
> # generate-test-sources
> # process-test-sources
> # generate-test-resources
> # process-test-resources
> # test-compile
> # pre-integration-test
> # test
> # post-integration-test
> # verify
>
> And then deploy-module is bound to pre-integration-test and undeploy-
> module is bound to post-integration-test.
>
>
> > In each *-testsuite, multiple modules can be deployed and a subset of
> > tests can be run against them. Eg: ATest runs on module1, BTest runs
> > on module2 and module3, CTest runs on module3 etc.
>
> I believe we want something more along the lines of:
>
> testsuite/
> console-testsuite/
> module1/
> src/test/java/...
> module2/
> src/test/java/...
> webcontainer-testsuite/
> module1/
> src/test/java/...
> module2/
> src/test/java/...
> ejbcontainer-testsuite/
> module1/
> src/test/java/...
> module2/
> src/test/java/...
>
> Where the *-testsuite poms contain the start-server, stop-server and
> run each module* in a separate child maven execution with the maven-
> maven-plugin. Each module can define a set of artifacts to deploy/
> undeploy.
>
> This allows different combinations of deployments to be tested
> against the same running server instance. It also provides a
> flexible setup to allow for better organization of tests and their
> deployment configuration. And, since each module is executed in a
> child maven process, the configuration of that execution is more
> flexible and not limited to working under the same profile that was
> used to start/stop the server.
>
> This is very similar to how the current testsuite is already setup.
>
>
> > Exception handling
> > ----------------------------
> > Only the exceptions from start-server and stop-server goals should
> > stop the test run completely. Failures from deploy-modules and
> > undeploy-modules should be logged and the test should continue.
> > Failures from TestNG testcases should be logged and ignored. Tests
> > should continue.
>
> Fair enough.
>
>
> > Cleanup will done by the stop-server goal at the end. The start-server
> > configuration in each *-testsuite should have the <refresh> set to
> > true. This will ensure that each suite will start with a clean slate.
>
> While I agree this may be needed (to use <refresh>) hopefully our
> tests will also function as expected with/out the refresh... as I
> hope that undeploy-module will function. So, I would extend the
> "cleanup" on to modules which deploy, then should also undeploy, so
> that the next module run does not have the previous modules clutter
> running in the server.
>
>
> > Reporting
> > --------------
> > The goals in g-m-p already support multiple reporting mechanisms
> > (SurefireReporter already implemented). The TestNG testcases will be
> > logged and reported by surefire anyway. So we have one reporting story
> > in place. Similar to the way g-m-p goals handles other reporter
> > implementations, we have to write a goal that will handle the TestNG
> > test exceptions for other reporters too.
>
> I believe that in general we will eventually want a reporting plugin,
> which can take a configurable set of "collectors" which can scrape
> build output (like surefire reports, pmd stats, clover results, etc)
> and then do something magical with them (like update a database, or
> inject the data into a reporting engine like pentaho)... and maybe
> one day when Maven has a built in rich reporting system we can use
> that instead... assuming its cooked enough.
>
> If we had such a plugin, then some of the reporter bits we added to g-
> m-p can be removed. g-m-p would only need to worry about dumping log
> files and maybe generating a properties file with more details. Then
> we would have a "collector" impl which would know how to take the g-m-
> p output files and massage them into a rich reporting system.
>
> I think the bulk of these collectors will take text inputs from
> existing plugins (ie. no mods to the plugin) and then generate some
> xml files, which would then be massaged by some reporting system
> specific handler, which may upload to a reporting server or insert
> into a database, yada yada yada.
>
> Build flow (for a given *-testsuite), probably ends up something like:
>
> generate build ticket (used to track all child module reports)
> do pre-integration-test (start-server, blah)
> for each module {
> run tests
> run plugins (like pmd, clover, etc)
> collect reports
> }
> do post-integration-test
> publish reports
>
> Published reports for a given run will all be indexed together by the
> build ticket, so it will be easy to see what results are for which
> build. In this example "generate build ticket", "collect reports"
> and "publish reports" would all be handled by the reporting plugin I
> am talking about, the rest is the existing testsuite bits.
>
> I think this will work out quite well... giving us flexibility w/o
> making our exiting plugins more complicated or waiting for other
> plugin developers to make modifications for us (which is not very
> likely to happen anyways).
>
> Anyways...
>
> I think that the reporters we have now (in g-m-p) is fine for a while
> though... but as soon as we need to start adding similar reporter-
> like functionality to other plugins, then we should really consider
> implementing a more generic reporting system that can function on
> plugin outputs alone (or using a maven reporting api if/when it
> becomes available).
>
> * * *
>
> Overall I think the testsuite work of late is positive movement for
> our community to be able to start building a more comprehensive set
> of integration and system tests... NOW.
>
> I believe that we do need to start working on making some real
> tests... so that we can validate that the framework we have now is
> sufficient, and if not refactor it now before it sits for too long.
>
> --jason
>
Re: Testsuite and maven-maven-plugin
Posted by Jason Dillon <ja...@planet57.com>.
On Sep 27, 2006, at 8:40 PM, Prasad Kashyap wrote:
> Custom packaging or lifecycle phase bindings
> ---------------------------------------------------------------------
> The geronimo-maven-plugin (g-m-p) goals like start/stop server,
> deploy/undeploy modules will be executed in each of these pom.xmls.
> The start-server and deploy-module goals will be bound to some early
> phase in the lifecycle, say "validate" (example only) using the
> @phase. The stop-server and undeploy-module goals will be bound to a
> later phase, say "install". The tests will be executed in the "test"
> phase in between the modules deploy and undeploy.
I think it is inappropriate to overload phases intended for a jar
package to be used for integration tests. As I have mentioned a few
times before, it is easy enough to create a new packaging which has
an appropriate set of phases, probably something like:
# validate
# generate-test-sources
# process-test-sources
# generate-test-resources
# process-test-resources
# test-compile
# pre-integration-test
# test
# post-integration-test
# verify
And then deploy-module is bound to pre-integration-test and undeploy-
module is bound to post-integration-test.
> In each *-testsuite, multiple modules can be deployed and a subset of
> tests can be run against them. Eg: ATest runs on module1, BTest runs
> on module2 and module3, CTest runs on module3 etc.
I believe we want something more along the lines of:
testsuite/
console-testsuite/
module1/
src/test/java/...
module2/
src/test/java/...
webcontainer-testsuite/
module1/
src/test/java/...
module2/
src/test/java/...
ejbcontainer-testsuite/
module1/
src/test/java/...
module2/
src/test/java/...
Where the *-testsuite poms contain the start-server, stop-server and
run each module* in a separate child maven execution with the maven-
maven-plugin. Each module can define a set of artifacts to deploy/
undeploy.
This allows different combinations of deployments to be tested
against the same running server instance. It also provides a
flexible setup to allow for better organization of tests and their
deployment configuration. And, since each module is executed in a
child maven process, the configuration of that execution is more
flexible and not limited to working under the same profile that was
used to start/stop the server.
This is very similar to how the current testsuite is already setup.
> Exception handling
> ----------------------------
> Only the exceptions from start-server and stop-server goals should
> stop the test run completely. Failures from deploy-modules and
> undeploy-modules should be logged and the test should continue.
> Failures from TestNG testcases should be logged and ignored. Tests
> should continue.
Fair enough.
> Cleanup will done by the stop-server goal at the end. The start-server
> configuration in each *-testsuite should have the <refresh> set to
> true. This will ensure that each suite will start with a clean slate.
While I agree this may be needed (to use <refresh>) hopefully our
tests will also function as expected with/out the refresh... as I
hope that undeploy-module will function. So, I would extend the
"cleanup" on to modules which deploy, then should also undeploy, so
that the next module run does not have the previous modules clutter
running in the server.
> Reporting
> --------------
> The goals in g-m-p already support multiple reporting mechanisms
> (SurefireReporter already implemented). The TestNG testcases will be
> logged and reported by surefire anyway. So we have one reporting story
> in place. Similar to the way g-m-p goals handles other reporter
> implementations, we have to write a goal that will handle the TestNG
> test exceptions for other reporters too.
I believe that in general we will eventually want a reporting plugin,
which can take a configurable set of "collectors" which can scrape
build output (like surefire reports, pmd stats, clover results, etc)
and then do something magical with them (like update a database, or
inject the data into a reporting engine like pentaho)... and maybe
one day when Maven has a built in rich reporting system we can use
that instead... assuming its cooked enough.
If we had such a plugin, then some of the reporter bits we added to g-
m-p can be removed. g-m-p would only need to worry about dumping log
files and maybe generating a properties file with more details. Then
we would have a "collector" impl which would know how to take the g-m-
p output files and massage them into a rich reporting system.
I think the bulk of these collectors will take text inputs from
existing plugins (ie. no mods to the plugin) and then generate some
xml files, which would then be massaged by some reporting system
specific handler, which may upload to a reporting server or insert
into a database, yada yada yada.
Build flow (for a given *-testsuite), probably ends up something like:
generate build ticket (used to track all child module reports)
do pre-integration-test (start-server, blah)
for each module {
run tests
run plugins (like pmd, clover, etc)
collect reports
}
do post-integration-test
publish reports
Published reports for a given run will all be indexed together by the
build ticket, so it will be easy to see what results are for which
build. In this example "generate build ticket", "collect reports"
and "publish reports" would all be handled by the reporting plugin I
am talking about, the rest is the existing testsuite bits.
I think this will work out quite well... giving us flexibility w/o
making our exiting plugins more complicated or waiting for other
plugin developers to make modifications for us (which is not very
likely to happen anyways).
Anyways...
I think that the reporters we have now (in g-m-p) is fine for a while
though... but as soon as we need to start adding similar reporter-
like functionality to other plugins, then we should really consider
implementing a more generic reporting system that can function on
plugin outputs alone (or using a maven reporting api if/when it
becomes available).
* * *
Overall I think the testsuite work of late is positive movement for
our community to be able to start building a more comprehensive set
of integration and system tests... NOW.
I believe that we do need to start working on making some real
tests... so that we can validate that the framework we have now is
sufficient, and if not refactor it now before it sits for too long.
--jason