You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geronimo.apache.org by Prasad Kashyap <go...@gmail.com> on 2006/01/29 15:12:28 UTC
Test strategy
On the list of Dain's "Roamap, tasks and things to do" list,
integration tests that cover servlets, webservices and jms was quite
at the top.
http://www.mail-archive.com/dev@geronimo.apache.org/msg16593.html
I would like to propose the following test strategy.
1. Fix existing openejb itests such that they can be reintegrated back
into the builds.
2. Create a testing subproject first in the sandbox that can be used
to test other services and components. This subproject will use maven
2. Integration test is already a phase in the Maven2 lifecycle.
3. Create a new deployment plugin for maven 2 for use by the testing
subproject. A new one is needed because the openejb itests still use
maven 1.
4. As the test suite grows, we can move the subproject from sandbox to
the main branch/trunk. Hopefully by then, openejb will have come out
of it's incubator status for it's itests to join the rest.
What say you all ? Comments, advice, suggestions more than welcome.
Cheers
Prasad.
Re: Test strategy
Posted by David Blevins <da...@visi.com>.
On Jan 30, 2006, at 7:43 PM, Prasad Kashyap wrote:
> I would like to solicit the views of others too like you and get their
> views and opinions.
Great. I poked some other people in irc to get involved as well.
Here is hoping.... :) More input is going to make for a better
community supported solution.
>>> - The tests can all be run or only a subset of suites can be run
>>> quite easily.
>
>> When you say "easily" do you mean by virtue of them being separate or
>> some other idea?
>
> When I said we could run a subset of tests "easily" when the itests
> are run "on" assemblies, I meant that it would be relatively easier to
> selectively include/exclude tests (by categories) than when compared
> to doing the same if the itests are embedded with their modules.
I think I follow what you mean. I guess the haze is in what you mean
by "embedded."
As I'm not sure, I will just state in more specific terms that I have
never argued for putting the actual testing source code inside any
particular assembly module -- this code is big, bulky, has a lot of
deps itself and some pretty specific packaging needs. I have argued
for simply running the big test suite during the lifecycle of an
assembly.
Not sure if that is coming across well, or if it's being assumed that
since I've recommended running the integration tests on an assembly,
during the integration test phase of it's lifecycle, that the test
source of course must be in the assembly as well. I believe we agree
on keeping the test source in modules separate and organized.
>>> - The tests will be grouped logically into suites or categories.
>>> Each
>>> individual test will fall in one of these suites.
>
>> Not sure of your usage of the word "test" in this sentence. In junit
>> terms is that TestSuite, TestCase, or individual test method. Can
>> you clarify?
>
> I meant TestCase. For. eg, we could have a TestSuite called, "System"
> which will have test cases from many different modules. We could say
> that these form the core set of tests. If the System suite passed, we
> have somewhat of a stable binary with the caveat that all functions
> may not work.
That is much clearer to me, thanks. I do like the sort of parent
module name you came up with of "System" too -- it's a keeper.
At this point, I will note that it is possible put a single
SuperServerSystemTest.java TestSuite in the SuperServer assemblies
module (for example) that adds only the tests from the many different
test modules that apply to that assembly.
> In summary, the cons of running the itests "on" assemblies is that it
> goes against the m2 lifecycle. And even though integration-test is an
> m2 lifecycle phase, we are not exploiting it's usage but calling it
> explicity again. The same has to be done for the deploy phase too.
You captured that quite well.
It occurred to me while writing a sentence above that there is no
clear distinction between "on" and "during". In all cases you are
quite literally running the tests on the assembly. The better
distinction is whether or not you wish to run them "during" it's
lifecycle or "after" in the lifecycle of a separate module.
-David
Re: Test strategy
Posted by Prasad Kashyap <go...@gmail.com>.
David,
Thanks for letting me make that decision. I, at this point, would now
have to get the insight and information from people who are more
knowledgeable about the project; something similiar to what you have
provided now. With my limited knowledge, I don't want to go down the
wrong path.
I would like to solicit the views of others too like you and get their
views and opinions.
>> - The tests can all be run or only a subset of suites can be run
>> quite easily.
> When you say "easily" do you mean by virtue of them being separate or
> some other idea?
When I said we could run a subset of tests "easily" when the itests
are run "on" assemblies, I meant that it would be relatively easier to
selectively include/exclude tests (by categories) than when compared
to doing the same if the itests are embedded with their modules.
>> - The tests will be grouped logically into suites or categories. Each
>> individual test will fall in one of these suites.
> Not sure of your usage of the word "test" in this sentence. In junit
> terms is that TestSuite, TestCase, or individual test method. Can
> you clarify?
I meant TestCase. For. eg, we could have a TestSuite called, "System"
which will have test cases from many different modules. We could say
that these form the core set of tests. If the System suite passed, we
have somewhat of a stable binary with the caveat that all functions
may not work.
In summary, the cons of running the itests "on" assemblies is that it
goes against the m2 lifecycle. And even though integration-test is an
m2 lifecycle phase, we are not exploiting it's usage but calling it
explicity again. The same has to be done for the deploy phase too.
Thanks again
Prasad.
On 1/30/06, David Blevins <da...@visi.com> wrote:
> It's probably going to be you doing this work, so ultimately, you are
> the one who decides what you want to write. I will just say that you
> and I share the same goals, but your proposal goes against the grain
> of maven which fundamentally combines build/test. Anytime you do
> that, it's painful.
>
> So let me just stop proposing and do my best to provide the
> information you'd need to get something done.
>
> On Jan 30, 2006, at 11:09 AM, Prasad Kashyap wrote:
> > 1. Build and Test will be kept separate functionally.
> >
> > 2. The itests 'subproject' can scale better. The testbucket could
> > become potentially large as more modules start adding their own itests
> > or increasing the number of test scenarios.
>
> A point of clarification. There is no technical limit to
> modularizing and building a set of tests in one area and reusing the
> artifacts produced by them elsewhere. In other words, where you keep
> the itests does not dictate where you run them. It is possible to
> build tests in maven 2 and run them in maven 1. The reverse could be
> true if a maven 2 version of the deployment plugin was created.
>
> > 3. The throughput of the itests will not determine the throughput of
> > the G build. Even if we do fix the openejb itests in G now to make it
> > very fast, when we have a potentially large number of itests
> > contributed by different folks for different modules, they tend to add
> > up.
>
> There are logically two different "build times" to consider. I would
> consider Developer build time to be more precious, and Automated
> built time to be less precious. Making the running of itests
> optional to developers is mandatory. Making the itests off by
> default may be a good idea.
>
> Either way, there is no technical limitation to achieving the above
> goals and running itest "on" or "during" the assemblies.
>
> > 4. It will be difficult to selectively include/exclude a few tests.
>
> This is where you pick your poison so to speak. Not all itest will
> apply to each assembly and vice versa. Meaning, there are only two
> ways to skin this cat.
>
> 1. You make each assembly say which tests apply to it
> 2. You make each set of tests say which assemblies apply to it
>
> With #1 you will have challenges specifying the tests to run. You
> will have to deploy several apps and run distinct sets of tests.
> With three assemblies and four distinct sets of tests (for example),
> you will unpack and start each server once and deploy into it four
> times (essentially). A total time of 3 unpacks and startups and 12
> deploys.
>
> With #2 you will have challenges specifying on which assemblies the
> distinct set of tests should run against. The distinct set of tests
> will have to unpack and start each assembly to run its set of tests.
> With four distinct sets of tests and three assemblies (same example),
> you will unpack, start and deploy on each assembly once for that set
> of tests. A total of 12 unpacks, starts and deploys will be required.
>
> > This is how I'd imagined it, at a very high level.
> > - The itests subproject will have it's own set of modules. These are
> > the modules in G that have dropped their itests into this subproject.
>
> Totally possible.
>
> > - The tests will be grouped logically into suites or categories. Each
> > individual test will fall in one of these suites.
>
> Not sure of your usage of the word "test" in this sentence. In junit
> terms is that TestSuite, TestCase, or individual test method. Can
> you clarify?
>
> > - The tests can all be run or only a subset of suites can be run
> > quite easily.
>
> When you say "easily" do you mean by virtue of them being separate or
> some other idea?
>
> > - Continuum will continue to build G first and then run itests on the
> > assemblies.
> > - If the assembly passes a certain set of tests, then it can be
> > certified good and then be uploaded to the repository.
>
> The maven 2 lifecycle supports this flow on a per module basis
> (truncated, but still in order):
>
> compile
> test-compile
> test
> package
> before-integration-test
> integration-test
> after-integration-test
> verify
> install
> deploy
>
> If there is a failure the next phase of the lifecycle is not called.
> So if "test" fails, "package" is not called. If "integration-test"
> fails then "install" or "deploy" are not called.
>
> Note, there are no "integration-test-complile" or "integration-
> test-package"
> phases as it is expected these exist in other modules.
>
> The life cycle of assembly SuperServer could work in maven 2 as follows:
>
> Geronimo :: Assembly :: Super Server
>
> compile (no op)
> test-compile (no op)
> test (no op)
> package build the assembly
> before-integration-test unpack and start the server, deploy
> required itest components
> integration-test run all itests applicable for this
> assembly
> after-integration-test stop server
> verify run any additional checks to verify the package is
> valid and meets quality criteria.
> install put in local maven repo
> deploy upload to public repo
>
>
> If you wish to split out integration-test from the maven lifecycle
> and delay publishing the SuperServer binary, you would need at least
> three modules to achieve the same.
>
> Geronimo :: Assembly :: Super Server
>
> compile (no op)
> test-compile (no op)
> test (no op)
> package build the assembly
> before-integration-test (no op)
> integration-test (no op)
> after-integration-test (no op)
> verify (no op)
> install put in local maven repo
> deploy WARNING: don't call this our it will be upload to
> public repo without testing
>
> Geronimo :: iTest :: Super Server
>
> compile (no op)
> test-compile (no op)
> test (no op)
> package (no op)
> before-integration-test unpack and start the SuperServer,
> deploy required itest components
> integration-test run all itests applicable for this
> assembly
> after-integration-test stop the SuperServer
> verify run any additional checks to verify the package is
> valid and meets quality criteria.
> install (no op)
> deploy (no op)
>
> Geronimo :: Deploy :: Super Server
>
> compile (no op)
> test-compile (no op)
> test (no op)
> package (no op)
> before-integration-test (no op)
> integration-test (no op)
> after-integration-test (no op)
> verify devise some way to verify the itests have run
> successfully on this exact SuperServer binary
> install (no op)
> deploy upload to public repo
>
> You will have to do a lot of tricking on maven to do things like
> this, but we've proven that with enough hacking you can push maven
> against it's grain.
>
> > - a download page will be available to download assemblies and
> > binaries from every single build. The builds that passed itests will
> > be clearly marked so.
>
> Great.
>
>
> -David
>
Re: Test strategy
Posted by David Blevins <da...@visi.com>.
It's probably going to be you doing this work, so ultimately, you are
the one who decides what you want to write. I will just say that you
and I share the same goals, but your proposal goes against the grain
of maven which fundamentally combines build/test. Anytime you do
that, it's painful.
So let me just stop proposing and do my best to provide the
information you'd need to get something done.
On Jan 30, 2006, at 11:09 AM, Prasad Kashyap wrote:
> 1. Build and Test will be kept separate functionally.
>
> 2. The itests 'subproject' can scale better. The testbucket could
> become potentially large as more modules start adding their own itests
> or increasing the number of test scenarios.
A point of clarification. There is no technical limit to
modularizing and building a set of tests in one area and reusing the
artifacts produced by them elsewhere. In other words, where you keep
the itests does not dictate where you run them. It is possible to
build tests in maven 2 and run them in maven 1. The reverse could be
true if a maven 2 version of the deployment plugin was created.
> 3. The throughput of the itests will not determine the throughput of
> the G build. Even if we do fix the openejb itests in G now to make it
> very fast, when we have a potentially large number of itests
> contributed by different folks for different modules, they tend to add
> up.
There are logically two different "build times" to consider. I would
consider Developer build time to be more precious, and Automated
built time to be less precious. Making the running of itests
optional to developers is mandatory. Making the itests off by
default may be a good idea.
Either way, there is no technical limitation to achieving the above
goals and running itest "on" or "during" the assemblies.
> 4. It will be difficult to selectively include/exclude a few tests.
This is where you pick your poison so to speak. Not all itest will
apply to each assembly and vice versa. Meaning, there are only two
ways to skin this cat.
1. You make each assembly say which tests apply to it
2. You make each set of tests say which assemblies apply to it
With #1 you will have challenges specifying the tests to run. You
will have to deploy several apps and run distinct sets of tests.
With three assemblies and four distinct sets of tests (for example),
you will unpack and start each server once and deploy into it four
times (essentially). A total time of 3 unpacks and startups and 12
deploys.
With #2 you will have challenges specifying on which assemblies the
distinct set of tests should run against. The distinct set of tests
will have to unpack and start each assembly to run its set of tests.
With four distinct sets of tests and three assemblies (same example),
you will unpack, start and deploy on each assembly once for that set
of tests. A total of 12 unpacks, starts and deploys will be required.
> This is how I'd imagined it, at a very high level.
> - The itests subproject will have it's own set of modules. These are
> the modules in G that have dropped their itests into this subproject.
Totally possible.
> - The tests will be grouped logically into suites or categories. Each
> individual test will fall in one of these suites.
Not sure of your usage of the word "test" in this sentence. In junit
terms is that TestSuite, TestCase, or individual test method. Can
you clarify?
> - The tests can all be run or only a subset of suites can be run
> quite easily.
When you say "easily" do you mean by virtue of them being separate or
some other idea?
> - Continuum will continue to build G first and then run itests on the
> assemblies.
> - If the assembly passes a certain set of tests, then it can be
> certified good and then be uploaded to the repository.
The maven 2 lifecycle supports this flow on a per module basis
(truncated, but still in order):
compile
test-compile
test
package
before-integration-test
integration-test
after-integration-test
verify
install
deploy
If there is a failure the next phase of the lifecycle is not called.
So if "test" fails, "package" is not called. If "integration-test"
fails then "install" or "deploy" are not called.
Note, there are no "integration-test-complile" or "integration-
test-package"
phases as it is expected these exist in other modules.
The life cycle of assembly SuperServer could work in maven 2 as follows:
Geronimo :: Assembly :: Super Server
compile (no op)
test-compile (no op)
test (no op)
package build the assembly
before-integration-test unpack and start the server, deploy
required itest components
integration-test run all itests applicable for this
assembly
after-integration-test stop server
verify run any additional checks to verify the package is
valid and meets quality criteria.
install put in local maven repo
deploy upload to public repo
If you wish to split out integration-test from the maven lifecycle
and delay publishing the SuperServer binary, you would need at least
three modules to achieve the same.
Geronimo :: Assembly :: Super Server
compile (no op)
test-compile (no op)
test (no op)
package build the assembly
before-integration-test (no op)
integration-test (no op)
after-integration-test (no op)
verify (no op)
install put in local maven repo
deploy WARNING: don't call this our it will be upload to
public repo without testing
Geronimo :: iTest :: Super Server
compile (no op)
test-compile (no op)
test (no op)
package (no op)
before-integration-test unpack and start the SuperServer,
deploy required itest components
integration-test run all itests applicable for this
assembly
after-integration-test stop the SuperServer
verify run any additional checks to verify the package is
valid and meets quality criteria.
install (no op)
deploy (no op)
Geronimo :: Deploy :: Super Server
compile (no op)
test-compile (no op)
test (no op)
package (no op)
before-integration-test (no op)
integration-test (no op)
after-integration-test (no op)
verify devise some way to verify the itests have run
successfully on this exact SuperServer binary
install (no op)
deploy upload to public repo
You will have to do a lot of tricking on maven to do things like
this, but we've proven that with enough hacking you can push maven
against it's grain.
> - a download page will be available to download assemblies and
> binaries from every single build. The builds that passed itests will
> be clearly marked so.
Great.
-David
Re: Test strategy
Posted by Prasad Kashyap <go...@gmail.com>.
Following the brief discussion we began having on irc, this is what I
believe you mean and are proposing -
that itests should be run (on modules), during the time of an assembly
creation. The key word here is "during". This is also why itests
should use the same version of maven that assembly plugin also uses.
I imagined this quite differently. I thought and still think we should
run the itests "on" the assemblies, outside the assembly process. Here
are a few reasons -
1. Build and Test will be kept separate functionally.
2. The itests 'subproject' can scale better. The testbucket could
become potentially large as more modules start adding their own itests
or increasing the number of test scenarios.
3. The throughput of the itests will not determine the throughput of
the G build. Even if we do fix the openejb itests in G now to make it
very fast, when we have a potentially large number of itests
contributed by different folks for different modules, they tend to add
up.
4. It will be difficult to selectively include/exclude a few tests.
This is how I'd imagined it, at a very high level.
- The itests subproject will have it's own set of modules. These are
the modules in G that have dropped their itests into this subproject.
- The tests will be grouped logically into suites or categories. Each
individual test will fall in one of these suites.
- The tests can all be run or only a subset of suites can be run quite easily.
- Continuum will continue to build G first and then run itests on the
assemblies.
- If the assembly passes a certain set of tests, then it can be
certified good and then be uploaded to the repository.
- a download page will be available to download assemblies and
binaries from every single build. The builds that passed itests will
be clearly marked so.
What do you think ?
Cheers
Prasad
On 1/30/06, Prasad Kashyap <go...@gmail.com> wrote:
> David,
>
> Can you please help clarify the following points
>
> You wrote, "...but in each assemblies module after an assembly is
> built is where they need to execute..". I'm sorry, I quite didn't
> catch what that meant.
>
> Next, if the assemblies are assembled using maven 1.. do they then
> have to deployed using maven 1 only ? Can't they be deployed using m2
> ?
>
> Cheers
> Prasad
>
> On 1/29/06, David Blevins <da...@visi.com> wrote:
> > On Jan 29, 2006, at 6:12 AM, Prasad Kashyap wrote:
> >
> > > On the list of Dain's "Roamap, tasks and things to do" list,
> > > integration tests that cover servlets, webservices and jms was quite
> > > at the top.
> > > http://www.mail-archive.com/dev@geronimo.apache.org/msg16593.html
> > >
> > > I would like to propose the following test strategy.
> > >
> > > 1. Fix existing openejb itests such that they can be reintegrated back
> > > into the builds.
> > >
> > > 2. Create a testing subproject first in the sandbox that can be used
> > > to test other services and components. This subproject will use maven
> > > 2. Integration test is already a phase in the Maven2 lifecycle.
> > >
> > > 3. Create a new deployment plugin for maven 2 for use by the testing
> > > subproject. A new one is needed because the openejb itests still use
> > > maven 1.
> > >
> > > 4. As the test suite grows, we can move the subproject from sandbox to
> > > the main branch/trunk. Hopefully by then, openejb will have come out
> > > of it's incubator status for it's itests to join the rest.
> > >
> > > What say you all ? Comments, advice, suggestions more than welcome.
> > >
> >
> > This is a good initative. Couple notes off the top of my head.
> >
> > A clarification on step 1 would be that we need to run the itest on
> > each assembly we create. Don't mind if they are off by default and
> > run only when we build in continuum, but in each assemblies module
> > after an assembly is built is where they need to execute.
> >
> > If we wanted to build the new itests in maven 2, that'd be ok. We
> > just need to make sure they run on our assemblies, which use maven
> > 1. We wouldn't need a maven 2 version of our deployment plugin
> > unless we started building the assemblies with maven 2.
> >
> > If more working itests start showing up, i'd be fine with an itests
> > module next to applications, assemblies, configs, etc.. We'd just
> > want to make sure that the test we add are actually running with the
> > main build and not just a bunch of test-to-get-working-someday like
> > David J. and I did when we tried to create itests about this time in
> > '04.
> >
> > Thoughts?
> >
> > -David
> >
> >
>
Re: Test strategy
Posted by Prasad Kashyap <go...@gmail.com>.
David,
Can you please help clarify the following points
You wrote, "...but in each assemblies module after an assembly is
built is where they need to execute..". I'm sorry, I quite didn't
catch what that meant.
Next, if the assemblies are assembled using maven 1.. do they then
have to deployed using maven 1 only ? Can't they be deployed using m2
?
Cheers
Prasad
On 1/29/06, David Blevins <da...@visi.com> wrote:
> On Jan 29, 2006, at 6:12 AM, Prasad Kashyap wrote:
>
> > On the list of Dain's "Roamap, tasks and things to do" list,
> > integration tests that cover servlets, webservices and jms was quite
> > at the top.
> > http://www.mail-archive.com/dev@geronimo.apache.org/msg16593.html
> >
> > I would like to propose the following test strategy.
> >
> > 1. Fix existing openejb itests such that they can be reintegrated back
> > into the builds.
> >
> > 2. Create a testing subproject first in the sandbox that can be used
> > to test other services and components. This subproject will use maven
> > 2. Integration test is already a phase in the Maven2 lifecycle.
> >
> > 3. Create a new deployment plugin for maven 2 for use by the testing
> > subproject. A new one is needed because the openejb itests still use
> > maven 1.
> >
> > 4. As the test suite grows, we can move the subproject from sandbox to
> > the main branch/trunk. Hopefully by then, openejb will have come out
> > of it's incubator status for it's itests to join the rest.
> >
> > What say you all ? Comments, advice, suggestions more than welcome.
> >
>
> This is a good initative. Couple notes off the top of my head.
>
> A clarification on step 1 would be that we need to run the itest on
> each assembly we create. Don't mind if they are off by default and
> run only when we build in continuum, but in each assemblies module
> after an assembly is built is where they need to execute.
>
> If we wanted to build the new itests in maven 2, that'd be ok. We
> just need to make sure they run on our assemblies, which use maven
> 1. We wouldn't need a maven 2 version of our deployment plugin
> unless we started building the assemblies with maven 2.
>
> If more working itests start showing up, i'd be fine with an itests
> module next to applications, assemblies, configs, etc.. We'd just
> want to make sure that the test we add are actually running with the
> main build and not just a bunch of test-to-get-working-someday like
> David J. and I did when we tried to create itests about this time in
> '04.
>
> Thoughts?
>
> -David
>
>
Re: Test strategy
Posted by David Blevins <da...@visi.com>.
On Jan 29, 2006, at 6:12 AM, Prasad Kashyap wrote:
> On the list of Dain's "Roamap, tasks and things to do" list,
> integration tests that cover servlets, webservices and jms was quite
> at the top.
> http://www.mail-archive.com/dev@geronimo.apache.org/msg16593.html
>
> I would like to propose the following test strategy.
>
> 1. Fix existing openejb itests such that they can be reintegrated back
> into the builds.
>
> 2. Create a testing subproject first in the sandbox that can be used
> to test other services and components. This subproject will use maven
> 2. Integration test is already a phase in the Maven2 lifecycle.
>
> 3. Create a new deployment plugin for maven 2 for use by the testing
> subproject. A new one is needed because the openejb itests still use
> maven 1.
>
> 4. As the test suite grows, we can move the subproject from sandbox to
> the main branch/trunk. Hopefully by then, openejb will have come out
> of it's incubator status for it's itests to join the rest.
>
> What say you all ? Comments, advice, suggestions more than welcome.
>
This is a good initative. Couple notes off the top of my head.
A clarification on step 1 would be that we need to run the itest on
each assembly we create. Don't mind if they are off by default and
run only when we build in continuum, but in each assemblies module
after an assembly is built is where they need to execute.
If we wanted to build the new itests in maven 2, that'd be ok. We
just need to make sure they run on our assemblies, which use maven
1. We wouldn't need a maven 2 version of our deployment plugin
unless we started building the assemblies with maven 2.
If more working itests start showing up, i'd be fine with an itests
module next to applications, assemblies, configs, etc.. We'd just
want to make sure that the test we add are actually running with the
main build and not just a bunch of test-to-get-working-someday like
David J. and I did when we tried to create itests about this time in
'04.
Thoughts?
-David