You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by Vincent Massol <vm...@pivolis.com> on 2005/12/28 17:40:40 UTC
[discussion] Integration testing location
Hi there,
I'd like to continue the discussion about integration testing (see
http://docs.codehaus.org/display/MAVEN/Testing+Strategies). Here are some
topics on which I'd like to get your opinion:
1) Need for a pre/post phase to integration-test. See
http://jira.codehaus.org/browse/MNG-1880.
I guess this could be implemented very quickly. Isn't it just a matter of
adding 2 new phases in components.xml?
Do we all agree on the need for this?
2) I think there are use cases when we want to have the integration tests in
the same module as what they are testing. For example, take the case of an
ejb project. I think the project could have both unit tests and integration
tests in the same module. Functional tests is another matter and I'd usually
see them in a separate module altogether (as they usually require several
modules to be built).
ejb/
src/
main/
test/
java/ (unit tests - shouldn't be called java really)
it/ (integration tests)
I have a use case for the daytrader build where I need to do this. I'd like
to have some it tests next to some unit tests. The reason I want them in the
same module is because I want to have 1 or 2 it tests to act as a quick
proof that the ejb code works. It's not meant to replace functional tests
which I'm also doing but in some other module.
I don't think the surefire plugin support this right now. The problem is
that it's using configuration elements defined outside of the <plugin>
section in the pom so I don't think it's currently possible to have several
executions of it with different configurations.
Another option is to have an it plugin which is basically the same as the
surefire plugin but bound at different phases and using different
configuration elements.
Which solution do we want?
Note: the current it plugin is probably not correctly named as it makes
sense only for performing integration tests for m2 plugin.
WDYT?
Thanks
-Vincent
___________________________________________________________________________
Nouveau : t�l�phonez moins cher avec Yahoo! Messenger ! D�couvez les tarifs exceptionnels pour appeler la France et l'international.
T�l�chargez sur http://fr.messenger.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: [discussion] Integration testing location
Posted by Jesse McConnell <je...@gmail.com>.
On 12/28/05, Vincent Massol <vm...@pivolis.com> wrote:
>
>
>
> Sure but there's a very common use case for integration testing: the need
> to
> have environment setup before the test and to clean it after the tests. Of
> course you could write all sort of plugin to that the plugin support doing
> this itself but then you're no longer flexible and you're not providing a
> solution to lots of other use cases.
>
I think that was the jist of that irc conversation...is that still an
integration test or is it something else, later in a formal testing
strategy..
I know a lot of people approach things as
unit tests - test code in the subproject
integration tests - test subproject interactions, with stubbing of
environmental factors and formally testing interactions between the project
and other projects/dependent projects
system tests - full system testing, including tests against a deployed
system
I had to scrape around for the right term for that last testing stage
name...found it on wikipedia at the bottom of
http://en.wikipedia.org/wiki/Software_testing
http://en.wikipedia.org/wiki/System_testing is the actual page for it, but
the summary is:
System testing is testing conducted on a complete, integrated system to
> evaluate the system's compliance with its specified requirements. System
> testing falls within the scope of Black box testing, and as such, should
> require no knowledge of the inner design of the code or logic. System
> testing takes, as its input, all of the "integrated" software components
> that have successfully passed Integration testing and also the software
> system itself integrated with any applicable hardware system(s). The purpose
> of Integration testing is to detect any inconsistencies between the software
> units that are integrated together called assemblages or between any of the
> assemblages and hardware. System testing is more of a limiting type of
> testing, where it seeks to detect both defects within the
> "inter-assemblages" and also the system as a whole.
>
so that is where I am coming from.. :) it sounds like your example there
ought to qualify as a seperate phase all together..
jesse
--
jesse mcconnell
jesseDOTmcconnellATgmailDOTcom
Re: [discussion] Integration testing location
Posted by Jason van Zyl <ja...@maven.org>.
Jesse McConnell wrote:
> I worry a bit about mixing unit and integration tests generally...
>
> maybe we have the recommended case for them go into
> src/integration-test/java or something along those lines...
+1
When I said parallel to unit testing that's what I meant so that
everything was distinct.
> logically the structure src/test/java and src/test/it doesn't do it for me
> since it tests are probably written in java anyway, so that kinda breaks the
> spirit of 'src'/'type'/'language' convention we kind of have going..
--
jvz.
Jason van Zyl
jason at maven.org
http://maven.apache.org
Our achievements speak for themselves. What we have to keep track
of are our failures, discouragements and doubts. We tend to forget
the past difficulties, the many false starts, and the painful
groping. We see our past achievements as the end result of a
clean forward thrust, and our present difficulties as
signs of decline and decay.
-- Eric Hoffer, Reflections on the Human Condition
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: [discussion] Integration testing location
Posted by Jason van Zyl <ja...@maven.org>.
Steve Loughran wrote:
> Vincent Massol wrote:
>
> what we actually do at work is just run cruise control on a single box
> and collect the results. Its that result collection which becomes
> critical and hard on a big distributed system.
Just some other things to note FYI.
Continuum recently gained an addition from David Blevins (from the
Geronimo team) which allows Continuum to run across several machines. So
David essentially created a distributed test environment for the
Geronimo TCK. It's still in the early stages but is working quite well:
it uses ActiveMQ to move build definitions and information around.
There is also a tool called SysUnit at Codeahaus which Bob McWhirter
created for distributed testing. Specifically for distributed caching
products.
I have also recently talked to the guy behind ControlTier (Alex Honor),
which has been recently open sourced, and would like to see where we can
go with that (open.controltier.com). I talked with Vincent briefly about
ControlTier the other day. Alex has a vast wealth of knowledge regarding
deployment which for us is beneficial as Maven is the preferred tool to
use in conjunction with ControlTier which is a plus in terms of
integration with Maven.
--
jvz.
Jason van Zyl
jason at maven.org
http://maven.apache.org
A man enjoys hsi work when he understands the whole and when he
is responsible for the quality of the whole
-- Christopher Alexander, A Pattern Language
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: [discussion] Integration testing location
Posted by Steve Loughran <st...@apache.org>.
Vincent Massol wrote:
>
>> I am working with a PhD student at CERN on distributed testing, and
>> there is a project gridunit that does some good stuff already, running
>> junit tests across a farm of nodes, collecting and presenting the
>> results.
>
> Wow, that sounds really cool! Especially as there's a grid already in place
> (http://www.lsd.ufcg.edu.br/~alex/doku.php?id=gridunit).
yes, the grid is there. current gridunit can run a jar full of junit
tests across many machines, I/we want to extend that to do testing of
distributed systems, where you bring up some machines with your app then
run the tests from a long way away.
There is also planetlab (http://planetlab.org), which is a large soup of
random boxes out there, with random network QoS. Its there to let
students/researchers test on a real network, not a simulation.
>
> Is there an integration with a CI already in place? (I'd love to try it out
> for building Cargo - It has lots of tests and this would help a lot).
oh, no, nothing yet from my work. Talk to Alex about what he has done. I
am part way through a cargo component for smartfrog though, also on my
todo list is something to deploy cruise control itself.
what we actually do at work is just run cruise control on a single box
and collect the results. Its that result collection which becomes
critical and hard on a big distributed system.
>
> Some time back I had blogged about distributed build
> (http://tinyurl.com/42zc7). This is still something I'd love to do. I agree
> that distributed test is a different topic than distributed build but
> combining the two would be even greater... ;-)
I havent looked at distributed builds much. Smartfrog can actually
deploy any Ant task, so you could write your build in the language,
though nobody does; we just use the tasks to avoid writing so many
utilities ourselves, though when we do do the latter we get to implement
cleanup on undeploy.
-steve
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
RE: [discussion] Integration testing location
Posted by Vincent Massol <vm...@pivolis.com>.
Hi Steve,
> -----Original Message-----
> From: Steve Loughran [mailto:stevel@apache.org]
> Sent: mercredi 4 janvier 2006 14:37
> To: Maven Developers List
> Subject: Re: [discussion] Integration testing location
[snip]
> > For example cargo could define a cargo:test goal which would start the
> > container, run the tests and stop the containers. But this doesn't lead
> to
> > all variations like:
> > - the user simply wants to redeploy the new artifacts in the container
> but
> > not start the container again
> > - the user want to have his/her tests written using tetstng and not
> junit
> > (or any other test framework)
>
>
> Sounds more like functional testing to me...anything that needs a
> deployed system is a v. more complex situation.
Ah, I guess it depends on the definition of it vs functional tests. For me
it's only a question of context. For example, I have a project with wars,
ejbs, ears. I want to be able to do two things:
- Inside the ejb project, I want to be able to run unit tests but also
integration tests. I don't want to run many its. Just one or two that proves
that my ejb can be deployed and works. For that I want to use cargo without
having to define any configuration. Cargo is perfect for this use case as it
provides a default config for me. Still I call this IT and not functional
testing.
- I want to have a system/ project where I'll run functional tests (aka
system tests in this case) where I'll have all the different components
deployed (the ear and all). For this functional test, I want to run with the
real configuration I'll use for deployment.
> I am working with a PhD student at CERN on distributed testing, and
> there is a project gridunit that does some good stuff already, running
> junit tests across a farm of nodes, collecting and presenting the
> results.
Wow, that sounds really cool! Especially as there's a grid already in place
(http://www.lsd.ufcg.edu.br/~alex/doku.php?id=gridunit).
Is there an integration with a CI already in place? (I'd love to try it out
for building Cargo - It has lots of tests and this would help a lot).
Some time back I had blogged about distributed build
(http://tinyurl.com/42zc7). This is still something I'd love to do. I agree
that distributed test is a different topic than distributed build but
combining the two would be even greater... ;-)
> Clearly presenting the results gets more complex once you have
> tested on 20 boxes; you want to know what failed everywhere, what failed
> somewhere, and if there is any commonality for the partial failures.
>
> The perspective we are taking is that a test run is just another thing
> to deploy; you have a test listener to collect results and logs from
> across the machines, then test runners on different machines running
> different tests. The listener collects the results, post-processes them
> and then you can act on the outcome (report failures, host the reports,
> etc).
>
> In this view, functional testing of a deployment is just another
> deployment. Its different from a production deployment, but not very;
> just a different deployment descriptor to process.
Cool
-Vincent
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: [discussion] Integration testing location
Posted by Steve Loughran <st...@apache.org>.
Vincent Massol wrote:
>
>> -----Original Message-----
>> From: Jesse McConnell [mailto:jesse.mcconnell@gmail.com]
>> Sent: mercredi 28 décembre 2005 20:44
>> To: Maven Developers List
>> Subject: Re: [discussion] Integration testing location
>>
>> I worry a bit about mixing unit and integration tests generally...
>>
>> maybe we have the recommended case for them go into
>> src/integration-test/java or something along those lines...
>>
>> logically the structure src/test/java and src/test/it doesn't do it for me
>> since it tests are probably written in java anyway, so that kinda breaks
>> the
>> spirit of 'src'/'type'/'language' convention we kind of have going..
>
> Right. I hadn't seen it this way (I thought src/test was for all tests and
> that src/main was for all runtime sources) but I think you're right. I'm
> fine with src/it/java.
>
>> as for the rest of it, I like adding the phases...
>
> Sure but there's a very common use case for integration testing: the need to
> have environment setup before the test and to clean it after the tests. Of
> course you could write all sort of plugin to that the plugin support doing
> this itself but then you're no longer flexible and you're not providing a
> solution to lots of other use cases.
>
> For example cargo could define a cargo:test goal which would start the
> container, run the tests and stop the containers. But this doesn't lead to
> all variations like:
> - the user simply wants to redeploy the new artifacts in the container but
> not start the container again
> - the user want to have his/her tests written using tetstng and not junit
> (or any other test framework)
Sounds more like functional testing to me...anything that needs a
deployed system is a v. more complex situation.
I am working with a PhD student at CERN on distributed testing, and
there is a project gridunit that does some good stuff already, running
junit tests across a farm of nodes, collecting and presenting the
results. Clearly presenting the results gets more complex once you have
tested on 20 boxes; you want to know what failed everywhere, what failed
somewhere, and if there is any commonality for the partial failures.
The perspective we are taking is that a test run is just another thing
to deploy; you have a test listener to collect results and logs from
across the machines, then test runners on different machines running
different tests. The listener collects the results, post-processes them
and then you can act on the outcome (report failures, host the reports,
etc).
In this view, functional testing of a deployment is just another
deployment. Its different from a production deployment, but not very;
just a different deployment descriptor to process.
-Steve
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
RE: [discussion] Integration testing location
Posted by Vincent Massol <vm...@pivolis.com>.
> -----Original Message-----
> From: Jesse McConnell [mailto:jesse.mcconnell@gmail.com]
> Sent: mercredi 28 décembre 2005 20:44
> To: Maven Developers List
> Subject: Re: [discussion] Integration testing location
>
> I worry a bit about mixing unit and integration tests generally...
>
> maybe we have the recommended case for them go into
> src/integration-test/java or something along those lines...
>
> logically the structure src/test/java and src/test/it doesn't do it for me
> since it tests are probably written in java anyway, so that kinda breaks
> the
> spirit of 'src'/'type'/'language' convention we kind of have going..
Right. I hadn't seen it this way (I thought src/test was for all tests and
that src/main was for all runtime sources) but I think you're right. I'm
fine with src/it/java.
> as for the rest of it, I like adding the phases...
Sure but there's a very common use case for integration testing: the need to
have environment setup before the test and to clean it after the tests. Of
course you could write all sort of plugin to that the plugin support doing
this itself but then you're no longer flexible and you're not providing a
solution to lots of other use cases.
For example cargo could define a cargo:test goal which would start the
container, run the tests and stop the containers. But this doesn't lead to
all variations like:
- the user simply wants to redeploy the new artifacts in the container but
not start the container again
- the user want to have his/her tests written using tetstng and not junit
(or any other test framework)
It's not modular at all (and btw encourages duplication of code - between
surefire and cargo if cargo implemented a cargo:test goal).
> I remember the last time
> we talked about this there was a bit of concern that arose based on 'where
> do we stop?' in supporting different testing strategies...integration
> tests
> make sense to support in the lifecycle, but should integration tests
> embody
> the deployment of a generated ear and automated tests against a container
> started by the build system? or the running of tests generated by
> something
> like the axistools plugin that can reach out and test
> connectivity/functionality for a webservices client? I think there was
> general consensus on the integration testing phase, but some question on
> the
> later type of testing...and I think the formalization of the integration
> testing into maven then was delayed to consider if these other types of
> testing might prompt some reworking of the existing framework...
>
> there, I think I captured the jist of that discussion, it was on irc a few
> months ago..
Thanks
-Vincent
___________________________________________________________________________
Nouveau : téléphonez moins cher avec Yahoo! Messenger ! Découvez les tarifs exceptionnels pour appeler la France et l'international.
Téléchargez sur http://fr.messenger.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: [discussion] Integration testing location
Posted by Jesse McConnell <je...@gmail.com>.
I worry a bit about mixing unit and integration tests generally...
maybe we have the recommended case for them go into
src/integration-test/java or something along those lines...
logically the structure src/test/java and src/test/it doesn't do it for me
since it tests are probably written in java anyway, so that kinda breaks the
spirit of 'src'/'type'/'language' convention we kind of have going..
as for the rest of it, I like adding the phases...I remember the last time
we talked about this there was a bit of concern that arose based on 'where
do we stop?' in supporting different testing strategies...integration tests
make sense to support in the lifecycle, but should integration tests embody
the deployment of a generated ear and automated tests against a container
started by the build system? or the running of tests generated by something
like the axistools plugin that can reach out and test
connectivity/functionality for a webservices client? I think there was
general consensus on the integration testing phase, but some question on the
later type of testing...and I think the formalization of the integration
testing into maven then was delayed to consider if these other types of
testing might prompt some reworking of the existing framework...
there, I think I captured the jist of that discussion, it was on irc a few
months ago..
jesse
On 12/28/05, Vincent Massol <vm...@pivolis.com> wrote:
>
>
>
> > -----Original Message-----
> > From: Jason van Zyl [mailto:jason@maven.org]
> > Sent: mercredi 28 décembre 2005 19:51
> > To: Maven Developers List
> > Subject: Re: [discussion] Integration testing location
> >
> > Vincent Massol wrote:
> > > Hi there,
> > >
> > > I'd like to continue the discussion about integration testing (see
> > > http://docs.codehaus.org/display/MAVEN/Testing+Strategies). Here are
> > some
> > > topics on which I'd like to get your opinion:
> > >
> > > 1) Need for a pre/post phase to integration-test. See
> > > http://jira.codehaus.org/browse/MNG-1880.
> >
> > Is this going to be a general pattern in that we might want to decorate
> > the phases. I like having the concrete phases but maybe for genericity
> > we need to allow the decoration of each of our concrete phases or we may
> > simply run into further limitations that require changes to lifecycles.
> >
> > > I guess this could be implemented very quickly. Isn't it just a matter
> > of
> > > adding 2 new phases in components.xml?
> > >
> > > Do we all agree on the need for this?
> >
> > I think a general decoration mechanism would be useful. Even though this
> > could be somewhat unstructured I think the utility gained is probably
> > more important. So I would be
> >
> > +1
> >
> > Unless we could think of some serious downsides. Can't think of any off
> > the top of my head.
>
> +1 for implicit pre/post phases to for all phases.
>
> [snip]
>
> > > I don't think the surefire plugin support this right now. The problem
> is
> > > that it's using configuration elements defined outside of the <plugin>
> > > section in the pom so I don't think it's currently possible to have
> > several
> > > executions of it with different configurations.
> >
> > Right, the test classes directory is pulled from the <build/> element so
> > we would need to do something there for integration test classes to be
> > picked up.
> >
> > > Another option is to have an it plugin which is basically the same as
> > the
> > > surefire plugin but bound at different phases and using different
> > > configuration elements.
> > >
> > > Which solution do we want?
> >
> > What about a parallel for integration tests like we have for unit tests?
> >
> > If we are going to consider integration testing on the same level as
> > unit testing then we might want to add an integration source setting to
> > the <build/> element because we will need to compile the integration
> > tests which will be in a different directory and they will need to be
> > output to a different directory so that surefire can pick them up. So
> > there will be some values that will need to be accessed by several mojos
> > to make this work. So for each element we have in the POM and each step
> > in the lifecycle we have for testing we need for integration testing.
>
> Right. This is a good thing to do I think.
>
> Thanks
> -Vincent
>
>
>
>
>
>
>
> ___________________________________________________________________________
> Nouveau : téléphonez moins cher avec Yahoo! Messenger ! Découvez les
> tarifs exceptionnels pour appeler la France et l'international.
> Téléchargez sur http://fr.messenger.yahoo.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
--
jesse mcconnell
jesseDOTmcconnellATgmailDOTcom
RE: [discussion] Integration testing location
Posted by Vincent Massol <vm...@pivolis.com>.
> -----Original Message-----
> From: Jason van Zyl [mailto:jason@maven.org]
> Sent: mercredi 28 décembre 2005 19:51
> To: Maven Developers List
> Subject: Re: [discussion] Integration testing location
>
> Vincent Massol wrote:
> > Hi there,
> >
> > I'd like to continue the discussion about integration testing (see
> > http://docs.codehaus.org/display/MAVEN/Testing+Strategies). Here are
> some
> > topics on which I'd like to get your opinion:
> >
> > 1) Need for a pre/post phase to integration-test. See
> > http://jira.codehaus.org/browse/MNG-1880.
>
> Is this going to be a general pattern in that we might want to decorate
> the phases. I like having the concrete phases but maybe for genericity
> we need to allow the decoration of each of our concrete phases or we may
> simply run into further limitations that require changes to lifecycles.
>
> > I guess this could be implemented very quickly. Isn't it just a matter
> of
> > adding 2 new phases in components.xml?
> >
> > Do we all agree on the need for this?
>
> I think a general decoration mechanism would be useful. Even though this
> could be somewhat unstructured I think the utility gained is probably
> more important. So I would be
>
> +1
>
> Unless we could think of some serious downsides. Can't think of any off
> the top of my head.
+1 for implicit pre/post phases to for all phases.
[snip]
> > I don't think the surefire plugin support this right now. The problem is
> > that it's using configuration elements defined outside of the <plugin>
> > section in the pom so I don't think it's currently possible to have
> several
> > executions of it with different configurations.
>
> Right, the test classes directory is pulled from the <build/> element so
> we would need to do something there for integration test classes to be
> picked up.
>
> > Another option is to have an it plugin which is basically the same as
> the
> > surefire plugin but bound at different phases and using different
> > configuration elements.
> >
> > Which solution do we want?
>
> What about a parallel for integration tests like we have for unit tests?
>
> If we are going to consider integration testing on the same level as
> unit testing then we might want to add an integration source setting to
> the <build/> element because we will need to compile the integration
> tests which will be in a different directory and they will need to be
> output to a different directory so that surefire can pick them up. So
> there will be some values that will need to be accessed by several mojos
> to make this work. So for each element we have in the POM and each step
> in the lifecycle we have for testing we need for integration testing.
Right. This is a good thing to do I think.
Thanks
-Vincent
___________________________________________________________________________
Nouveau : téléphonez moins cher avec Yahoo! Messenger ! Découvez les tarifs exceptionnels pour appeler la France et l'international.
Téléchargez sur http://fr.messenger.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: [discussion] Integration testing location
Posted by Jason van Zyl <ja...@maven.org>.
Vincent Massol wrote:
> Hi there,
>
> I'd like to continue the discussion about integration testing (see
> http://docs.codehaus.org/display/MAVEN/Testing+Strategies). Here are some
> topics on which I'd like to get your opinion:
>
> 1) Need for a pre/post phase to integration-test. See
> http://jira.codehaus.org/browse/MNG-1880.
Is this going to be a general pattern in that we might want to decorate
the phases. I like having the concrete phases but maybe for genericity
we need to allow the decoration of each of our concrete phases or we may
simply run into further limitations that require changes to lifecycles.
> I guess this could be implemented very quickly. Isn't it just a matter of
> adding 2 new phases in components.xml?
>
> Do we all agree on the need for this?
I think a general decoration mechanism would be useful. Even though this
could be somewhat unstructured I think the utility gained is probably
more important. So I would be
+1
Unless we could think of some serious downsides. Can't think of any off
the top of my head.
> 2) I think there are use cases when we want to have the integration tests in
> the same module as what they are testing. For example, take the case of an
> ejb project. I think the project could have both unit tests and integration
> tests in the same module. Functional tests is another matter and I'd usually
> see them in a separate module altogether (as they usually require several
> modules to be built).
>
> ejb/
> src/
> main/
> test/
> java/ (unit tests - shouldn't be called java really)
> it/ (integration tests)
>
> I have a use case for the daytrader build where I need to do this. I'd like
> to have some it tests next to some unit tests. The reason I want them in the
> same module is because I want to have 1 or 2 it tests to act as a quick
> proof that the ejb code works. It's not meant to replace functional tests
> which I'm also doing but in some other module.
>
> I don't think the surefire plugin support this right now. The problem is
> that it's using configuration elements defined outside of the <plugin>
> section in the pom so I don't think it's currently possible to have several
> executions of it with different configurations.
Right, the test classes directory is pulled from the <build/> element so
we would need to do something there for integration test classes to be
picked up.
> Another option is to have an it plugin which is basically the same as the
> surefire plugin but bound at different phases and using different
> configuration elements.
>
> Which solution do we want?
What about a parallel for integration tests like we have for unit tests?
If we are going to consider integration testing on the same level as
unit testing then we might want to add an integration source setting to
the <build/> element because we will need to compile the integration
tests which will be in a different directory and they will need to be
output to a different directory so that surefire can pick them up. So
there will be some values that will need to be accessed by several mojos
to make this work. So for each element we have in the POM and each step
in the lifecycle we have for testing we need for integration testing.
> Note: the current it plugin is probably not correctly named as it makes
> sense only for performing integration tests for m2 plugin.
>
> WDYT?
>
> Thanks
> -Vincent
>
>
>
>
>
>
> ___________________________________________________________________________
> Nouveau : téléphonez moins cher avec Yahoo! Messenger ! Découvez les tarifs exceptionnels pour appeler la France et l'international.
> Téléchargez sur http://fr.messenger.yahoo.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
>
--
jvz.
Jason van Zyl
jason at maven.org
http://maven.apache.org
you are never dedicated to something you have complete confidence in.
No one is fanatically shouting that the sun is going to rise tomorrow.
They know it is going to rise tomorrow. When people are fanatically
dedicated to political or religious faiths or any other kind of
dogmas or goals, it's always because these dogmas or
goals are in doubt.
-- Robert Pirzig, Zen and the Art of Motorcycle Maintenance
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: [discussion] Integration testing location
Posted by Matt Brozowski <br...@opennms.org>.
On Dec 28, 2005, at 10:00 PM, Jason van Zyl wrote:
> Brett Porter wrote:
>
>> -1 to generic pre/post phases. For one, there's no difference between
>> pre-process-classes and post-compile.
>
> Maybe that indicates that the process-classes is really a decoration
> for the compile phase. We have several places in the lifecycle where
> the same argument applies. Are there phases that are more concrete
> then others i.e. like compile? Do we have them all? I don't know. We
> can easily change the lifecycle phases like we are proposing now but
> will we be continually doing that in the future.
Just to add my two cents to this discussion. It seems that the compile
and resources phases already have pre/post phases (although combined as
Jason suggests) and now we're discussing the need to add it to the
tests. That doesn't leave to many phases besides test except for
possibly package and deploy. But it seems like I've seen plenty of
discussions on the list that ask how that can do X right before they
and Y right before they deploy, etc.
As a result I would +1 this one.
Matt
________________________________________________________________________
___
Matt Brozowski, OpenNMS Maintainer Main: +1 919 812 4984
The OpenNMS Group, Inc. Fax: +1 503 961 7746
Email: brozow@opennms.org URL: http://www.opennms.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: [discussion] Integration testing location
Posted by Rinku <ra...@gmail.com>.
Hi,
Has this discussion thread reached a decision yet?
Jason's idea of pre/post phase decorators sounds great. My understanding was
that the decorators will be able to address cases like pre-pre-compile etc... -
so basically, upto the user to use 'n' number of phase decorators.
I think its a great feature to have in Maven. I'd suppose a change like this
would be targeted for 2.1, right?
Cheers,
Rahul
----- Original Message -----
From: "Jason van Zyl" <ja...@maven.org>
To: "Maven Developers List" <de...@maven.apache.org>
Sent: Thursday, December 29, 2005 4:04 PM
Subject: Re: [discussion] Integration testing location
> Brett Porter wrote:
>
>>
>> If we just do pre/post, won't the next point be that pre-pre-compile is
>> needed?
>
> I imagine that 90% of what people do will be covered by the standard
> lifecycle, 5% of people might make custom lifecycle and 5% might need to do
> some really weird stuff with a big pile of rope like the pre-pre-compile. I
> think the majority of people would use the standard lifecycle where possible
> and resort to the less encourage approached only where absolutely necessary.
> If all we have is two cases which might require lifecycle alterations then the
> pre/post decoration maybe not be required. But it's the things that we can't
> think about that we should try to account for. Sometimes you may not want to
> create a plugin to trigger a customized lifecycle for a simple decoration.
>
>> - Brett
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>>
>>
>
>
> --
>
> jvz.
>
> Jason van Zyl
> jason at maven.org
> http://maven.apache.org
>
> Simplex sigillum veri. (Simplicity is the seal of truth.)
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: [discussion] Integration testing location
Posted by Jason van Zyl <ja...@maven.org>.
Brett Porter wrote:
>
> If we just do pre/post, won't the next point be that pre-pre-compile is
> needed?
I imagine that 90% of what people do will be covered by the standard
lifecycle, 5% of people might make custom lifecycle and 5% might need to
do some really weird stuff with a big pile of rope like the
pre-pre-compile. I think the majority of people would use the standard
lifecycle where possible and resort to the less encourage approached
only where absolutely necessary. If all we have is two cases which might
require lifecycle alterations then the pre/post decoration maybe not be
required. But it's the things that we can't think about that we should
try to account for. Sometimes you may not want to create a plugin to
trigger a customized lifecycle for a simple decoration.
> - Brett
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
>
--
jvz.
Jason van Zyl
jason at maven.org
http://maven.apache.org
Simplex sigillum veri. (Simplicity is the seal of truth.)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: [discussion] Integration testing location
Posted by Brett Porter <br...@apache.org>.
Jason van Zyl wrote:
> Brett Porter wrote:
>
>> -1 to generic pre/post phases. For one, there's no difference between
>> pre-process-classes and post-compile.
>
> Maybe that indicates that the process-classes is really a decoration
> for the compile phase. We have several places in the lifecycle where
> the same argument applies. Are there phases that are more concrete
> then others i.e. like compile? Do we have them all? I don't know. We
> can easily change the lifecycle phases like we are proposing now but
> will we be continually doing that in the future.
You're right - and I think what we actually have now is a set of
concrete phases, and one phase in between that is effectively the
mergere of pre/post of the proceeding and current.
I doubt we'd be changing anything continually, and once something is
there we certainly can't remove it.
>
> So far it only seems to be the testing scenerios that are driving this
> discussion. Has there ever been anything else that has been mentioned
> which would require a change to the lifecycle? Don't get me wrong, I
> love the concrete lifecycle. I just don't want it to be a continual
> limitation either.
I think the only other use case I've heard is generating resources that
generate sources. We knew the testing capacity was limited, but I don't
recall hearing much outside of that.
>
> It would just be a simpler way to bind mojos as decorations in the POM
> as opposed to creating a lifecycle file.
The separate lifecycle file is more about completely customising it
(even changing the order, dangerous though I'm sure that is).
If we just do pre/post, won't the next point be that pre-pre-compile is
needed?
- Brett
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: [discussion] Integration testing location
Posted by Jason van Zyl <ja...@maven.org>.
Brett Porter wrote:
> -1 to generic pre/post phases. For one, there's no difference between
> pre-process-classes and post-compile.
Maybe that indicates that the process-classes is really a decoration for
the compile phase. We have several places in the lifecycle where the
same argument applies. Are there phases that are more concrete then
others i.e. like compile? Do we have them all? I don't know. We can
easily change the lifecycle phases like we are proposing now but will we
be continually doing that in the future.
So far it only seems to be the testing scenerios that are driving this
discussion. Has there ever been anything else that has been mentioned
which would require a change to the lifecycle? Don't get me wrong, I
love the concrete lifecycle. I just don't want it to be a continual
limitation either.
I think the other phases are
> already granular enough. "It can't hurt" is probably not a good
> justification :)
As with the lifecycle specification, not something encouraged but allows
someone in a bind to do something they need to do.
FWIW, I'm pretty sure you can specify your own
> lifecycle now, though it remains discouraged and undocumented.
It would just be a simpler way to bind mojos as decorations in the POM
as opposed to creating a lifecycle file.
> +1 to renaming the maven-it-plugin to the maven-plugin-it-plugin or
> something similar. Or even the maven-embedder-plugin if it is that generic.
>
> Cheers,
> Brett
>
> Vincent Massol wrote:
>> Hi there,
>>
>> I'd like to continue the discussion about integration testing (see
>> http://docs.codehaus.org/display/MAVEN/Testing+Strategies). Here are some
>> topics on which I'd like to get your opinion:
>>
>> 1) Need for a pre/post phase to integration-test. See
>> http://jira.codehaus.org/browse/MNG-1880.
>>
>> I guess this could be implemented very quickly. Isn't it just a matter of
>> adding 2 new phases in components.xml?
>>
>> Do we all agree on the need for this?
>>
>> 2) I think there are use cases when we want to have the integration tests in
>> the same module as what they are testing. For example, take the case of an
>> ejb project. I think the project could have both unit tests and integration
>> tests in the same module. Functional tests is another matter and I'd usually
>> see them in a separate module altogether (as they usually require several
>> modules to be built).
>>
>> ejb/
>> src/
>> main/
>> test/
>> java/ (unit tests - shouldn't be called java really)
>> it/ (integration tests)
>>
>> I have a use case for the daytrader build where I need to do this. I'd like
>> to have some it tests next to some unit tests. The reason I want them in the
>> same module is because I want to have 1 or 2 it tests to act as a quick
>> proof that the ejb code works. It's not meant to replace functional tests
>> which I'm also doing but in some other module.
>>
>> I don't think the surefire plugin support this right now. The problem is
>> that it's using configuration elements defined outside of the <plugin>
>> section in the pom so I don't think it's currently possible to have several
>> executions of it with different configurations.
>>
>> Another option is to have an it plugin which is basically the same as the
>> surefire plugin but bound at different phases and using different
>> configuration elements.
>>
>> Which solution do we want?
>>
>> Note: the current it plugin is probably not correctly named as it makes
>> sense only for performing integration tests for m2 plugin.
>>
>> WDYT?
>>
>> Thanks
>> -Vincent
>>
>>
>>
>>
>>
>>
>> ___________________________________________________________________________
>> Nouveau : téléphonez moins cher avec Yahoo! Messenger ! Découvez les tarifs exceptionnels pour appeler la France et l'international.
>> Téléchargez sur http://fr.messenger.yahoo.com
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
>
--
jvz.
Jason van Zyl
jason at maven.org
http://maven.apache.org
Simplex sigillum veri. (Simplicity is the seal of truth.)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: [discussion] Integration testing location
Posted by Brett Porter <br...@apache.org>.
Sorry to top reply, am trying to capture a response to several emails in
the thread:
+1 to src/it/java, and a reproduction of all test phases for integration
testing
We may want to have a functional test set as well that is separate to
integration testing (I know there a bunch of different names, but most
everyone seems to agree there are 3 types).
If there are 3 types, maybe we need an abstraction in the lifecycle to
say "repeat this set of steps" rather than copy pasting them all
+1 to setup/teardown phases on any of the test parts
-1 to generic pre/post phases. For one, there's no difference between
pre-process-classes and post-compile. I think the other phases are
already granular enough. "It can't hurt" is probably not a good
justification :) FWIW, I'm pretty sure you can specify your own
lifecycle now, though it remains discouraged and undocumented.
+1 to renaming the maven-it-plugin to the maven-plugin-it-plugin or
something similar. Or even the maven-embedder-plugin if it is that generic.
Cheers,
Brett
Vincent Massol wrote:
> Hi there,
>
> I'd like to continue the discussion about integration testing (see
> http://docs.codehaus.org/display/MAVEN/Testing+Strategies). Here are some
> topics on which I'd like to get your opinion:
>
> 1) Need for a pre/post phase to integration-test. See
> http://jira.codehaus.org/browse/MNG-1880.
>
> I guess this could be implemented very quickly. Isn't it just a matter of
> adding 2 new phases in components.xml?
>
> Do we all agree on the need for this?
>
> 2) I think there are use cases when we want to have the integration tests in
> the same module as what they are testing. For example, take the case of an
> ejb project. I think the project could have both unit tests and integration
> tests in the same module. Functional tests is another matter and I'd usually
> see them in a separate module altogether (as they usually require several
> modules to be built).
>
> ejb/
> src/
> main/
> test/
> java/ (unit tests - shouldn't be called java really)
> it/ (integration tests)
>
> I have a use case for the daytrader build where I need to do this. I'd like
> to have some it tests next to some unit tests. The reason I want them in the
> same module is because I want to have 1 or 2 it tests to act as a quick
> proof that the ejb code works. It's not meant to replace functional tests
> which I'm also doing but in some other module.
>
> I don't think the surefire plugin support this right now. The problem is
> that it's using configuration elements defined outside of the <plugin>
> section in the pom so I don't think it's currently possible to have several
> executions of it with different configurations.
>
> Another option is to have an it plugin which is basically the same as the
> surefire plugin but bound at different phases and using different
> configuration elements.
>
> Which solution do we want?
>
> Note: the current it plugin is probably not correctly named as it makes
> sense only for performing integration tests for m2 plugin.
>
> WDYT?
>
> Thanks
> -Vincent
>
>
>
>
>
>
> ___________________________________________________________________________
> Nouveau : téléphonez moins cher avec Yahoo! Messenger ! Découvez les tarifs exceptionnels pour appeler la France et l'international.
> Téléchargez sur http://fr.messenger.yahoo.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org