You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by Vincent Massol <vm...@pivolis.com> on 2005/10/30 22:37:44 UTC
Packaging type for functional/integration test projects?
Hi,
Is there a <packaging> type for projects that are only executing
integration/functional tests and not generating any artifact? If I use a pom
packaging there are several issues:
- the eclipse plugin doesn't generate eclipse project files for me
- more importantly the tests that I currently put in src/tests/java are not
executed.
So I think we should have a "integration-test" packaging that would map the
test directory to be src/it (for example).
Obviously another solution would be to use the it plugin but it doesn't do
what I need. I need to write my test logic above the execution of the m2
projects so that I can do some setup and perform possibly complex assertions
that wouldn't be supported by the verifier plugin for example).
Thanks
-Vincent
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
[m2] Testing strategy and conventions (was RE: Packaging type for functional/integration test projects?)
Posted by Vincent Massol <vm...@pivolis.com>.
> -----Original Message-----
> From: Kenney Westerhof [mailto:kenney@apache.org]
> Sent: lundi 31 octobre 2005 17:17
> To: Maven Developers List
> Subject: RE: Packaging type for functional/integration test projects?
[snip]
> Sorry, my mistake. I meant src/it/[project]/.
>
> > that there are 2 types of sources:
> > - the one that will be packaged and executed at runtime. They go in
> src/main
> > - the one that are tests (any kind of test and test resources). They go
> in
> > src/test.
>
> Yes.
>
> > Thus for example:
> >
> > src/test
> > |_ java (unit tests)
> > |_ it (or integration or functional for functional tests)
>
> I believe functional tests fall in another category dimension.
> We have 2 axes:
>
> Unit test --------- Integration Test,
>
> and orthogonally to that:
>
> Basic test --------- Functional test
>
> Unit/integration test depends on wheter sub-components of a project,
> or the entire project, possibly in a larger 'environment' is being tested.
> Basic/Functional test is internal/external api testing. At least that's
> what I've come to understand at my last job. :)
>
> > |_ groovy (for groovy tests)
> > |_ cactus (for cactus tests)
> > |_ resources (for tests resources)
> > |_ resources-it (for it test resources - for example)
>
> I don't think we should separate sources depending on testing framework.
> Since they have to be defined in the pom anyway (the plugins for that),
> you can just use src/test/ or src/it for that, and if you need multiple
> tests (are they guaranteed to work on different source sets?) you can
> always manually add directories...
What is sure is that the m2 testing strategy is a real mess.
I think we have several options:
Solution 1:
============
Decide that we do not mix unit and integration/functional tests in the same
maven project. In that case, we can have a simply directory structure:
src/test/
|_ java/
|_ resources/
Which will work both for unit, integration/functional tests.
<side note>I don't think we need to differentiate integration and functional
tests</side note>
So the best practice would be to tell users that if they have
integration/functional tests they should put them in a separate subproject.
In any case this is almost always the case for functional tests because they
require some running environment.
Solution 2:
============
Try to cope with all possible situation and have a more complex directory
structure so that users can have unit tests alongside integration/functional
tests in the same project.
That would lead to something like:
A)
src/test/
|_ unit/
|_ junit/
|_ resources-junit/
|_ groovy/
|_ resources-groovy/
|_ [...]
|_ integration/
|_ cactus/
|_ [...]
|_ functional/
Or
B)
src/test/
|_ unit/
|_ java/
|_ resources/
|_ integration/ or functional/ (or both)
|_ java/
|_ resources/
That would mean that wouldn't mix tests written using different tools and
belong to the same testing category (unit, integration, functional) in the
same project. That should be ok though. A separate project would be required
if needed.
However I personally still don't like src/it as it *are* tests and this you
cannot have src/test and src/it as directory siblings.
[snip]
> > I still don't understand why you want to have src/it.
>
> Integration tests! Multiple possible projects to run the code of the
> 'main' project (including runtime dependencies) against. Say,
> maven-core-it/it*/ -> src/it/*, and for the maven-eclipse-plugin:
> src/test/projects/project-X -> src/itest/project-X and drop src/test/.
It seems you're not talking about integration tests but about integration
test resources. I would put them in src/integration/resources (see option B
above).
[snip]
I'll provide more feedback to your other questions when I have progressed
more on my cargo m2 tests.
Thanks
-Vincent
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
RE: Packaging type for functional/integration test projects?
Posted by Kenney Westerhof <ke...@apache.org>.
On Mon, 31 Oct 2005, Vincent Massol wrote:
Hi Vincent,
>
> I'm not sure.... Have a look at the POM lifecycle. You'll see it doesn't do
> much. Of course I have been able to solve my need by adding to the POM
> lifecycle:
[snip manually adding compiler/surefire]
> But that's a bit heavyweight and I think integration or functional tests is
> common-enough use case for offering a built-in solution for it.
Agreed. M2 itself uses 'jar' packaging for integration tests. The only
difference is possibly that those projects don't need to be
installed/deployed, but otherwise are the same as normal jar projects.
So you might have a good point here.
> > > - more importantly the tests that I currently put in src/tests/java are
> > not
> > > executed.
> >
> > If this is a typo: either use src/test/java or src/it/java (if the it
> > plugin is finished ;))
>
> I don't understand why you are promoting src/it/java. My understanding is
Sorry, my mistake. I meant src/it/[project]/.
> that there are 2 types of sources:
> - the one that will be packaged and executed at runtime. They go in src/main
> - the one that are tests (any kind of test and test resources). They go in
> src/test.
Yes.
> Thus for example:
>
> src/test
> |_ java (unit tests)
> |_ it (or integration or functional for functional tests)
I believe functional tests fall in another category dimension.
We have 2 axes:
Unit test --------- Integration Test,
and orthogonally to that:
Basic test --------- Functional test
Unit/integration test depends on wheter sub-components of a project,
or the entire project, possibly in a larger 'environment' is being tested.
Basic/Functional test is internal/external api testing. At least that's
what I've come to understand at my last job. :)
> |_ groovy (for groovy tests)
> |_ cactus (for cactus tests)
> |_ resources (for tests resources)
> |_ resources-it (for it test resources - for example)
I don't think we should separate sources depending on testing framework.
Since they have to be defined in the pom anyway (the plugins for that),
you can just use src/test/ or src/it for that, and if you need multiple
tests (are they guaranteed to work on different source sets?) you can
always manually add directories...
> What we definitely need are naming conventions that we would all agree on
> but this will come when plugins are there.
Right.
> I still don't understand why you want to have src/it.
Integration tests! Multiple possible projects to run the code of the
'main' project (including runtime dependencies) against. Say,
maven-core-it/it*/ -> src/it/*, and for the maven-eclipse-plugin:
src/test/projects/project-X -> src/itest/project-X and drop src/test/.
> > If this is NOT a typo: weird, I didn't know specifying packaging pom
> > didn't run any tests.
>
> No it doesn't. Again have a look at the pom packaging lifecycle. You'll see
> that test compilation and test execution are not bound.
I'm lazy.. ;))
> > How about just specify packaging 'jar' for now?
>
> I'm not generating a jar. My current solution is to use pom and augment its
> lifecycle as shown above.
>
> > > So I think we should have a "integration-test" packaging that would map
> > the
> > > test directory to be src/it (for example).
> >
> > If the above are resolved I don't know why.. it's a phase. It's weird
> > enough that the test phase is also a valid attachment, but since that road
> > was taken, I guess integration-test could be a valid attachment too.
> >
> > > Obviously another solution would be to use the it plugin but it doesn't
> > do
> > > what I need. I need to write my test logic above the execution of the m2
> > > projects so that I can do some setup and perform possibly complex
> > assertions
> > > that wouldn't be supported by the verifier plugin for example).
> >
> > You were going to give me some feedback about this :)
>
> Yep. Here's some feedback in this email. I'll provide more in a few days.
> :-)
Great, thanks ;)
> > I think the it plugin does exactly this. You specify tests per project (or
> > even 'inherit' them using dependencies) so you can do whatever you want in
> > the test phase - check for created files, etc.. You can even bind stuff to
> > the integration test phase of the integration test projects so you'll have
> > a finished artifact to run tests on.
>
> Ok, I may have misunderstood the it plugin. Could you tell me how do I
> control the setup of a test using the it plugin? Let's imagine I want to
> dynamically change a m2 property before the tests is executed or that I want
> to execute several goals depending on some code logic. How do I do that?
>
> The example you've listed above seem to be linked to assertion about
> results. This is the quivalent of an after() advice in AOP-AspectJ-parlance.
> What I'm after is the equivalent of an around() advice. I've been able to
> get this easily using the embedder.
>
> Can this be done with the it plugin?
Not yet, because I didn't see the need for it yet. I believe that you can
specify all settings/parameters in the pom itself (pre() advice), and can
write after() advice as unit tests.
> > You can specify the configuration in the pom - the only official api to
> > communicate with plugins. You don't need to write unit test code just to
> > set up a plugin. You can always write a custom 'verifier' plugin and bind
> > that to your integration test phase, doing the complex assertions.
> >
> > So why isn't the it plugin good enough? Too complex, too repetitive work
> > (configuring different pom.xml's?) Not flexible enough?
>
> See above.
- Changing an m2 property.. which one, why?
- Run several goals: just specify them in the pom as defaultGoal or
bind some non-lifecycle mojo's to the lifecyle, or specify the goal to
run in the it plugin's config (currently it's global - one 'goal' for
all projects, but the defaultGoal might help here).
I agree that the it plugin could be augmented so this will be easier.
> > As for the assertions, I was thinking about porting some tests from the
> > verifier plugin (the maven-core-it one) to the it plugin, but I need to
> > know what kind of assertions you want to do.
> >
> > Hope to get some feedback!
>
> Yes, I'll give you some more detailed use cases once I have progressed a bit
> more. Give me 1 or 2 days more! I'm working on it today and tomorrow.
Ok, good luck!
>
> Thanks
> -Vincent
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
--
Kenney Westerhof
http://www.neonics.com
GPG public key: http://www.gods.nl/~forge/kenneyw.key
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
RE: Packaging type for functional/integration test projects?
Posted by Vincent Massol <vm...@pivolis.com>.
Hi Kenney,
> -----Original Message-----
> From: Kenney Westerhof [mailto:kenney@apache.org]
> Sent: lundi 31 octobre 2005 11:46
> To: Maven Developers List
> Subject: Re: Packaging type for functional/integration test projects?
>
> On Sun, 30 Oct 2005, Vincent Massol wrote:
>
> Hi,
>
> > Hi,
> >
> > Is there a <packaging> type for projects that are only executing
> > integration/functional tests and not generating any artifact? If I use a
> pom
> > packaging there are several issues:
>
> Since they don't produce artifacts pom is appropriate here..
I'm not sure.... Have a look at the POM lifecycle. You'll see it doesn't do
much. Of course I have been able to solve my need by adding to the POM
lifecycle:
<build>
<testSourceDirectory>src/test/it</testSourceDirectory>
<!-- We need to force the compiler plugin to compile tests and the
surefire plugin to execute them because we're using a pom packaging
that doesn't have those mapped in its lifecycle.
A better solution would be for m2 to define a default
integration-test lifecycle -->
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
But that's a bit heavyweight and I think integration or functional tests is
common-enough use case for offering a built-in solution for it.
> > - the eclipse plugin doesn't generate eclipse project files for me
>
> That's because eclipse can't handle nested projects, and for now that is
> solved by not producing projects with packaging pom. However when you
> specify -Declipse.workspace= (or I believe someone changed that param to
> be just 'workspace' - why?) it WILL generate pom projects.
>
> I guess the logic should be changed to detect 'leaf' projects and warn
> about skipped non-leaf projects.
Yes, agreed.
> > - more importantly the tests that I currently put in src/tests/java are
> not
> > executed.
>
> If this is a typo: either use src/test/java or src/it/java (if the it
> plugin is finished ;))
I don't understand why you are promoting src/it/java. My understanding is
that there are 2 types of sources:
- the one that will be packaged and executed at runtime. They go in src/main
- the one that are tests (any kind of test and test resources). They go in
src/test.
Thus for example:
src/test
|_ java (unit tests)
|_ it (or integration or functional for functional tests)
|_ groovy (for groovy tests)
|_ cactus (for cactus tests)
|_ resources (for tests resources)
|_ resources-it (for it test resources - for example)
What we definitely need are naming conventions that we would all agree on
but this will come when plugins are there.
I still don't understand why you want to have src/it.
> If this is NOT a typo: weird, I didn't know specifying packaging pom
> didn't run any tests.
No it doesn't. Again have a look at the pom packaging lifecycle. You'll see
that test compilation and test execution are not bound.
> How about just specify packaging 'jar' for now?
I'm not generating a jar. My current solution is to use pom and augment its
lifecycle as shown above.
> > So I think we should have a "integration-test" packaging that would map
> the
> > test directory to be src/it (for example).
>
> If the above are resolved I don't know why.. it's a phase. It's weird
> enough that the test phase is also a valid attachment, but since that road
> was taken, I guess integration-test could be a valid attachment too.
>
> > Obviously another solution would be to use the it plugin but it doesn't
> do
> > what I need. I need to write my test logic above the execution of the m2
> > projects so that I can do some setup and perform possibly complex
> assertions
> > that wouldn't be supported by the verifier plugin for example).
>
> You were going to give me some feedback about this :)
Yep. Here's some feedback in this email. I'll provide more in a few days.
:-)
> I think the it plugin does exactly this. You specify tests per project (or
> even 'inherit' them using dependencies) so you can do whatever you want in
> the test phase - check for created files, etc.. You can even bind stuff to
> the integration test phase of the integration test projects so you'll have
> a finished artifact to run tests on.
Ok, I may have misunderstood the it plugin. Could you tell me how do I
control the setup of a test using the it plugin? Let's imagine I want to
dynamically change a m2 property before the tests is executed or that I want
to execute several goals depending on some code logic. How do I do that?
The example you've listed above seem to be linked to assertion about
results. This is the quivalent of an after() advice in AOP-AspectJ-parlance.
What I'm after is the equivalent of an around() advice. I've been able to
get this easily using the embedder.
Can this be done with the it plugin?
> You can specify the configuration in the pom - the only official api to
> communicate with plugins. You don't need to write unit test code just to
> set up a plugin. You can always write a custom 'verifier' plugin and bind
> that to your integration test phase, doing the complex assertions.
>
> So why isn't the it plugin good enough? Too complex, too repetitive work
> (configuring different pom.xml's?) Not flexible enough?
See above.
> As for the assertions, I was thinking about porting some tests from the
> verifier plugin (the maven-core-it one) to the it plugin, but I need to
> know what kind of assertions you want to do.
>
> Hope to get some feedback!
Yes, I'll give you some more detailed use cases once I have progressed a bit
more. Give me 1 or 2 days more! I'm working on it today and tomorrow.
Thanks
-Vincent
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Packaging type for functional/integration test projects?
Posted by Kenney Westerhof <ke...@apache.org>.
On Sun, 30 Oct 2005, Vincent Massol wrote:
Hi,
> Hi,
>
> Is there a <packaging> type for projects that are only executing
> integration/functional tests and not generating any artifact? If I use a pom
> packaging there are several issues:
Since they don't produce artifacts pom is appropriate here..
> - the eclipse plugin doesn't generate eclipse project files for me
That's because eclipse can't handle nested projects, and for now that is
solved by not producing projects with packaging pom. However when you
specify -Declipse.workspace= (or I believe someone changed that param to
be just 'workspace' - why?) it WILL generate pom projects.
I guess the logic should be changed to detect 'leaf' projects and warn
about skipped non-leaf projects.
> - more importantly the tests that I currently put in src/tests/java are not
> executed.
If this is a typo: either use src/test/java or src/it/java (if the it
plugin is finished ;))
If this is NOT a typo: weird, I didn't know specifying packaging pom
didn't run any tests.
How about just specify packaging 'jar' for now?
> So I think we should have a "integration-test" packaging that would map the
> test directory to be src/it (for example).
If the above are resolved I don't know why.. it's a phase. It's weird
enough that the test phase is also a valid attachment, but since that road
was taken, I guess integration-test could be a valid attachment too.
> Obviously another solution would be to use the it plugin but it doesn't do
> what I need. I need to write my test logic above the execution of the m2
> projects so that I can do some setup and perform possibly complex assertions
> that wouldn't be supported by the verifier plugin for example).
You were going to give me some feedback about this :)
I think the it plugin does exactly this. You specify tests per project (or
even 'inherit' them using dependencies) so you can do whatever you want in
the test phase - check for created files, etc.. You can even bind stuff to
the integration test phase of the integration test projects so you'll have
a finished artifact to run tests on.
You can specify the configuration in the pom - the only official api to
communicate with plugins. You don't need to write unit test code just to
set up a plugin. You can always write a custom 'verifier' plugin and bind
that to your integration test phase, doing the complex assertions.
So why isn't the it plugin good enough? Too complex, too repetitive work
(configuring different pom.xml's?) Not flexible enough?
As for the assertions, I was thinking about porting some tests from the
verifier plugin (the maven-core-it one) to the it plugin, but I need to
know what kind of assertions you want to do.
Hope to get some feedback!
-- Kenney
>
> Thanks
> -Vincent
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
--
Kenney Westerhof
http://www.neonics.com
GPG public key: http://www.gods.nl/~forge/kenneyw.key
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org