You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by Brett Porter <br...@apache.org> on 2007/01/04 06:32:50 UTC
Re: Integration Testing
Jason - any further thoughts on this?
On 18/12/2006, at 4:35 PM, Brett Porter wrote:
>
> On 12/12/2006, at 5:08 AM, Jason van Zyl wrote:
>
>> So, in response to John's email: I think we need to settle on what
>> we're going to use and stop writing new tools. We have several
>> invokers, several verifiers, several IT plugins, and the plugin
>> harness. It is simply out of control. We have different methods
>> being used in different plugins, nothing is standard and it's
>> going to kill us. The most prevalent tool, as defective as it
>> might be is what is being used in the ITs themselves. Stephane
>> managed to use this successfully in some of his plugins. Then
>> after that we have an array of usages. What should happen before
>> we start writing more stuff is to figure out something we can use
>> now, and how to merge what we have together instead of writing
>> more tools.
>
> Agreed. I think that's what John was getting at too, but by doing
> it clean rather than rewriting something that was in use somewhere.
> So the work left to do, in either case, is apply it consistently
> and get rid of the stuff we aren't going to use.
>
> To me, it looks like:
> - the plugin-testing-harness needs to go. They should be
> integration tests that use a proper pom, or use pure mocks rather
> than the stubs that tend to just have a bunch of impossible-to-get-
> under-real-condition values.
> - John's test tools have the most complete invocation options, and
> tools for managing repositories that we can reuse, so I'd opt for
> that in that area
> - the verifier is well utilised, so if that is merged with the code
> from the verifier plugin then we can lose the invocation stuff and
> repository management stuff and merge it all together
>
> Can we have a wiki page with this work list? Or, can we check in
> the omni outliner files + an export for the non-mac users to review?
>
> [snip points I agree with]
>
>> - [+] The ITs should be in a project of their own so that we can
>> reuse them across versions of Maven. We could actually run
>> new versions of integration tests against old versions of
>> Maven. Solution: the ITs are now in a separate build and it
>> is possible to run them
>
> How should this play out in plugins? I would be in favour of
> separate projects for these.
>
> Two things that are must haves for me:
> - integration tests / anything that forks a Maven instance must
> *not* be part of the normal test build for a plugin. They take way
> too long, and lead to use of maven.test.skip :(
> - as far as integration testing in general, I think we recommend a
> separate project, but enable them to be part of the same project
> for simplicity of development (this is more specific to things like
> web applications where it is probably beneficial).
>
>> - [ ] We should be able to easily integrate the IT into a larger
>> run where we can use forked or embedded execution.
>
> Not sure what you mean here. What is an IT that *doesn't* use
> forked or embedded execution?
>
>> - [ ] automate the testing of ITs submitted by users
>
> what does this mean? I think, like a patch, a submitted IT still
> needs to be reviewed, and incorporated into the main test suite
> (with corresponding fix-for version so it only runs when it is
> expected to work). Otherwise, long term, we'll have lots of
> duplicated or poorly conceived ITs. I've seen test cases submitted
> that are quite useful at demonstrating something, but contain half
> of the user's proect which is not good for our long term scalability.
>
>> - [ ] Each IT should have its own repository if it needs
>> resources
>> from repository. We can't mess with a users repository when
>> testing.
>> - [ ] We need to have a file system based remote repository for
>> testing
>
> Agreed. Isn't that what John's tool already does?
>
>> - [ ] We need to standardize on integration testing in
>> general. We
>> have people going all over the place and it's a disaster.
>> - [ ] We have too many IT plugins (3)
>> - [ ] We have too many invokers (5)
>> - [ ] We have too many verifiers (3)
>
> Let's specifically get these mapped out and a path forward so that
> everyone can push towards it, rather than relying on you to do the
> work.
>
>> - [ ] The ITs should run nicely from an IDE. Solution: this does
>> work but requires that you run mvn clean
>> resources:testResources first as the IDE doesn't know
>> how to
>> set that up. Needs to be fully fixed. But it is much nicer
>> running this stuff in your IDE.
>
> Agree with the point, but not sure what you are referring to about
> testResources - the generated projects for IDEA and I think Eclipse
> already do this (and obviously better integration will bring it).
>
> I think the dangerous thing is using resources for non-classpath
> resources. It's better for the tests to setup and use a clean
> project instance for an IT itself (using helper tools).
>
> [snip specific notes on old ITs that need to be updated]
>
>> - [ ] artifactIds should be aligned with directories
>
> Agreed. Also, as I did in the last test I wrote, what do we think
> about using purposeful names rather than numbers for integration
> tests?
>
> Thanks for this.
>
> Cheers,
> Brett
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Integration Testing
Posted by John Casey <ca...@gmail.com>.
my only recommendation would be to take a look at the maven-assembly-plugin
(SVN:
http://svn.apache.org/repos/asf/maven/plugins/trunk/maven-assembly-plugin)
and go for something like that...
On 4/16/07, Barrie Treloar <ba...@gmail.com> wrote:
>
> On 4/17/07, John Casey <ca...@gmail.com> wrote:
> > Jerome, that link contains information on the strategy I've been using
> in
> > the assembly plugin, and it seems to work well (although it's a bit
> slower,
> > since it's spawning so many Maven builds). I'm hoping to converge the
> tools
> > we're using in the core ITs with those in the
> maven-component-test-plugin
> > (used to be maven-plugin-test-plugin), and the maven-invoker-plugin, and
> > arrive at some single solution (a plugin) that we can use
> everywhere...but I
> > haven't even been able to come up with a unified feature list yet.
>
> So, does anyone have a recommended approach so I can move forward in
> the mean time?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
Re: Integration Testing
Posted by Barrie Treloar <ba...@gmail.com>.
On 4/17/07, John Casey <ca...@gmail.com> wrote:
> Jerome, that link contains information on the strategy I've been using in
> the assembly plugin, and it seems to work well (although it's a bit slower,
> since it's spawning so many Maven builds). I'm hoping to converge the tools
> we're using in the core ITs with those in the maven-component-test-plugin
> (used to be maven-plugin-test-plugin), and the maven-invoker-plugin, and
> arrive at some single solution (a plugin) that we can use everywhere...but I
> haven't even been able to come up with a unified feature list yet.
So, does anyone have a recommended approach so I can move forward in
the mean time?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Integration Testing
Posted by John Casey <ca...@gmail.com>.
Jerome, that link contains information on the strategy I've been using in
the assembly plugin, and it seems to work well (although it's a bit slower,
since it's spawning so many Maven builds). I'm hoping to converge the tools
we're using in the core ITs with those in the maven-component-test-plugin
(used to be maven-plugin-test-plugin), and the maven-invoker-plugin, and
arrive at some single solution (a plugin) that we can use everywhere...but I
haven't even been able to come up with a unified feature list yet.
-john
On 4/15/07, Jerome Lacoste <je...@gmail.com> wrote:
>
> On 2/24/07, Brett Porter <br...@apache.org> wrote:
> > On 04/01/2007, at 4:32 PM, Brett Porter wrote:
> >
> > > Jason - any further thoughts on this?
> >
> > Ping... No is a valid answer :)
> >
> > I'd like to get your summary put somewhere individuals can pick
> > things off to work on - probably a jira project for shared. WDYT?
> >
> > I'm overcommitted for working on things right now, but I know a
> > couple of people are confused about the IT testing, and we've got all
> > those chronically broken plugins. Volunteers?
>
> I want to improve the state of IT for the mojos I work with (in
> particular in the mojo project). I am particularly interested in doing
> it in an sort of official/standard way.
>
>
> - [ ] We need to standardize on integration testing in general. We
> have people going all over the place and it's a disaster.
> - [ ] We have too many IT plugins (3)
> - [ ] We have too many invokers (5)
> - [ ] We have too many verifiers (3)
>
> I've found this:
> http://maven.apache.org/developers/committer-testing-plugins.html
>
> Does that represent the current IT strategy 'standard' ? Was that
> written after the point above was resolved ?
>
> Cheers,
>
> Jerome
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
Re: Integration Testing
Posted by Jerome Lacoste <je...@gmail.com>.
On 2/24/07, Brett Porter <br...@apache.org> wrote:
> On 04/01/2007, at 4:32 PM, Brett Porter wrote:
>
> > Jason - any further thoughts on this?
>
> Ping... No is a valid answer :)
>
> I'd like to get your summary put somewhere individuals can pick
> things off to work on - probably a jira project for shared. WDYT?
>
> I'm overcommitted for working on things right now, but I know a
> couple of people are confused about the IT testing, and we've got all
> those chronically broken plugins. Volunteers?
I want to improve the state of IT for the mojos I work with (in
particular in the mojo project). I am particularly interested in doing
it in an sort of official/standard way.
- [ ] We need to standardize on integration testing in general. We
have people going all over the place and it's a disaster.
- [ ] We have too many IT plugins (3)
- [ ] We have too many invokers (5)
- [ ] We have too many verifiers (3)
I've found this:
http://maven.apache.org/developers/committer-testing-plugins.html
Does that represent the current IT strategy 'standard' ? Was that
written after the point above was resolved ?
Cheers,
Jerome
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Integration Testing
Posted by Barrie Treloar <ba...@gmail.com>.
On 2/24/07, Jason van Zyl <ja...@maven.org> wrote:
> > I'd like to get your summary put somewhere individuals can pick
> > things off to work on - probably a jira project for shared. WDYT?
> >
>
> I put the summary here for now, it's pretty elaborate and can most
> likely be worked on easily. Given one or two people have expressed
> interested or worked on the ITs I think that will suffice.
>
> http://svn.apache.org/repos/asf/maven/core-integration-testing/trunk/
> ITProblems.txt
>
Is there something that describes how to setup IT tests for plugins
and how to go about it?
I'm trying to use TDD to verify that the plugin I am testing, is
correctly attaching the write output to the project. via setArtifact
and addAttachedArtifact.
I'm not sure how to go about doing this.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Integration Testing
Posted by Jason van Zyl <ja...@maven.org>.
On 23 Feb 07, at 10:08 PM 23 Feb 07, Brett Porter wrote:
> On 04/01/2007, at 4:32 PM, Brett Porter wrote:
>
>> Jason - any further thoughts on this?
>
> Ping... No is a valid answer :)
>
> I'd like to get your summary put somewhere individuals can pick
> things off to work on - probably a jira project for shared. WDYT?
>
I put the summary here for now, it's pretty elaborate and can most
likely be worked on easily. Given one or two people have expressed
interested or worked on the ITs I think that will suffice.
http://svn.apache.org/repos/asf/maven/core-integration-testing/trunk/
ITProblems.txt
The top things that could be done:
1) The issues that would be most helpful that could be tackled on a
piecewise basis by many would be to take plugin specific ITs out of
the ITs. There are many in there for the surefire plugin so while
you're doing that you can look at it. Piecewise but probably totally
simple because you have to replace it with an IT that actually tests
what it was testing. A lot of time I have had to make a new IT plugin
flavour.
2) The next issue of importance would be to collect all the in IT
plugin plugins, invokers and verifiers and align all theses.
3) Once 2) is done then we wire the embedder option into the
resulting invoker.
As far as them working in situ: the ITs could now run from the top
level of http://svn.apache.org/repos/asf/maven/core-integration-
testing/trunk/ with an addition to the POM. It's not there because I
generally build from the top-level and then walk into http://
svn.apache.org/repos/asf/maven/core-integration-testing/trunk/core-
integration-tests/. The invoker plugin has been released and that's
what I used to install the supporting artifacts because they can't
install in a reactor run normally because there are 5 artifacts with
the id.
I am not entirely happy with the structure we have but I don't think
it's worth changing right now until 1) and 2) are done. It's not
obvious how to run them from and IDE and that's where I've found it
to be most convenient to run them.
> I'm overcommitted for working on things right now, but I know a
> couple of people are confused about the IT testing, and we've got
> all those chronically broken plugins. Volunteers?
>
>> On 18/12/2006, at 4:35 PM, Brett Porter wrote:
>>
>>>
>>> On 12/12/2006, at 5:08 AM, Jason van Zyl wrote:
>>>
>>>> So, in response to John's email: I think we need to settle on
>>>> what we're going to use and stop writing new tools. We have
>>>> several invokers, several verifiers, several IT plugins, and the
>>>> plugin harness. It is simply out of control. We have different
>>>> methods being used in different plugins, nothing is standard and
>>>> it's going to kill us. The most prevalent tool, as defective as
>>>> it might be is what is being used in the ITs themselves.
>>>> Stephane managed to use this successfully in some of his
>>>> plugins. Then after that we have an array of usages. What should
>>>> happen before we start writing more stuff is to figure out
>>>> something we can use now, and how to merge what we have together
>>>> instead of writing more tools.
>>>
>>> Agreed. I think that's what John was getting at too, but by doing
>>> it clean rather than rewriting something that was in use
>>> somewhere. So the work left to do, in either case, is apply it
>>> consistently and get rid of the stuff we aren't going to use.
>>>
>>> To me, it looks like:
>>> - the plugin-testing-harness needs to go. They should be
>>> integration tests that use a proper pom, or use pure mocks rather
>>> than the stubs that tend to just have a bunch of impossible-to-
>>> get-under-real-condition values.
>>> - John's test tools have the most complete invocation options,
>>> and tools for managing repositories that we can reuse, so I'd opt
>>> for that in that area
>>> - the verifier is well utilised, so if that is merged with the
>>> code from the verifier plugin then we can lose the invocation
>>> stuff and repository management stuff and merge it all together
>>>
>>> Can we have a wiki page with this work list? Or, can we check in
>>> the omni outliner files + an export for the non-mac users to review?
>>>
>>> [snip points I agree with]
>>>
>>>> - [+] The ITs should be in a project of their own so that we
>>>> can
>>>> reuse them across versions of Maven. We could actually
>>>> run
>>>> new versions of integration tests against old versions of
>>>> Maven. Solution: the ITs are now in a separate build
>>>> and it
>>>> is possible to run them
>>>
>>> How should this play out in plugins? I would be in favour of
>>> separate projects for these.
>>>
>>> Two things that are must haves for me:
>>> - integration tests / anything that forks a Maven instance must
>>> *not* be part of the normal test build for a plugin. They take
>>> way too long, and lead to use of maven.test.skip :(
>>> - as far as integration testing in general, I think we recommend
>>> a separate project, but enable them to be part of the same
>>> project for simplicity of development (this is more specific to
>>> things like web applications where it is probably beneficial).
>>>
>>>> - [ ] We should be able to easily integrate the IT into a
>>>> larger
>>>> run where we can use forked or embedded execution.
>>>
>>> Not sure what you mean here. What is an IT that *doesn't* use
>>> forked or embedded execution?
>>>
>>>> - [ ] automate the testing of ITs submitted by users
>>>
>>> what does this mean? I think, like a patch, a submitted IT still
>>> needs to be reviewed, and incorporated into the main test suite
>>> (with corresponding fix-for version so it only runs when it is
>>> expected to work). Otherwise, long term, we'll have lots of
>>> duplicated or poorly conceived ITs. I've seen test cases
>>> submitted that are quite useful at demonstrating something, but
>>> contain half of the user's proect which is not good for our long
>>> term scalability.
>>>
>>>> - [ ] Each IT should have its own repository if it needs
>>>> resources
>>>> from repository. We can't mess with a users repository
>>>> when
>>>> testing.
>>>> - [ ] We need to have a file system based remote repository for
>>>> testing
>>>
>>> Agreed. Isn't that what John's tool already does?
>>>
>>>> - [ ] We need to standardize on integration testing in
>>>> general. We
>>>> have people going all over the place and it's a disaster.
>>>> - [ ] We have too many IT plugins (3)
>>>> - [ ] We have too many invokers (5)
>>>> - [ ] We have too many verifiers (3)
>>>
>>> Let's specifically get these mapped out and a path forward so
>>> that everyone can push towards it, rather than relying on you to
>>> do the work.
>>>
>>>> - [ ] The ITs should run nicely from an IDE. Solution: this
>>>> does
>>>> work but requires that you run mvn clean
>>>> resources:testResources first as the IDE doesn't know
>>>> how to
>>>> set that up. Needs to be fully fixed. But it is much
>>>> nicer
>>>> running this stuff in your IDE.
>>>
>>> Agree with the point, but not sure what you are referring to
>>> about testResources - the generated projects for IDEA and I think
>>> Eclipse already do this (and obviously better integration will
>>> bring it).
>>>
>>> I think the dangerous thing is using resources for non-classpath
>>> resources. It's better for the tests to setup and use a clean
>>> project instance for an IT itself (using helper tools).
>>>
>>> [snip specific notes on old ITs that need to be updated]
>>>
>>>> - [ ] artifactIds should be aligned with directories
>>>
>>> Agreed. Also, as I did in the last test I wrote, what do we think
>>> about using purposeful names rather than numbers for integration
>>> tests?
>>>
>>> Thanks for this.
>>>
>>> Cheers,
>>> Brett
>>>
>>>
>>> --------------------------------------------------------------------
>>> -
>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Integration Testing
Posted by Brett Porter <br...@apache.org>.
On 04/01/2007, at 4:32 PM, Brett Porter wrote:
> Jason - any further thoughts on this?
Ping... No is a valid answer :)
I'd like to get your summary put somewhere individuals can pick
things off to work on - probably a jira project for shared. WDYT?
I'm overcommitted for working on things right now, but I know a
couple of people are confused about the IT testing, and we've got all
those chronically broken plugins. Volunteers?
> On 18/12/2006, at 4:35 PM, Brett Porter wrote:
>
>>
>> On 12/12/2006, at 5:08 AM, Jason van Zyl wrote:
>>
>>> So, in response to John's email: I think we need to settle on
>>> what we're going to use and stop writing new tools. We have
>>> several invokers, several verifiers, several IT plugins, and the
>>> plugin harness. It is simply out of control. We have different
>>> methods being used in different plugins, nothing is standard and
>>> it's going to kill us. The most prevalent tool, as defective as
>>> it might be is what is being used in the ITs themselves. Stephane
>>> managed to use this successfully in some of his plugins. Then
>>> after that we have an array of usages. What should happen before
>>> we start writing more stuff is to figure out something we can use
>>> now, and how to merge what we have together instead of writing
>>> more tools.
>>
>> Agreed. I think that's what John was getting at too, but by doing
>> it clean rather than rewriting something that was in use
>> somewhere. So the work left to do, in either case, is apply it
>> consistently and get rid of the stuff we aren't going to use.
>>
>> To me, it looks like:
>> - the plugin-testing-harness needs to go. They should be
>> integration tests that use a proper pom, or use pure mocks rather
>> than the stubs that tend to just have a bunch of impossible-to-get-
>> under-real-condition values.
>> - John's test tools have the most complete invocation options, and
>> tools for managing repositories that we can reuse, so I'd opt for
>> that in that area
>> - the verifier is well utilised, so if that is merged with the
>> code from the verifier plugin then we can lose the invocation
>> stuff and repository management stuff and merge it all together
>>
>> Can we have a wiki page with this work list? Or, can we check in
>> the omni outliner files + an export for the non-mac users to review?
>>
>> [snip points I agree with]
>>
>>> - [+] The ITs should be in a project of their own so that we can
>>> reuse them across versions of Maven. We could actually run
>>> new versions of integration tests against old versions of
>>> Maven. Solution: the ITs are now in a separate build
>>> and it
>>> is possible to run them
>>
>> How should this play out in plugins? I would be in favour of
>> separate projects for these.
>>
>> Two things that are must haves for me:
>> - integration tests / anything that forks a Maven instance must
>> *not* be part of the normal test build for a plugin. They take way
>> too long, and lead to use of maven.test.skip :(
>> - as far as integration testing in general, I think we recommend a
>> separate project, but enable them to be part of the same project
>> for simplicity of development (this is more specific to things
>> like web applications where it is probably beneficial).
>>
>>> - [ ] We should be able to easily integrate the IT into a larger
>>> run where we can use forked or embedded execution.
>>
>> Not sure what you mean here. What is an IT that *doesn't* use
>> forked or embedded execution?
>>
>>> - [ ] automate the testing of ITs submitted by users
>>
>> what does this mean? I think, like a patch, a submitted IT still
>> needs to be reviewed, and incorporated into the main test suite
>> (with corresponding fix-for version so it only runs when it is
>> expected to work). Otherwise, long term, we'll have lots of
>> duplicated or poorly conceived ITs. I've seen test cases submitted
>> that are quite useful at demonstrating something, but contain half
>> of the user's proect which is not good for our long term scalability.
>>
>>> - [ ] Each IT should have its own repository if it needs
>>> resources
>>> from repository. We can't mess with a users repository
>>> when
>>> testing.
>>> - [ ] We need to have a file system based remote repository for
>>> testing
>>
>> Agreed. Isn't that what John's tool already does?
>>
>>> - [ ] We need to standardize on integration testing in
>>> general. We
>>> have people going all over the place and it's a disaster.
>>> - [ ] We have too many IT plugins (3)
>>> - [ ] We have too many invokers (5)
>>> - [ ] We have too many verifiers (3)
>>
>> Let's specifically get these mapped out and a path forward so that
>> everyone can push towards it, rather than relying on you to do the
>> work.
>>
>>> - [ ] The ITs should run nicely from an IDE. Solution: this does
>>> work but requires that you run mvn clean
>>> resources:testResources first as the IDE doesn't know
>>> how to
>>> set that up. Needs to be fully fixed. But it is much nicer
>>> running this stuff in your IDE.
>>
>> Agree with the point, but not sure what you are referring to about
>> testResources - the generated projects for IDEA and I think
>> Eclipse already do this (and obviously better integration will
>> bring it).
>>
>> I think the dangerous thing is using resources for non-classpath
>> resources. It's better for the tests to setup and use a clean
>> project instance for an IT itself (using helper tools).
>>
>> [snip specific notes on old ITs that need to be updated]
>>
>>> - [ ] artifactIds should be aligned with directories
>>
>> Agreed. Also, as I did in the last test I wrote, what do we think
>> about using purposeful names rather than numbers for integration
>> tests?
>>
>> Thanks for this.
>>
>> Cheers,
>> Brett
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org