You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by John Casey <ca...@gmail.com> on 2006/08/11 00:06:54 UTC
Integration testing plugins
Hi,
I wanted to let people know about something I've been working on for the
assembly plugin. As anyone watching the commit notifications has probably
noticed, I've recently completed a large refactor of the assembly plugin,
with the aim of making it more modular and easier to unit test. While this
has made the source easier to comprehend and unit test, I've learned an
important lesson in the process: test coverage numbers don't tell the whole
story. Some of you have tried to tell me this in the past, but let's just
say there's nothing like experiencing something firsthand...
So, to make up for deficiencies in the unit tests (not checking every
permutation of nulls, for example), I wanted to put together some functional
tests which would help us set a high water mark for future releases, to
prevent regressions (or try to). The trouble has been that this really
requires running the plugin within its platform - i.e. Maven itself - and
checking the results of various builds. Further complicating things is the
fact that plugins are sometimes resolved early in the build lifecycle,
meaning the instance that actually gets tested will be something resolved
from the local repository instead of the one you just built.
This brings me to my announcement: I've written two new plugins, which are
currently in the sandbox, and which allow developers to run integration-test
builds for a plugin. The way it works is relatively simple:
1. backup the local repository plugin directory structure, and stage out the
fresh one
2. clear the resolved-plugin cache of the plugin you just built
3. run a series of Maven builds, executing pre-build and verification
scripts before/after each build to setup and verify the test, respectively
4. de-stage the plugin, meaning restore the local repository state prior to
the integration test setup
For items 1, 2, and 4, I've written the maven-plugin-test-plugin (I know,
ugly name). Item 3 is another plugin I wrote, called the
maven-invoker-plugin, which can be used to run any [set of] builds. It uses
beanshell scripts for prebuild and verification, can take CLI-injected
properties from a test.properties file in the build directory, and supports
a goals.txt file for customizing the build command used per-project.
As I mentioned, both of these plugins are in the sandbox. I've also deployed
them to the apache.snapshots repository, and you can find a sample
configuration in a profile in the maven-assembly-plugin POM.
Hopefully, we can use and evolve this pattern to start enabling
functional/integration tests for plugins which are easy to understand and
maintain.
Cheers,
John
Re: Integration testing plugins
Posted by John Casey <ca...@gmail.com>.
Yeah, I will follow up this thread with wiki documentation. I'd like to
approach it as more of a "testing maven components" with a special section
on plugins, personally...because I think a lot of the principles will apply
in both cases. WDYT?
Kenney: yeah, I actually revamped/gutted the it plugin for this one...it's
dependent on the maven-invoker API that I developed in the open for a
client. This API has a good set of tests, and is intentionally close to the
embedder API so they can eventually be merged with an optional 'fork' flag.
I wanted to use this plugin name rather than revamping the IT plugin, just
in case there were plans for other things in the existing plugin...and also
to escape the concept that this plugin is only for integration testing...
;-) I'd like to look at what features the IT plugin provides that this one
doesn't, and maybe converging them where the behavior can be generalized as
non-IT-specific.
-j
On 8/10/06, Brett Porter <br...@apache.org> wrote:
>
> cool. Can we get this written up in the wiki/docs like Jesse's stuff was?
>
> How about starting the "plugin developers centre" on the website? :)
>
> - Brett
>
> On 11/08/2006 8:06 AM, John Casey wrote:
> > Hi,
> >
> > I wanted to let people know about something I've been working on for the
> > assembly plugin. As anyone watching the commit notifications has
> probably
> > noticed, I've recently completed a large refactor of the assembly
> plugin,
> > with the aim of making it more modular and easier to unit test. While
> this
> > has made the source easier to comprehend and unit test, I've learned an
> > important lesson in the process: test coverage numbers don't tell the
> whole
> > story. Some of you have tried to tell me this in the past, but let's
> just
> > say there's nothing like experiencing something firsthand...
> >
> > So, to make up for deficiencies in the unit tests (not checking every
> > permutation of nulls, for example), I wanted to put together some
> > functional
> > tests which would help us set a high water mark for future releases, to
> > prevent regressions (or try to). The trouble has been that this really
> > requires running the plugin within its platform - i.e. Maven itself -
> and
> > checking the results of various builds. Further complicating things is
> the
> > fact that plugins are sometimes resolved early in the build lifecycle,
> > meaning the instance that actually gets tested will be something
> resolved
> > from the local repository instead of the one you just built.
> >
> > This brings me to my announcement: I've written two new plugins, which
> are
> > currently in the sandbox, and which allow developers to run
> > integration-test
> > builds for a plugin. The way it works is relatively simple:
> >
> > 1. backup the local repository plugin directory structure, and stage out
> > the
> > fresh one
> > 2. clear the resolved-plugin cache of the plugin you just built
> > 3. run a series of Maven builds, executing pre-build and verification
> > scripts before/after each build to setup and verify the test,
> respectively
> > 4. de-stage the plugin, meaning restore the local repository state prior
> to
> > the integration test setup
> >
> > For items 1, 2, and 4, I've written the maven-plugin-test-plugin (I
> know,
> > ugly name). Item 3 is another plugin I wrote, called the
> > maven-invoker-plugin, which can be used to run any [set of] builds. It
> uses
> > beanshell scripts for prebuild and verification, can take CLI-injected
> > properties from a test.properties file in the build directory, and
> supports
> > a goals.txt file for customizing the build command used per-project.
> >
> > As I mentioned, both of these plugins are in the sandbox. I've also
> > deployed
> > them to the apache.snapshots repository, and you can find a sample
> > configuration in a profile in the maven-assembly-plugin POM.
> >
> > Hopefully, we can use and evolve this pattern to start enabling
> > functional/integration tests for plugins which are easy to understand
> and
> > maintain.
> >
> > Cheers,
> >
> > John
> >
>
>
> --
> Apache Maven - http://maven.apache.org/
> Better Builds with Maven - http://library.mergere.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
Re: Integration testing plugins
Posted by Brett Porter <br...@apache.org>.
cool. Can we get this written up in the wiki/docs like Jesse's stuff was?
How about starting the "plugin developers centre" on the website? :)
- Brett
On 11/08/2006 8:06 AM, John Casey wrote:
> Hi,
>
> I wanted to let people know about something I've been working on for the
> assembly plugin. As anyone watching the commit notifications has probably
> noticed, I've recently completed a large refactor of the assembly plugin,
> with the aim of making it more modular and easier to unit test. While this
> has made the source easier to comprehend and unit test, I've learned an
> important lesson in the process: test coverage numbers don't tell the whole
> story. Some of you have tried to tell me this in the past, but let's just
> say there's nothing like experiencing something firsthand...
>
> So, to make up for deficiencies in the unit tests (not checking every
> permutation of nulls, for example), I wanted to put together some
> functional
> tests which would help us set a high water mark for future releases, to
> prevent regressions (or try to). The trouble has been that this really
> requires running the plugin within its platform - i.e. Maven itself - and
> checking the results of various builds. Further complicating things is the
> fact that plugins are sometimes resolved early in the build lifecycle,
> meaning the instance that actually gets tested will be something resolved
> from the local repository instead of the one you just built.
>
> This brings me to my announcement: I've written two new plugins, which are
> currently in the sandbox, and which allow developers to run
> integration-test
> builds for a plugin. The way it works is relatively simple:
>
> 1. backup the local repository plugin directory structure, and stage out
> the
> fresh one
> 2. clear the resolved-plugin cache of the plugin you just built
> 3. run a series of Maven builds, executing pre-build and verification
> scripts before/after each build to setup and verify the test, respectively
> 4. de-stage the plugin, meaning restore the local repository state prior to
> the integration test setup
>
> For items 1, 2, and 4, I've written the maven-plugin-test-plugin (I know,
> ugly name). Item 3 is another plugin I wrote, called the
> maven-invoker-plugin, which can be used to run any [set of] builds. It uses
> beanshell scripts for prebuild and verification, can take CLI-injected
> properties from a test.properties file in the build directory, and supports
> a goals.txt file for customizing the build command used per-project.
>
> As I mentioned, both of these plugins are in the sandbox. I've also
> deployed
> them to the apache.snapshots repository, and you can find a sample
> configuration in a profile in the maven-assembly-plugin POM.
>
> Hopefully, we can use and evolve this pattern to start enabling
> functional/integration tests for plugins which are easy to understand and
> maintain.
>
> Cheers,
>
> John
>
--
Apache Maven - http://maven.apache.org/
Better Builds with Maven - http://library.mergere.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Integration testing plugins
Posted by John Casey <ca...@gmail.com>.
Actually, pointing at a different local repo for the purposes of a build is
trivial. However, in the case of testing a plugin, you have to install the
plugin somewhere, then run a build that uses that plugin...this is not the
case with anything other than Maven components, which makes it an
exceptional use case. If Geronimo wants to fire off a build during their
integration tests, they will be able to specify the local repository
location without a problem, using either the embedder or the invoker.
Once we get nearer a release of the new embedder, I'd like to converge the
two, and hopefully get it down to a fork flag to determine which is used.
But the plugin-test stuff is different, and it's an exotic use case.
Also, if we're going to stop touching the local repository during a build,
when will we use it? If you were to use the repository-assembly stuff from
the assembly plugin to generate a one-time use repository containing all of
your dependencies, then I can imagine forking off a local repository for
integration testing. However, unless you're on a busy build server where you
could reasonably expect to encounter race conditions on artifact resolution
by way of the local repo, I would really hope that Maven would resolve to
the same set of artifacts over two successive builds like this.
I do think this would be a good option for many people when running
integration tests (for instance, on large Continuum deployments). But for
many users, this will be an unnecessary performance degradation.
-john
On 8/11/06, Jason van Zyl <ja...@maven.org> wrote:
>
>
> On 11 Aug 06, at 11:25 AM 11 Aug 06, John Casey wrote:
>
> > it doesn't backup the whole local repository, only the files that
> > will be
> > affected by a plugin install. It should be relatively stable in
> > terms of the
> > backup/restore data transfer size...varying only by the number of
> > repositories that were consulted for that artifact (because there
> > is a set
> > of metadata files for each remote repository).
> >
> > I'm not sure what you mean by encapsulating things needed by the
> > runtime.
> > What I want to avoid is introducing excessive design changes to the
> > core
> > just so we can enable plugin manipulation that wouldn't happen in a
> > normal
> > use case.
> >
>
> I think this is a normal use case. A concrete example being the
> embedder pointing at a completely separate repository structure for,
> say, Geronimo. They use a Maven repository for their own use and you
> should be able to flip a parameter and use the encapsulated
> requirements for runtime: a repository, any configuration, any
> directories which hold configuration, caches, or anything of that
> nature. I think touching the repository that is actually used in the
> day-to-day is sort of fugly. If the plugin interacts with any
> artifacts then all those are going to be affected as so if you're
> doing integration tests on the Jetty plugin then you're going to pull
> in a whole raft of other dependencies, you're sure you're picking all
> that up and restoring it fully? I think some things with artifact
> resolution are a little loosy goosy at moment and not touching the
> users repository I think would be a better option.
>
> > -john
> >
> > On 8/11/06, Jason van Zyl <ja...@maven.org> wrote:
> >>
> >>
> >> On 11 Aug 06, at 9:18 AM 11 Aug 06, John Casey wrote:
> >>
> >> > This should be solved by the stage/unstage mojos in the plugin-test
> >> > plugin.
> >> > That's its only current purpose. It will stage a new plugin
> >> > artifact into
> >> > the local repository, backing up what was there before, then
> >> clear the
> >> > pluginManager's cache of that plugin, in case it's been resolved
> >> > already.
> >> > Unstaging (de-staging?) will restore the backed up data, and clear
> >> > the cache
> >> > again.
> >> >
> >>
> >> We should move toward encapsulating anything the runtime requires so
> >> we could parameterize it on execution so that you don't have to
> >> tamper with the local repository. If someone has a massive local
> >> repository that could get pretty time consuming.
> >>
> >> > -john
> >> >
> >> > On 8/11/06, Kenney Westerhof <fo...@gods.nl> wrote:
> >> >>
> >> >>
> >> >>
> >> >> Vincent Massol wrote:
> >> >>
> >> >> > Also, Kenney do you know if the it plugin can now be used to
> >> test a
> >> >> plugin
> >> >> > that is being built for the first time? I remember that I didn't
> >> >> activate it
> >> >> > on the clover plugin because it would work only if the clover
> >> >> plugin was
> >> >> > installed in the local repository and this came after
> >> integration
> >> >> testing.
> >> >> > Thus you would always run it tests on the previous version of
> >> >> the plugin
> >> >> > (looked up from the local repo). I think I had a jira issue for
> >> >> this.
> >> >> I'm
> >> >> > offline now but I'll try to dig it up if need be.
> >> >>
> >> >> The following issue [1] should provide that: it was blocking the
> >> >> use of
> >> >> the it plugin; once fixed, it worked fine. If the local repository
> >> >> version of the plugin is used instead, then I suggest we re-
> >> open the
> >> >> issue.
> >> >>
> >> >> [1] http://jira.codehaus.org/browse/MNG-870
> >> >>
> >> >> -- Kenney
> >> >>
> >> >>
> >> ---------------------------------------------------------------------
> >> >> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >> >> For additional commands, e-mail: dev-help@maven.apache.org
> >> >>
> >> >>
> >>
> >> Jason van Zyl
> >> jason@maven.org
> >>
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >> For additional commands, e-mail: dev-help@maven.apache.org
> >>
> >>
>
> Jason van Zyl
> jason@maven.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
Re: Integration testing plugins
Posted by Jason van Zyl <ja...@maven.org>.
On 11 Aug 06, at 11:25 AM 11 Aug 06, John Casey wrote:
> it doesn't backup the whole local repository, only the files that
> will be
> affected by a plugin install. It should be relatively stable in
> terms of the
> backup/restore data transfer size...varying only by the number of
> repositories that were consulted for that artifact (because there
> is a set
> of metadata files for each remote repository).
>
> I'm not sure what you mean by encapsulating things needed by the
> runtime.
> What I want to avoid is introducing excessive design changes to the
> core
> just so we can enable plugin manipulation that wouldn't happen in a
> normal
> use case.
>
I think this is a normal use case. A concrete example being the
embedder pointing at a completely separate repository structure for,
say, Geronimo. They use a Maven repository for their own use and you
should be able to flip a parameter and use the encapsulated
requirements for runtime: a repository, any configuration, any
directories which hold configuration, caches, or anything of that
nature. I think touching the repository that is actually used in the
day-to-day is sort of fugly. If the plugin interacts with any
artifacts then all those are going to be affected as so if you're
doing integration tests on the Jetty plugin then you're going to pull
in a whole raft of other dependencies, you're sure you're picking all
that up and restoring it fully? I think some things with artifact
resolution are a little loosy goosy at moment and not touching the
users repository I think would be a better option.
> -john
>
> On 8/11/06, Jason van Zyl <ja...@maven.org> wrote:
>>
>>
>> On 11 Aug 06, at 9:18 AM 11 Aug 06, John Casey wrote:
>>
>> > This should be solved by the stage/unstage mojos in the plugin-test
>> > plugin.
>> > That's its only current purpose. It will stage a new plugin
>> > artifact into
>> > the local repository, backing up what was there before, then
>> clear the
>> > pluginManager's cache of that plugin, in case it's been resolved
>> > already.
>> > Unstaging (de-staging?) will restore the backed up data, and clear
>> > the cache
>> > again.
>> >
>>
>> We should move toward encapsulating anything the runtime requires so
>> we could parameterize it on execution so that you don't have to
>> tamper with the local repository. If someone has a massive local
>> repository that could get pretty time consuming.
>>
>> > -john
>> >
>> > On 8/11/06, Kenney Westerhof <fo...@gods.nl> wrote:
>> >>
>> >>
>> >>
>> >> Vincent Massol wrote:
>> >>
>> >> > Also, Kenney do you know if the it plugin can now be used to
>> test a
>> >> plugin
>> >> > that is being built for the first time? I remember that I didn't
>> >> activate it
>> >> > on the clover plugin because it would work only if the clover
>> >> plugin was
>> >> > installed in the local repository and this came after
>> integration
>> >> testing.
>> >> > Thus you would always run it tests on the previous version of
>> >> the plugin
>> >> > (looked up from the local repo). I think I had a jira issue for
>> >> this.
>> >> I'm
>> >> > offline now but I'll try to dig it up if need be.
>> >>
>> >> The following issue [1] should provide that: it was blocking the
>> >> use of
>> >> the it plugin; once fixed, it worked fine. If the local repository
>> >> version of the plugin is used instead, then I suggest we re-
>> open the
>> >> issue.
>> >>
>> >> [1] http://jira.codehaus.org/browse/MNG-870
>> >>
>> >> -- Kenney
>> >>
>> >>
>> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> >> For additional commands, e-mail: dev-help@maven.apache.org
>> >>
>> >>
>>
>> Jason van Zyl
>> jason@maven.org
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>>
Jason van Zyl
jason@maven.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Integration testing plugins
Posted by John Casey <ca...@gmail.com>.
it doesn't backup the whole local repository, only the files that will be
affected by a plugin install. It should be relatively stable in terms of the
backup/restore data transfer size...varying only by the number of
repositories that were consulted for that artifact (because there is a set
of metadata files for each remote repository).
I'm not sure what you mean by encapsulating things needed by the runtime.
What I want to avoid is introducing excessive design changes to the core
just so we can enable plugin manipulation that wouldn't happen in a normal
use case.
-john
On 8/11/06, Jason van Zyl <ja...@maven.org> wrote:
>
>
> On 11 Aug 06, at 9:18 AM 11 Aug 06, John Casey wrote:
>
> > This should be solved by the stage/unstage mojos in the plugin-test
> > plugin.
> > That's its only current purpose. It will stage a new plugin
> > artifact into
> > the local repository, backing up what was there before, then clear the
> > pluginManager's cache of that plugin, in case it's been resolved
> > already.
> > Unstaging (de-staging?) will restore the backed up data, and clear
> > the cache
> > again.
> >
>
> We should move toward encapsulating anything the runtime requires so
> we could parameterize it on execution so that you don't have to
> tamper with the local repository. If someone has a massive local
> repository that could get pretty time consuming.
>
> > -john
> >
> > On 8/11/06, Kenney Westerhof <fo...@gods.nl> wrote:
> >>
> >>
> >>
> >> Vincent Massol wrote:
> >>
> >> > Also, Kenney do you know if the it plugin can now be used to test a
> >> plugin
> >> > that is being built for the first time? I remember that I didn't
> >> activate it
> >> > on the clover plugin because it would work only if the clover
> >> plugin was
> >> > installed in the local repository and this came after integration
> >> testing.
> >> > Thus you would always run it tests on the previous version of
> >> the plugin
> >> > (looked up from the local repo). I think I had a jira issue for
> >> this.
> >> I'm
> >> > offline now but I'll try to dig it up if need be.
> >>
> >> The following issue [1] should provide that: it was blocking the
> >> use of
> >> the it plugin; once fixed, it worked fine. If the local repository
> >> version of the plugin is used instead, then I suggest we re-open the
> >> issue.
> >>
> >> [1] http://jira.codehaus.org/browse/MNG-870
> >>
> >> -- Kenney
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >> For additional commands, e-mail: dev-help@maven.apache.org
> >>
> >>
>
> Jason van Zyl
> jason@maven.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
Re: Integration testing plugins
Posted by Jason van Zyl <ja...@maven.org>.
On 11 Aug 06, at 9:18 AM 11 Aug 06, John Casey wrote:
> This should be solved by the stage/unstage mojos in the plugin-test
> plugin.
> That's its only current purpose. It will stage a new plugin
> artifact into
> the local repository, backing up what was there before, then clear the
> pluginManager's cache of that plugin, in case it's been resolved
> already.
> Unstaging (de-staging?) will restore the backed up data, and clear
> the cache
> again.
>
We should move toward encapsulating anything the runtime requires so
we could parameterize it on execution so that you don't have to
tamper with the local repository. If someone has a massive local
repository that could get pretty time consuming.
> -john
>
> On 8/11/06, Kenney Westerhof <fo...@gods.nl> wrote:
>>
>>
>>
>> Vincent Massol wrote:
>>
>> > Also, Kenney do you know if the it plugin can now be used to test a
>> plugin
>> > that is being built for the first time? I remember that I didn't
>> activate it
>> > on the clover plugin because it would work only if the clover
>> plugin was
>> > installed in the local repository and this came after integration
>> testing.
>> > Thus you would always run it tests on the previous version of
>> the plugin
>> > (looked up from the local repo). I think I had a jira issue for
>> this.
>> I'm
>> > offline now but I'll try to dig it up if need be.
>>
>> The following issue [1] should provide that: it was blocking the
>> use of
>> the it plugin; once fixed, it worked fine. If the local repository
>> version of the plugin is used instead, then I suggest we re-open the
>> issue.
>>
>> [1] http://jira.codehaus.org/browse/MNG-870
>>
>> -- Kenney
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>>
Jason van Zyl
jason@maven.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Integration testing plugins
Posted by John Casey <ca...@gmail.com>.
This should be solved by the stage/unstage mojos in the plugin-test plugin.
That's its only current purpose. It will stage a new plugin artifact into
the local repository, backing up what was there before, then clear the
pluginManager's cache of that plugin, in case it's been resolved already.
Unstaging (de-staging?) will restore the backed up data, and clear the cache
again.
-john
On 8/11/06, Kenney Westerhof <fo...@gods.nl> wrote:
>
>
>
> Vincent Massol wrote:
>
> > Also, Kenney do you know if the it plugin can now be used to test a
> plugin
> > that is being built for the first time? I remember that I didn't
> activate it
> > on the clover plugin because it would work only if the clover plugin was
> > installed in the local repository and this came after integration
> testing.
> > Thus you would always run it tests on the previous version of the plugin
> > (looked up from the local repo). I think I had a jira issue for this.
> I'm
> > offline now but I'll try to dig it up if need be.
>
> The following issue [1] should provide that: it was blocking the use of
> the it plugin; once fixed, it worked fine. If the local repository
> version of the plugin is used instead, then I suggest we re-open the
> issue.
>
> [1] http://jira.codehaus.org/browse/MNG-870
>
> -- Kenney
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
RE: Integration testing plugins
Posted by Vincent Massol <vi...@massol.net>.
> -----Original Message-----
> From: Kenney Westerhof [mailto:forge@gods.nl]
> Sent: vendredi 11 août 2006 13:05
> To: Maven Developers List
> Subject: Re: Integration testing plugins
>
>
>
> Vincent Massol wrote:
>
> > Also, Kenney do you know if the it plugin can now be used to test a
> plugin
> > that is being built for the first time? I remember that I didn't
> activate it
> > on the clover plugin because it would work only if the clover plugin was
> > installed in the local repository and this came after integration
> testing.
> > Thus you would always run it tests on the previous version of the plugin
> > (looked up from the local repo). I think I had a jira issue for this.
> I'm
> > offline now but I'll try to dig it up if need be.
>
> The following issue [1] should provide that: it was blocking the use of
> the it plugin; once fixed, it worked fine. If the local repository
> version of the plugin is used instead, then I suggest we re-open the
> issue.
>
> [1] http://jira.codehaus.org/browse/MNG-870
Thanks Kenney for your answer. I've tried it and I have 3 problems:
1) It seems the it plugin is not available on ibiblio and thus the build
failed when I added it to the clover plugin:
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-it-plugin</artifactId>
<configuration>
<goals>clean,install,clean,site</goals>
</configuration>
<executions>
<execution>
<phase>integration-test</phase>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
2) My directory structure is as follows in the clover plugin:
src/it/
|_ multiproject/
|_ jar1/
|_ ear1/
|_ [...]
|_ pom.xml
|_ simple/
|_ [...]
|_ pom.xml
When I ran the test goal of the it plugin it only executed the top level
multiproject/pom.xm but not the modules defined in it.
3) When I removed the clover plugin from my local repo and then ran "mvn
clean install" on the clover plugin it looked up the clover plugin from the
remote repo (ibiblio) for running the tests. This is not the right behavior
as the it plugin would need to use the clover plugin just built rather than
the latest version available in the remot repo.
Should I open JIRA issues for these?
Thanks
-Vincent
p5.vert.ukl.yahoo.com uncompressed Wed Aug 23 12:27:00 GMT 2006
___________________________________________________________________________
Découvrez un nouveau moyen de poser toutes vos questions quelque soit le sujet !
Yahoo! Questions/Réponses pour partager vos connaissances, vos opinions et vos expériences.
http://fr.answers.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Integration testing plugins
Posted by Kenney Westerhof <fo...@gods.nl>.
Vincent Massol wrote:
> Also, Kenney do you know if the it plugin can now be used to test a plugin
> that is being built for the first time? I remember that I didn't activate it
> on the clover plugin because it would work only if the clover plugin was
> installed in the local repository and this came after integration testing.
> Thus you would always run it tests on the previous version of the plugin
> (looked up from the local repo). I think I had a jira issue for this. I'm
> offline now but I'll try to dig it up if need be.
The following issue [1] should provide that: it was blocking the use of
the it plugin; once fixed, it worked fine. If the local repository
version of the plugin is used instead, then I suggest we re-open the issue.
[1] http://jira.codehaus.org/browse/MNG-870
-- Kenney
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
RE: Integration testing plugins
Posted by Vincent Massol <vi...@massol.net>.
> -----Original Message-----
> From: Kenney Westerhof [mailto:forge@gods.nl]
> Sent: vendredi 11 août 2006 00:13
> To: Maven Developers List
> Subject: Re: Integration testing plugins
>
>
> Hi John,
>
> It seems that the maven-invoker-plugin does actually the same thing
> as the maven-it-plugin (still in the sandbox), except you added some
> prebuild and verification steps.
>
> Maybe we can merge them somehow, or delete the maven-it-plugin (which I
> actually wrote in the pre-embedder era and lots of it's code was moved
> to the embedder).
>
> Do you think the maven-it-plugin is obsolete now? It runs a series of
> builds on src/it/*/pom.xml, using the embedder. The idea was to have
> unit tests in those it projects that tested plugins or other stuff (like
> deployments).
Cool stuff.
Also in the same vein, does it make sense for the new
maven-plugin-test-plugin (optionally) to use the verifier plugin to verify
build results? Or maybe that's simply transparent as the verifier plugin is
meant to be bound to the verify phase and is part of the build... The
initial reason I wrote the verifier plugin was for plugin it testing in
order to assert results from the build.
I guess it would be nice to have a page somewhere on plugin testing
explaining the different options: it plugin, plugin-test plugin, verifier
plugin, test harness, any more?
Also, Kenney do you know if the it plugin can now be used to test a plugin
that is being built for the first time? I remember that I didn't activate it
on the clover plugin because it would work only if the clover plugin was
installed in the local repository and this came after integration testing.
Thus you would always run it tests on the previous version of the plugin
(looked up from the local repo). I think I had a jira issue for this. I'm
offline now but I'll try to dig it up if need be.
Thanks
-Vincent
> John Casey wrote:
> > Hi,
> >
> > I wanted to let people know about something I've been working on for the
> > assembly plugin. As anyone watching the commit notifications has
> probably
> > noticed, I've recently completed a large refactor of the assembly
> plugin,
> > with the aim of making it more modular and easier to unit test. While
> this
> > has made the source easier to comprehend and unit test, I've learned an
> > important lesson in the process: test coverage numbers don't tell the
> whole
> > story. Some of you have tried to tell me this in the past, but let's
> just
> > say there's nothing like experiencing something firsthand...
> >
> > So, to make up for deficiencies in the unit tests (not checking every
> > permutation of nulls, for example), I wanted to put together some
> > functional
> > tests which would help us set a high water mark for future releases, to
> > prevent regressions (or try to). The trouble has been that this really
> > requires running the plugin within its platform - i.e. Maven itself -
> and
> > checking the results of various builds. Further complicating things is
> the
> > fact that plugins are sometimes resolved early in the build lifecycle,
> > meaning the instance that actually gets tested will be something
> resolved
> > from the local repository instead of the one you just built.
> >
> > This brings me to my announcement: I've written two new plugins, which
> are
> > currently in the sandbox, and which allow developers to run
> > integration-test
> > builds for a plugin. The way it works is relatively simple:
> >
> > 1. backup the local repository plugin directory structure, and stage out
> > the
> > fresh one
> > 2. clear the resolved-plugin cache of the plugin you just built
> > 3. run a series of Maven builds, executing pre-build and verification
> > scripts before/after each build to setup and verify the test,
> respectively
> > 4. de-stage the plugin, meaning restore the local repository state prior
> to
> > the integration test setup
> >
> > For items 1, 2, and 4, I've written the maven-plugin-test-plugin (I
> know,
> > ugly name). Item 3 is another plugin I wrote, called the
> > maven-invoker-plugin, which can be used to run any [set of] builds. It
> uses
> > beanshell scripts for prebuild and verification, can take CLI-injected
> > properties from a test.properties file in the build directory, and
> supports
> > a goals.txt file for customizing the build command used per-project.
> >
> > As I mentioned, both of these plugins are in the sandbox. I've also
> > deployed
> > them to the apache.snapshots repository, and you can find a sample
> > configuration in a profile in the maven-assembly-plugin POM.
> >
> > Hopefully, we can use and evolve this pattern to start enabling
> > functional/integration tests for plugins which are easy to understand
> and
> > maintain.
> >
> > Cheers,
> >
> > John
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
p4.vert.ukl.yahoo.com uncompressed Fri Aug 11 09:27:01 GMT 2006
___________________________________________________________________________
Découvrez un nouveau moyen de poser toutes vos questions quelque soit le sujet !
Yahoo! Questions/Réponses pour partager vos connaissances, vos opinions et vos expériences.
http://fr.answers.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Integration testing plugins
Posted by Kenney Westerhof <fo...@gods.nl>.
Hi John,
It seems that the maven-invoker-plugin does actually the same thing
as the maven-it-plugin (still in the sandbox), except you added some
prebuild and verification steps.
Maybe we can merge them somehow, or delete the maven-it-plugin (which I
actually wrote in the pre-embedder era and lots of it's code was moved
to the embedder).
Do you think the maven-it-plugin is obsolete now? It runs a series of
builds on src/it/*/pom.xml, using the embedder. The idea was to have
unit tests in those it projects that tested plugins or other stuff (like
deployments).
-- Kenney
John Casey wrote:
> Hi,
>
> I wanted to let people know about something I've been working on for the
> assembly plugin. As anyone watching the commit notifications has probably
> noticed, I've recently completed a large refactor of the assembly plugin,
> with the aim of making it more modular and easier to unit test. While this
> has made the source easier to comprehend and unit test, I've learned an
> important lesson in the process: test coverage numbers don't tell the whole
> story. Some of you have tried to tell me this in the past, but let's just
> say there's nothing like experiencing something firsthand...
>
> So, to make up for deficiencies in the unit tests (not checking every
> permutation of nulls, for example), I wanted to put together some
> functional
> tests which would help us set a high water mark for future releases, to
> prevent regressions (or try to). The trouble has been that this really
> requires running the plugin within its platform - i.e. Maven itself - and
> checking the results of various builds. Further complicating things is the
> fact that plugins are sometimes resolved early in the build lifecycle,
> meaning the instance that actually gets tested will be something resolved
> from the local repository instead of the one you just built.
>
> This brings me to my announcement: I've written two new plugins, which are
> currently in the sandbox, and which allow developers to run
> integration-test
> builds for a plugin. The way it works is relatively simple:
>
> 1. backup the local repository plugin directory structure, and stage out
> the
> fresh one
> 2. clear the resolved-plugin cache of the plugin you just built
> 3. run a series of Maven builds, executing pre-build and verification
> scripts before/after each build to setup and verify the test, respectively
> 4. de-stage the plugin, meaning restore the local repository state prior to
> the integration test setup
>
> For items 1, 2, and 4, I've written the maven-plugin-test-plugin (I know,
> ugly name). Item 3 is another plugin I wrote, called the
> maven-invoker-plugin, which can be used to run any [set of] builds. It uses
> beanshell scripts for prebuild and verification, can take CLI-injected
> properties from a test.properties file in the build directory, and supports
> a goals.txt file for customizing the build command used per-project.
>
> As I mentioned, both of these plugins are in the sandbox. I've also
> deployed
> them to the apache.snapshots repository, and you can find a sample
> configuration in a profile in the maven-assembly-plugin POM.
>
> Hopefully, we can use and evolve this pattern to start enabling
> functional/integration tests for plugins which are easy to understand and
> maintain.
>
> Cheers,
>
> John
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org