You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by Brett Porter <br...@apache.org> on 2007/12/26 16:49:47 UTC
More on plugin integration testing
Hi,
I wanted to write some plugin integration tests today and bumped into
the same problems that others have been articulating - it is currently
unclear what direction a new plugin developer should take. The doc on
the web site is not really complete. Though John had started to set
out a path with the assembly plugin, but it has some limitations that
have led to other efforts. Maven's IT structure (also used for
Surefire) is pretty solid but still has some gaps and isn't suitable
in all scenarios.
So instead of beating the same path, I wanted to capture everything
that's been/being done in one place: http://docs.codehaus.org/display/MAVENUSER/Review+of+Plugin+Testing+Strategies
. I trawled through all the current sites and code to pick out the
features and draw them into a matrix. Links are at the bottom for the
things I considered relevant.
Firstly - does that feature matrix look accurate to everyone involved
in this?
I wanted to use this doc as a baseline for any further discussion and
as a place to capture any output so that we have one path forward -
much like the effort made with the Maven ITs in the past.
It's my expectation that the outcomes would be:
1) a set of test projects that exercise different scenarios using the
plugins available today
2) document how to do so
3) identify areas of duplication, and of potential convergence
4) come up with a list of small, doable tasks that will bring this
about, updating docs as it goes
I'm not aiming for a grand unified theory of testing - there will
always be needs for different types of plugins. However, separated
responsibilities (setup/verification vs execution, for example) will
help, esp. when mapped in through the lifecycle (though with the
caveat that configuring more plugins makes testing less approachable).
But what I do need is to beat a path forward, and the Maven ITs were
not suitable for me - so I wanted to capture the output of my efforts
in some way.
WDYT?
Cheers,
Brett
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: More on plugin integration testing
Posted by Brett Porter <br...@apache.org>.
On 27/12/2007, at 4:15 PM, Dan Fabulich wrote:
> 1) This document draws a distinction between "integration testing"
> and "functional testing." The two terms are defined, but I don't
> think I quite understand the distinction in practice.
>
> In particular, this doc states that only two of the available
> solutions "can be used for integration testing:" the maven-it-plugin
> "ITP" and the maven-verifier-plugin "VRP."
>
> I find that puzzling, because VRP does VERY little: it just verifies
> that a file exists and (optionally) that its content matches a
> regex. It's hard to see how it could be used in anything but a
> functional test, in which you'd run a full Maven build and then
> assert on the output...?
You're right - typo. (I did make quite a few :)
>
>
> As for maven-it-plugin, I didn't think maven-it-plugin could do a
> different kind of testing from what maven-invoker-plugin does...
> There's even a remark to that effect in the Notes section of your
> wiki article: "IT plugin is now duplicated with invoker."
>
> What does it mean when this doc says that maven-it-plugin can do
> integration testing but maven-invoker-plugin can't?
You're right on this too - I wasn't really thinking straight about it.
Let me elaborate on what I was getting at.
Integration testing can legitimately take a small subset of modules to
integrate or all of them so it has a pretty broad definition. I've
generally thought of the Maven ITs to be functional/system tests
rather than integration tests. What I was looking for is something
integrated, but only used in the context of the thing being tested
(the plugin). So you'd use the embedder to construct the system as
usual, but then pre-set the state and only execute the portion of the
lifecycle that runs the mojo. The example I was thinking of was
testing source generation - there's really no need to compile the
sources later on to verify that source generation worked as you
expected, and it may be a time consuming thing better reserved for a
system test.
Anyway, I've updated the table and re-labeled "contextual integration
testing" (if that's even the right terminology) and indicated there is
no current solution for it, and made the integration testing results
more accurate.
>
>
> 2) Three of the solutions have the feature "Runs with Surefire:"
> maven-verifier (VER), maven-plugin-testing-harness (PTH), and "maven-
> component-it-plugin with Java wrappers generated" (CIJ). In those
> cases where they run under Surefire, I would think that a number of
> the categories listed would be N/A.
>
> For example, "able to run/timeout tests in parallel", "can ignore
> failures", and "can skip execution" are pretty clearly Surefire
> features. Weirdly, CIJ marks this feature as N/A, PTH gets a (/) and
> VER gets a (x). But surely all three of these should have the same
> value, right?
>
> For those, I think it's pretty clear that all three of them should
> get a (/). But consider ANSI color support. Surefire doesn't
> support that right now (I just filed it as SUREFIRE-420), so I guess
> those tools that use Surefire would get a (x). But if we do wind up
> supporting ANSI color some day, then they all get a (/), without
> ever touching the old plugins.
>
> As for Groovy support, you can write Groovy tests and run them under
> Surefire right now (I just tried it); Surefire just runs classes in
> test-classes, ignorant of how they got there. Any language that
> compiles to .class and extends TestCase should work, including
> Jython and Rhino-compiled JavaScript. More generally, anything that
> runs in JUnit will run under Surefire.
>
> I think that means that CIJ, VER and PTH all support writing tests
> in Groovy, right?
Yeah, the medium for capturing the information wasn't particularly
helpful here, but some consistency would have helped :)
For Surefire - I've added a flag for features that they derive from
Surefire. For Groovy - I've clarified that this was WRT to just the
setup/verify scripts.
I've also made some updates to the structure of the table that should
help better clarify and tried to make it more consistent - does it help?
>
>
> 3) What is "able to run a regex set of tests?" Is that just running
> a specific subset of tests defined by a regex? e.g. if you had
> tests FooBar, FooBaz, and BlahBar, you could run just "Foo.*" or
> just ".*Bar"?
>
> Do Surefire's wildcard-based includes count? (You know you can even
> run them from the command line, with -Dtest=Foo* ...)
>
> If not, should we introduce a new feature called "able to run a
> wildcard set of tests"?
I didn't know Surefire supported that :) Have renamed regex to
wildcard/regex and update.
>
>
> 4) What is "interpolate goals/properties to run Maven with?"
the goals file can contain expressions (maybe John can explain why
it's useful, I was just going through the feature list of the invoker
plugin).
>
>
> 5) What is "Provides alternate remote repository handling?"
> Apparently nothing has it. What would it do if someone did support
> it?
Something Brian wanted in one of the mail threads. I think what he
wanted was to be able to have Maven behave like it would talking to a
remote repo, without having to talk to a remote repo, and without
having to do a lot of legwork to set up a file based one to test with?
I've moved it to the unit testing specific section.
> Finally, I've got some quibbling. By my count, VER gets 39 (x)s.
> Of those, I asked questions about 5 of them ("integration" testing,
> regexes, interpolation of goals/properties, alternate remote repo
> handling), 16 I'd let stand, and I'd like to quibble about the other
> 19. :-)
>
> 6) These three I think are just mistakes... they clearly work today.
> * Suite teardown steps (public void tearDown)
fixed
>
> * Cleans up logs, targets (verifier does this...?)
it doesn't remove things from /tmp I believe - I clarified the text to
include that
>
> * Configurable working directory (that's the only argument to
> Verifier's constructor!)
fixed
>
>
> 7) These five should all be (/), because Surefire supports them:
> * Able to run tests in parallel
> * Able to timeout tests run in parallel
> * Can ignore failures
> * Can skip execution
> * Groovy support
fixed. I guess the groovy one could be either way - verifier doesn't
support using it for it's setup/teardown (it has it's own format), but
you could write them into the unit test itself...
> 8) VER got six (x)s that all amount to roughly the same thing:
> controlling the flags that get passed to the invoked Maven, whether
> JVM flags or command line arguments.
> * Configurable individual Maven settings
> * Pass profiles to run Maven with
> * Pass profiles to run Maven with in a file
> * Pass flags to run Maven with
> * Pass flags to run Maven with in a file
> * Able to run Maven in debug mode
True, updated.
>
> 9) Can be used for report plugin testing: I test the surefire-report
> plugin using VER; it verifies that the report ran successfully
> without failing the Maven build. You can't verify a report
> completely without also adding HtmlUnit (which I haven't bothered to
> do), but HtmlUnit is definitely compatible with Verifier. That's at
> least as good as maven-invoker-plugin, which got a (/).
Updated.
>
>
> 10) "Isolated from user's environment/settings" and "Able to inherit
> shell's environment:" In fact, Verifier inherits the shell
> environment by default, which is what makes it less likely to be
> isolated from the user's environment. Still, you CAN deliberately
> isolate yourself further, if you wish, by specifying your
> environment variables when you call executeGoal, and by passing in
> an explicit Maven settings file.
>
> We should split this into "Isolated from user's environment/settings
> by default" and "Can isolate from user's environment/settings." And
> "Able to inherit shell environment" should be (/).
Done.
>
>
> 11) Can select which projects to run (include/exclude). Projects
> are just resources. Tests are what you run. This should be N/A or
> (/); you can certainly select which tests to run (include/exclude)
> using Surefire.
>
Done.
> 12) Able to drop-in an IT easily from a user submission. I don't
> think this is specific enough. I think VER is pretty easy; other
> people apparently disagree. This should be phrased as "able to drop
> in an IT without _______" where "_______" is something hard.
>
> That, of course, requires us to agree on what "______" should be and
> whether it's hard. As I've argued in the past, some people have
> argued that "_____" is "writing a JUnit test," which I insist is
> very easy.
>
Took it out... I agree we could add some more specific stuff.
Cheers,
Brett
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: More on plugin integration testing
Posted by Dan Fabulich <da...@fabulich.com>.
Thanks very much for working on this, Brett! This is a great document.
Brett Porter wrote:
> Firstly - does that feature matrix look accurate to everyone involved in
> this?
I do have some remarks, most of which are aimed towards improving
maven-verifier's grade.
1) This document draws a distinction between "integration testing" and
"functional testing." The two terms are defined, but I don't think I
quite understand the distinction in practice.
In particular, this doc states that only two of the available solutions
"can be used for integration testing:" the maven-it-plugin "ITP" and the
maven-verifier-plugin "VRP."
I find that puzzling, because VRP does VERY little: it just verifies that
a file exists and (optionally) that its content matches a regex. It's
hard to see how it could be used in anything but a functional test, in
which you'd run a full Maven build and then assert on the output...?
As for maven-it-plugin, I didn't think maven-it-plugin could do a
different kind of testing from what maven-invoker-plugin does... There's
even a remark to that effect in the Notes section of your wiki article:
"IT plugin is now duplicated with invoker."
What does it mean when this doc says that maven-it-plugin can do
integration testing but maven-invoker-plugin can't?
2) Three of the solutions have the feature "Runs with Surefire:"
maven-verifier (VER), maven-plugin-testing-harness (PTH), and
"maven-component-it-plugin with Java wrappers generated" (CIJ). In those
cases where they run under Surefire, I would think that a number of the
categories listed would be N/A.
For example, "able to run/timeout tests in parallel", "can ignore
failures", and "can skip execution" are pretty clearly Surefire features.
Weirdly, CIJ marks this feature as N/A, PTH gets a (/) and VER gets a (x).
But surely all three of these should have the same value, right?
For those, I think it's pretty clear that all three of them should get a
(/). But consider ANSI color support. Surefire doesn't support that
right now (I just filed it as SUREFIRE-420), so I guess those tools that
use Surefire would get a (x). But if we do wind up supporting ANSI color
some day, then they all get a (/), without ever touching the old plugins.
As for Groovy support, you can write Groovy tests and run them under
Surefire right now (I just tried it); Surefire just runs classes in
test-classes, ignorant of how they got there. Any language that compiles
to .class and extends TestCase should work, including Jython and
Rhino-compiled JavaScript. More generally, anything that runs in JUnit
will run under Surefire.
I think that means that CIJ, VER and PTH all support writing tests in
Groovy, right?
3) What is "able to run a regex set of tests?" Is that just running a
specific subset of tests defined by a regex? e.g. if you had tests
FooBar, FooBaz, and BlahBar, you could run just "Foo.*" or just ".*Bar"?
Do Surefire's wildcard-based includes count? (You know you can even run
them from the command line, with -Dtest=Foo* ...)
If not, should we introduce a new feature called "able to run a wildcard
set of tests"?
4) What is "interpolate goals/properties to run Maven with?"
5) What is "Provides alternate remote repository handling?" Apparently
nothing has it. What would it do if someone did support it?
--------------
Finally, I've got some quibbling. By my count, VER gets 39 (x)s. Of
those, I asked questions about 5 of them ("integration" testing, regexes,
interpolation of goals/properties, alternate remote repo handling), 16 I'd
let stand, and I'd like to quibble about the other 19. :-)
6) These three I think are just mistakes... they clearly work today.
* Suite teardown steps (public void tearDown)
* Cleans up logs, targets (verifier does this...?)
* Configurable working directory (that's the only argument to Verifier's
constructor!)
7) These five should all be (/), because Surefire supports them:
* Able to run tests in parallel
* Able to timeout tests run in parallel
* Can ignore failures
* Can skip execution
* Groovy support
8) VER got six (x)s that all amount to roughly the same thing: controlling
the flags that get passed to the invoked Maven, whether JVM flags or
command line arguments.
* Configurable individual Maven settings
* Pass profiles to run Maven with
* Pass profiles to run Maven with in a file
* Pass flags to run Maven with
* Pass flags to run Maven with in a file
* Able to run Maven in debug mode
But you can absolutely pass arbitrary arguments using VER, passing any
arbitrary JVM arg or command line argument. (You pass JVM args using the
MAVEN_OPTS environment variable.) Since you can pass arbitrary command
line arguments, you pass arbitrary profiles in, and can also specify
arbitrary settings.xml files with those profiles defined. And of course
you can also do it "in a file." ;-)
9) Can be used for report plugin testing: I test the surefire-report
plugin using VER; it verifies that the report ran successfully without
failing the Maven build. You can't verify a report completely without
also adding HtmlUnit (which I haven't bothered to do), but HtmlUnit is
definitely compatible with Verifier. That's at least as good as
maven-invoker-plugin, which got a (/).
10) "Isolated from user's environment/settings" and "Able to inherit
shell's environment:" In fact, Verifier inherits the shell environment by
default, which is what makes it less likely to be isolated from the user's
environment. Still, you CAN deliberately isolate yourself further, if you
wish, by specifying your environment variables when you call executeGoal,
and by passing in an explicit Maven settings file.
We should split this into "Isolated from user's environment/settings by
default" and "Can isolate from user's environment/settings." And "Able to
inherit shell environment" should be (/).
11) Can select which projects to run (include/exclude). Projects are just
resources. Tests are what you run. This should be N/A or (/); you can
certainly select which tests to run (include/exclude) using Surefire.
12) Able to drop-in an IT easily from a user submission. I don't think
this is specific enough. I think VER is pretty easy; other people
apparently disagree. This should be phrased as "able to drop in an IT
without _______" where "_______" is something hard.
That, of course, requires us to agree on what "______" should be and
whether it's hard. As I've argued in the past, some people have argued
that "_____" is "writing a JUnit test," which I insist is very easy.
> It's my expectation that the outcomes would be:
> 1) a set of test projects that exercise different scenarios using the plugins
> available today
> 2) document how to do so
> 3) identify areas of duplication, and of potential convergence
> 4) come up with a list of small, doable tasks that will bring this about,
> updating docs as it goes
I've added one proposal to the Wiki which I don't think anybody had
objected to: make maven-verifier and maven-invoker share code.
I think if we did that, as well as improving local repository isolation in
maven-invoker (and thereby maven-verifier) would knock out most of the
remaining (x)s in maven-verifier and maven-invoker.
I'd like to ultimately tell testers: "Use PTH for unit testing, use
maven-verifier and/or maven-invoker for functional testing, and that's
it."
-Dan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org