You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by Dan Fabulich <da...@fabulich.com> on 2007/12/12 00:28:43 UTC
Invoker vs. Verifier?
Forgive me if I'm picking at a sore spot, but can someone help me
understand the difference/overlap between maven-invoker and
maven-verifier?
As I understand it, they both do roughly the same thing, except one of
them is a Maven plugin where you write your test in a goals.txt file +
beanshell, and the other is meant to be run programmatically from a JUnit
test. Is that essentially correct?
(Whereas maven-verifier-plugin is another beast entirely, simply checking
for the presence/absence of certian files.)
Am I right in thinking that, despite their substantial overlap,
maven-verifier and maven-invoker share no code?
-Dan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Invoker vs. Verifier?
Posted by Dan Fabulich <da...@fabulich.com>.
John Casey wrote:
>
> On Dec 12, 2007, at 9:47 PM, Dan Fabulich wrote:
>
>> I tweak the test to add a MAVEN_OPTS environment variable, including the
>> -Xrunjdwp string. (It would be easy to add some sugar to Verifier and/or
>> Invoker to make this easier; I didn't want to go fooling around with the
>> Verifier, so I just copied and pasted out of my notes when I needed this.)
>
> Do you have an actual example of this that I can look at?
In Eclipse/IDEA you can launch the test and modify the test configuration
to include different environment variables; it's all GUI and hard to
provide an "example."
However, you can also do it by modifying the code of the test like this:
Verifier verifier = new Verifier( testDir.getAbsolutePath() );
HashMap envVars = new HashMap();
envVars.put( "MAVEN_OPTS", "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8001 -Xnoagent -Djava.compiler=NONE" );
verifier.executeGoal( "test" , envVars );
verifier.verifyErrorFreeLog();
verifier.resetStreams();
-Dan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Invoker vs. Verifier?
Posted by John Casey <jd...@commonjava.org>.
On Dec 12, 2007, at 9:47 PM, Dan Fabulich wrote:
> I tweak the test to add a MAVEN_OPTS environment variable,
> including the -Xrunjdwp string. (It would be easy to add some
> sugar to Verifier and/or Invoker to make this easier; I didn't want
> to go fooling around with the Verifier, so I just copied and pasted
> out of my notes when I needed this.)
Do you have an actual example of this that I can look at?
As for the rest, I'll be interested to see what you come up with. I
definitely agree that this Tower of Babel approach to things that we
have in place now isn't helpful. I'm still not completely convinced
that it's effective to do all of this through the IDE (yet), but I'm
definitely willing to see an example of how it can be improved, as
long as this doesn't mean compromising readability or debug-ability
of headless build logs, as in the case of a test failing on the CI
server, but not on my localhost.
---
John Casey
Committer and PMC Member, Apache Maven
mail: jdcasey at commonjava dot org
blog: http://www.ejlife.net/blogs/john
rss: http://feeds.feedburner.com/ejlife/john
Re: Invoker vs. Verifier?
Posted by Dan Fabulich <da...@fabulich.com>.
John Casey wrote:
> Not at all; I mean running the test. In order to run one of these tests
> (which are orchestrated by something akin to the maven-verifier from a JUnit
> or other java-driven test case), you must run JUnit or whatever, so you can
> be sure you have the same options enabled, environment present, and
> assertions tested as take place in the test-case code.
I stand by my claim that running JUnit is very easy.
> For instance, simply cd into src/test/resources/it0105 in
> core-integration-tests, and see if you can figure out what it tests, and
> how. You can't, not without looking at the JUnit code that drives it, to
> determine what the criteria for success is, and which flags and
> goals/lifecycle phases to invoke.
I have two remarks about this. 1) Don't do it that way, and 2) We can
compromise on the directory structure.
1) Don't do it that way: I think this is a very weird way to approach the
tests that only seems natural because people are used to doing it that
way. I'll try to explain what I mean with a hypothetical story.
Suppose you were testing some XML serializer/deserializer, and you had a
bunch of JUnit tests checked into src/test/java and a bunch of XML files
checked into src/test/resources.
If you're going to go browsing around to see what kinds of tests are
available, where are you going to go first? I'd argue that the most
natural thing to do is to go look at the tests themselves in
src/test/java. The XML files aren't tests... they're just test RESOURCES.
Looking at the tests is a good idea for another reason also: the tests can
(must) have some reference in their code to the resources they're using,
telling you (and the JVM!) where to find the resources. The resources, on
the other hand, don't have to (and typically won't) have a reference to
the tests that will run them.
Similarly, Maven projects aren't tests. They may be test resources.
It's weird to go browsing around the test resources trying to figure out
what tests might run them. Instead, you should be looking at the tests
themselves. Indeed, looking at the tests it's very easy to see what
resources they're using.
Even in the maven-invoker-plugin case, you shouldn't go digging around in
the project-under-test *first*. First you should look at the
commands/assertions in the POM; see what it's doing. THEN go look at the
project-under-test. It's the same thing: look at the test first, not the
resource.
2) We can compromise the directory structure: Despite those remarks, sure,
it would be nicer if you could have your JUnit/TestNG integration test in
the same directory as its own test resources.
That's true of all JUnit/TestNG tests by the way, not just Maven
integration tests. The Maven standard is to keep test resources in
src/test/resources, separate from the tests in src/test/java. But it
doesn't have to be that way... lots of people put their resources in the
same directory as the classes that use them.
You could, if you wished, just put them all together into one directory,
claiming that this directory is both the testSourceDirectory AND the
testResource directory. (I think you'd want some exclusion/inclusion rules
to keep everything straight, but that's not so bad.)
With that said, I think that one should resist the temptation to do this
in all but the simplest cases. What about integration tests that test
multiple projects? You can't put the same test in multiple directories.
Or what about projects that are used by multiple integration tests? The
relationship of tests to resources can be many-to-many.
I claim that the clearest thing is to follow the Maven standard, even if
it's sub-optimal in this case: keep your resources in one directory and
have your tests refer to them.
Ultimately, this part of the argument is about which directory structure
is clearest. As we know, arguments about clarity of directory structure
can be highly controversial, but are never conclusive and don't amount to
very much. Nobody "wins" arguments about directory structure; everyone's
a loser.
If you buy my argument that it's good to write tests in a real test
framework in a normal language, that writing tests in a POM is not ideal,
then I'm pretty sure we can deal with any remaining concerns about
directory structure. I'm easy! :-)
> Again, it's not just about running the tests, but being able to actually
> debug a failing test effectively. Tests work best when they're easy to
> understand and work with, and when a maven core-integration-test fails,
> you can definitely see how this setup falls down. Running and re-running
> the same test without change from the IDE isn't useful for debugging,
> and running the invoker from this kind of code with the remote debugging
> enabled is virtually impossible...incidentally, if you've figured out
> how to do it, I'd be interested in learning.
I wrote a lot of new integration tests in the maven-verifier style for
Surefire, and I think debugging it doesn't suck, and I'll happily share my
"secret." Here's my work cycle:
0) I do some work in the IDE. When I feel good about what I've done, I
mvn install to get the plugin into my local repository. (My IDE has a
button to run mvn.) Now we're ready for integration testing.
1) Run the tests from the IDE; tests fail. I read the log file in the
stacktrace.
Suppose I don't understand why the test is failing yet.
I've got two options now: I can run the test under a debugger purely in
the IDE, or I can attempt to reproduce the test failure manually.
2a) Run the test in a debugger purely in the IDE
I tweak the test to add a MAVEN_OPTS environment variable, including the
-Xrunjdwp string. (It would be easy to add some sugar to Verifier and/or
Invoker to make this easier; I didn't want to go fooling around with the
Verifier, so I just copied and pasted out of my notes when I needed this.)
I re-run the test in Run mode (not Debug mode); eventually the forked
process suspends for me to attach a debugger. I attach and step through
both processes and analyze the problem. [I could have run the test in
Debug mode, and then had two debuggers: one for the test and its
assertions, and another for the forked Maven. But that's rarely useful.]
(You can also use -Dmaven.surefire.debug=true if you need to debug
Surefire's forked JUnit processes. [In 2.4, you can even configure the
port!])
2b) Manual repro
I double-click on the test failure in my IDE. The test states clearly
which project it's using, and what it's doing to the resource. I open a
command line window to that directory and manually reproduce the test by
calling mvn with appropriate environment variables, system properties,
etc.
[In particular, I DON'T "bounce back and forth" between the resources
directory and the test. The test is open in a window in my IDE; my
command window is only open to the resources subdirectory.]
If I can't manually repro the failure, I examine the test more closely to
see if there's a bug in the test or if I made a mistake. If I can repro
it, I might launch the test with mvnDebug, allowing me to debug Surefire's
forked JUnit tests remotely.
> In my opinion, there should certainly be hooks available to generate a
> JUnit wrapper around an integration test, but that wrapper should not
> carry information only exists outside the test project directory. I'd
> favor something more like having an orchestrator POM that called
> something like the invoker to run the real build using the real POM,
> then verifies the results of that build.
-0 on putting the test in the same directory as the project-under-test, -1
on putting the assertions in a POM.
Verifications can get pretty interesting. For example, for Surefire, I
verify that it's working correctly by instantiating the
SurefireReportParser and passing it the XML files in
target/surefire-reports. I even made a little reusable helper function
called assertTestSuiteResults that lets me cleanly assert in one line how
many passes, failures, and skipped tests there should be in a given
Surefire run.
Writing snippets of reusable test code is trivial when you're writing your
tests in Java with JUnit/TestNG, but a pain in the butt when you need to
state your assertions in a POM file.
And what if you want to write data driven tests? Re-run failures only?
Mark tests as temporarily ignored (showing up as "yellow" on the report)?
Make one test depend on the results of another, auto-skipping if earlier
tests fail? These features are nice when you're running unit tests but
they are lifesavers when you're running slow integration tests.
My real point is that by trying to write another test runner plugin, you
have to write another test framework; we should not do this, because the
existing test frameworks are great. We DEFINITELY shouldn't do this
simply for directory structure reasons... we can tweak those if we want.
> Best of all, it could be written as a very simple archetype that
> generates a portable test case which can live in almost any directory
> structure within the integration-test aggregator build to make it easier
> to organize the test cases according to functionality.
I believe someone (Brian?) has already done this for tests written in the
maven-verifier style. I approve.
-Dan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Invoker vs. Verifier?
Posted by John Casey <jd...@commonjava.org>.
On Dec 12, 2007, at 4:18 PM, Dan Fabulich wrote:
> John Casey wrote:
>
>> First things first. maven-invoker and maven-invoker-plugin are not
>> separate things. The maven-invoker-plugin uses maven-invoker, but
>> maven-invoker is meant to be a reusable library, not just a plugin.
>
> I find this remark quite confusing... if one is a library, and one
> is a plugin, and they have two separate POMs, one for the library
> JAR ("maven-invoker") and one for the plugin ("maven-invoker-
> plugin") doesn't that make them two things that share code?
>
> Actually, I think our "disagreement" is mostly a misunderstanding,
> and I think it mostly turns on this point of whether maven-invoker
> and maven-invoker plugin are "two things."
>
> In particular, I said that we should not use maven-invoker-plugin
> for integration testing. But I didn't mean that we shouldn't use
> maven-invoker!
To me, shared code implies some sort of mutual dependency on
something, whether that's cyclical or dependent on the same third
library. Sorry for the confusion. Having said that, I definitely
disagree with the approach you outline below. See my comments inline.
>
>> The maven-verifier (not the verifier plugin, I know it's
>> confusing) gets away from this, in that it requires a project
>> directory in src/test/resources AND a JUnit test case to
>> orchestrate the test. IMO, this makes it extremely difficult to
>> run a single core integration test from the command line, so you
>> can capture logs for offline examination, for instance.
>
> With that, I definitely disagree. Can you say more about what's
> difficult about running a single test?
>
> Certainly it's trivial to run a single JUnit test from the command
> line using Surefire: "mvn test -Dtest=FooTest" does the job nicely;
> I use that all the time. And it's REALLY easy to run a single test
> from Eclipse/IDEA.
>
> Maybe you meant that you think maven-verifier tests are harder to
> WRITE than writing goals.txt + beanshell tests? (I disagree with
> that, too, but it's worth clarifying what we're talking about.)
Not at all; I mean running the test. In order to run one of these
tests (which are orchestrated by something akin to the maven-verifier
from a JUnit or other java-driven test case), you must run JUnit or
whatever, so you can be sure you have the same options enabled,
environment present, and assertions tested as take place in the test-
case code. For instance, simply cd into src/test/resources/it0105 in
core-integration-tests, and see if you can figure out what it tests,
and how. You can't, not without looking at the JUnit code that drives
it, to determine what the criteria for success is, and which flags
and goals/lifecycle phases to invoke. If I needed to re-run this test
to actually diagnose a failure (which is the whole point here), I
have to dig around in source code that's completely outside the
it0105 directory, then come back and replicate that maven command,
with the addition of the -X flag and a pipe to the tee command so I
can analyze the output outside of the build.
This is much harder than it needs to be, and the same is true for
plugin integration tests.
>
>> Again, the idea behind test builds driven by the invoker is to
>> provide a test suite with a low barrier to entry, and with all the
>> potential for configuration present in the main user's experience
>> with Maven. It's not a panacea, but neither is writing JUnit tests
>> that orchestrate and verify test-build resources in a separate
>> directory structure, which can really only be run properly from
>> the JUnit api.
>
> You call it the "JUnit API" like it's this big hassle to run JUnit
> tests... but it's really easy to run JUnit tests, both from the
> command line and from an IDE; certainly 99% of our users know how
> to do it, and I hope most of them do it frequently! :-)
>
> Furthermore, you seem to imply here that you can't just cd into src/
> test/resources and run the projects there by hand directly... but
> of course you can do that when you want/need to. I often do that
> when I'm first setting up an integration test, before I've written
> the JUnit assertions for that test. Occasionally I do that when a
> test fails just so I can make sure I can reproduce manually what
> the test is doing automatically.
Again, it's not just about running the tests, but being able to
actually debug a failing test effectively. Tests work best when
they're easy to understand and work with, and when a maven core-
integration-test fails, you can definitely see how this setup falls
down. Running and re-running the same test without change from the
IDE isn't useful for debugging, and running the invoker from this
kind of code with the remote debugging enabled is virtually
impossible...incidentally, if you've figured out how to do it, I'd be
interested in learning.
>
>
>> However, it's also critical to allow a project's POM to remain
>> unmodified for the purposes of orchestrating or verifying the
>> build. Modifying the POM introduces the potential to taint the
>> test results, and can limit flexibility in terms of verification.
>> For instance, if you need to simply scan the console output for a
>> particular message, it becomes much more difficult to do this if
>> you try to do it while the build is still running.
>
> I don't understand this remark, because I don't think either
> strategy requires anyone to "modify the POM"...? I hope it's just
> an artifact of the earlier misunderstanding.
This remark refers to one alternative to the JUnit/maven-verifier
approach that I've heard in the past, which is to inject the
assertions directly into the POM via something like the maven-
verifier-plugin, and always run to the verify phase. Even though I
now know you're against that approach, it's worth expanding a little.
This approach limits flexibility to test things like running a
multimodule build to the package or compile phases, not to mention
the log-checking that I mentioned before. It could also interfere
with other things bound to the verify phase, especially when those
are the items under test (thinking of integration tests for the
verifier plugin itself here).
In my opinion, there should certainly be hooks available to generate
a JUnit wrapper around an integration test, but that wrapper should
not carry information only exists outside the test project directory.
I'd favor something more like having an orchestrator POM that called
something like the invoker to run the real build using the real POM,
then verifies the results of that build. Then, all assertions are
contained within that orchestrator POM, and anyone could step into
that directory and either run something like 'mvn -f test-pom.xml
test' or else simply crack open the test-pom.xml - which is right
alongside the rest of the test resources for that case - and read
what it's doing. This makes it simpler to debug a failing integration-
test without stepping outside the test-case directory, and it also
provides a degree of documentation for the test. Best of all, it
could be written as a very simple archetype that generates a portable
test case which can live in almost any directory structure within the
integration-test aggregator build to make it easier to organize the
test cases according to functionality.
-john
>
> -Dan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
---
John Casey
Committer and PMC Member, Apache Maven
mail: jdcasey at commonjava dot org
blog: http://www.ejlife.net/blogs/john
rss: http://feeds.feedburner.com/ejlife/john
Re: Invoker vs. Verifier?
Posted by Dan Fabulich <da...@fabulich.com>.
John Casey wrote:
> First things first. maven-invoker and maven-invoker-plugin are not
> separate things. The maven-invoker-plugin uses maven-invoker, but
> maven-invoker is meant to be a reusable library, not just a plugin.
I find this remark quite confusing... if one is a library, and one is a
plugin, and they have two separate POMs, one for the library JAR
("maven-invoker") and one for the plugin ("maven-invoker-plugin") doesn't
that make them two things that share code?
Actually, I think our "disagreement" is mostly a misunderstanding, and I
think it mostly turns on this point of whether maven-invoker and
maven-invoker plugin are "two things."
In particular, I said that we should not use maven-invoker-plugin for
integration testing. But I didn't mean that we shouldn't use
maven-invoker!
e.g. when you said:
> Also, I'd like to know how you would suggest writing integration tests
> using junit without the invoker functionality (the real meat of the
> invoker plugin) and without access to the maven 2.1 embedder (it's not
> available in any real way in 2.0.x).
I never meant to suggest that we'd write integration tests without the
invoker. I meant that maven-verifier should *use* maven-invoker (the
library) and should continue to execute a separate instance of Maven.
We're in total agreement that integration testing should happen by running
real Maven projects and executing a real Maven to make that happen, for
all the reasons you laid out.
[I also argued that we should rename maven-verifier, because its name is
confusing.]
With that said, I think there is still a point of actual disagreement
lurking here which I'd like to flush out... namely whether it's best to
write your tests in Java/JUnit/TestNG or whether you'd want to
have/use/write a test-runner plugin (beyond Surefire that just runs
JUnit/TestNG) to handle these tests.
More generally, I think plugin integration test writers should write their
tests in the maven-verifier style (using a maven invocation library [e.g.
maven-invoker] under a JUnit test), and not using maven-invoker-plugin.
I think that because both tests of Maven Core and integration tests of a
plugin call for doing the same thing: invoking Maven and asserting on the
results.
> JUnit is best suited to unit tests, as its name suggests.
True, though JUnit is acceptable for integration tests too, (especially
JUnit 4) and TestNG is great for both. The two of them together beat the
pants off of almost everything else in Java land.
> The maven-verifier (not the verifier plugin, I know it's confusing) gets
> away from this, in that it requires a project directory in
> src/test/resources AND a JUnit test case to orchestrate the test. IMO,
> this makes it extremely difficult to run a single core integration test
> from the command line, so you can capture logs for offline examination,
> for instance.
With that, I definitely disagree. Can you say more about what's difficult
about running a single test?
Certainly it's trivial to run a single JUnit test from the command line
using Surefire: "mvn test -Dtest=FooTest" does the job nicely; I use that
all the time. And it's REALLY easy to run a single test from
Eclipse/IDEA.
Maybe you meant that you think maven-verifier tests are harder to WRITE
than writing goals.txt + beanshell tests? (I disagree with that, too, but
it's worth clarifying what we're talking about.)
> Again, the idea behind test builds driven by the invoker is to provide a
> test suite with a low barrier to entry, and with all the potential for
> configuration present in the main user's experience with Maven. It's not
> a panacea, but neither is writing JUnit tests that orchestrate and
> verify test-build resources in a separate directory structure, which can
> really only be run properly from the JUnit api.
You call it the "JUnit API" like it's this big hassle to run JUnit
tests... but it's really easy to run JUnit tests, both from the command
line and from an IDE; certainly 99% of our users know how to do it, and I
hope most of them do it frequently! :-)
Furthermore, you seem to imply here that you can't just cd into
src/test/resources and run the projects there by hand directly... but of
course you can do that when you want/need to. I often do that when I'm
first setting up an integration test, before I've written the JUnit
assertions for that test. Occasionally I do that when a test fails just
so I can make sure I can reproduce manually what the test is doing
automatically.
> However, it's also critical to allow a project's POM to remain
> unmodified for the purposes of orchestrating or verifying the build.
> Modifying the POM introduces the potential to taint the test results,
> and can limit flexibility in terms of verification. For instance, if you
> need to simply scan the console output for a particular message, it
> becomes much more difficult to do this if you try to do it while the
> build is still running.
I don't understand this remark, because I don't think either strategy
requires anyone to "modify the POM"...? I hope it's just an artifact of
the earlier misunderstanding.
-Dan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Invoker vs. Verifier?
Posted by John Casey <jd...@commonjava.org>.
First things first. maven-invoker and maven-invoker-plugin are not
separate things. The maven-invoker-plugin uses maven-invoker, but
maven-invoker is meant to be a reusable library, not just a plugin.
Aside from this, there is actually a good reason for running
integration tests using maven and maven plugins. Many users can
submit stripped-down project builds that exhibit a certain error;
fewer would be able (or willing) to submit the same failure in the
form of a junit test. A failing build should be able to be
incorporated directly into the test suite, and if it makes sense,
later be split into lower-level junit tests to check the behavior of
a certain component. JUnit is best suited to unit tests, as its name
suggests. For integration testing, where you actually need to
integrate the container (Maven, I mean) with a suite of plugins, IMO
it makes much more sense to provide the flexibility of executing
maven as a separate process (or embedded, this option simply hasn't
been developed for the invoker plugin yet...mainly because the 2.0.x
embedder sucks). This makes it trivial to test command-line options,
different versions of maven, or any number of configurations that are
somewhat external to Maven's core apis, without having to develop and
constantly verify the validity of testing harnesses. The maven-
verifier (not the verifier plugin, I know it's confusing) gets away
from this, in that it requires a project directory in src/test/
resources AND a JUnit test case to orchestrate the test. IMO, this
makes it extremely difficult to run a single core integration test
from the command line, so you can capture logs for offline
examination, for instance.
Again, the idea behind test builds driven by the invoker is to
provide a test suite with a low barrier to entry, and with all the
potential for configuration present in the main user's experience
with Maven. It's not a panacea, but neither is writing JUnit tests
that orchestrate and verify test-build resources in a separate
directory structure, which can really only be run properly from the
JUnit api.
Another thing to keep in mind is that you have two different
integration-testing profiles to deal with, and they are distinct:
1. testing the core features built into maven
2. testing the functionality of a single plugin in different situations
I'm not at all sure that it makes sense to try and merge these two
scenarios behind a single api...what would it give you? I do think
the verifier used in core-integration-tests should be merged with
some other, more general verification mechanism. However, it's also
critical to allow a project's POM to remain unmodified for the
purposes of orchestrating or verifying the build. Modifying the POM
introduces the potential to taint the test results, and can limit
flexibility in terms of verification. For instance, if you need to
simply scan the console output for a particular message, it becomes
much more difficult to do this if you try to do it while the build is
still running.
Just to be clear, I don't think writing invoker-plugin-driven test
builds is a good substitute for writing unit tests. Unit tests will
cover the code far more completely in most cases, and as you pointed
out, can be graphed and rendered into all sorts of shiny reports much
more readily.
Also, I'd like to know how you would suggest writing integration
tests using junit without the invoker functionality (the real meat of
the invoker plugin) and without access to the maven 2.1 embedder
(it's not available in any real way in 2.0.x).
I guess what I'm wondering is how these tests will really look, once
you've taken out the verifier and eliminated the use of the invoker
(for the most part, at least)? You say JUnit or even Beanshell, but
those are just technologies; how will you use them?
-john
On Dec 11, 2007, at 10:06 PM, Dan Fabulich wrote:
> John Casey wrote:
>
>> What you're seeing as overlap is a mixture of concerns in the
>> invoker plugin. The verifications beanshell really needs to be
>> migrated out to some sort of proper integration-testing plugin
>> (or, even better, a plugin that unites invoker and verifier under
>> a common configuration...then extend the verifier with the
>> invoker's beanshell functionality). Regardless, the invoker plugin
>> can be used for any sort of scenario where you need to fork a new
>> maven process. I've personally used it to proxy secondary builds
>> in some sticky client use cases. You don't have to use the
>> beanshell script to verify the build, it's just an [admittedly
>> confusing] option.
>
> As I've remarked before, I find it weird that various Maven
> developers have gone and written _plugins_ to do Maven integration
> testing.
>
> Integration tests are just tests; we know how to write/run tests
> using real test frameworks like JUnit and TestNG. Those frameworks
> are pretty cool; you can do stuff like rerun failures-only, graph
> results over time, write data-driven tests, etc. You can even use
> them to write tests in scripting languages like Groovy, BeanShell,
> etc. All that AND you get excellent IDE integration.
>
> More generally, while I certainly see the value of a maven-invoker-
> plugin, I don't expect that you'd want that to be the "normal" way
> people would write Maven integration tests.
>
> Right now there are four things: maven-verifier, maven-verifier-
> plugin (no relation!), maven-invoker, and maven-invoker-plugin.
>
> I think I'd like to advocate ripping out the bulk of maven-verifier
> and make it depend entirely on maven-invoker. Since maven-verifier
> is so confusingly named, I think I'd want to take the good bits out
> and put them in maven-integration-test-helper (which is what maven-
> verifier really is, anyway).
>
> More controversially (?) I'd like to deprecate the idea of writing
> *tests* using the maven-invoker-plugin, instead preferring to write
> them in Java (or BeanShell, I'm easy!) running them using a "real"
> test framework. maven-invoker-plugin should still be used for
> spawning sub-builds in those delightful cases where that's necessary.
>
> Thoughts?
>
> -Dan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
---
John Casey
Committer and PMC Member, Apache Maven
mail: jdcasey at commonjava dot org
blog: http://www.ejlife.net/blogs/john
rss: http://feeds.feedburner.com/ejlife/john
Re: Invoker vs. Verifier?
Posted by olivier lamy <ol...@apache.org>.
Hi,
IMHO, don't use maven-embedder.
Because for a plugin with a required maven version >= 2.0.6, tests with
embedder will failed.
Look at the jar plugin all tests has been migrated to maven-invoker-plugin
due to this.
--
Olivier
2007/12/12, Johan Kindgren <jo...@acrend.se>:
>
> I created a few integration-tests for the jar-plugin using the Maven
> Embedder as described in
> http://maven.apache.org/developers/committer-testing-plugins.html
> under both maven-it-plugin and maven-plugin-test-plugin (I didn't use
> any testing plugins, just the install-plugin). Add the Embedder to
> your list of testing utilities.
>
> I think a lot of things could be improved in the testing strategies.
> The bugs that I fixed in the jar-plugin took me less than an hour to
> complete, but creating the testcases to me about three days before I
> finally found the above mentioned page.
>
> Maybe I've missunderstood something completely, but as far as I
> understand some tests are auctually testing the result from the
> previous build?
>
> /Johan
>
> 2007/12/12, Brian E. Fox <br...@reply.infinity.nu>:
> > You forgot the maven-plugin-testing-harness ;-)
> >
> >
> >
> > -----Original Message-----
> > From: Dan Fabulich [mailto:dan@fabulich.com]
> > Sent: Tuesday, December 11, 2007 9:07 PM
> > To: Maven Developers List
> > Subject: Re: Invoker vs. Verifier?
> >
> > John Casey wrote:
> >
> > > What you're seeing as overlap is a mixture of concerns in the invoker
> > > plugin. The verifications beanshell really needs to be migrated out to
> >
> > > some sort of proper integration-testing plugin (or, even better, a
> > > plugin that unites invoker and verifier under a common
> > > configuration...then extend the verifier with the invoker's beanshell
> > > functionality). Regardless, the invoker plugin can be used for any
> > sort
> > > of scenario where you need to fork a new maven process. I've
> > personally
> > > used it to proxy secondary builds in some sticky client use cases. You
> >
> > > don't have to use the beanshell script to verify the build, it's just
> > an
> > > [admittedly confusing] option.
> >
> > As I've remarked before, I find it weird that various Maven developers
> > have gone and written _plugins_ to do Maven integration testing.
> >
> > Integration tests are just tests; we know how to write/run tests using
> > real test frameworks like JUnit and TestNG. Those frameworks are pretty
> >
> > cool; you can do stuff like rerun failures-only, graph results over
> > time,
> > write data-driven tests, etc. You can even use them to write tests in
> > scripting languages like Groovy, BeanShell, etc. All that AND you get
> > excellent IDE integration.
> >
> > More generally, while I certainly see the value of a
> > maven-invoker-plugin,
> > I don't expect that you'd want that to be the "normal" way people would
> > write Maven integration tests.
> >
> > Right now there are four things: maven-verifier, maven-verifier-plugin
> > (no
> > relation!), maven-invoker, and maven-invoker-plugin.
> >
> > I think I'd like to advocate ripping out the bulk of maven-verifier and
> > make it depend entirely on maven-invoker. Since maven-verifier is so
> > confusingly named, I think I'd want to take the good bits out and put
> > them
> > in maven-integration-test-helper (which is what maven-verifier really
> > is,
> > anyway).
> >
> > More controversially (?) I'd like to deprecate the idea of writing
> > *tests*
> > using the maven-invoker-plugin, instead preferring to write them in Java
> >
> > (or BeanShell, I'm easy!) running them using a "real" test framework.
> > maven-invoker-plugin should still be used for spawning sub-builds in
> > those
> > delightful cases where that's necessary.
> >
> > Thoughts?
> >
> > -Dan
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > For additional commands, e-mail: dev-help@maven.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > For additional commands, e-mail: dev-help@maven.apache.org
> >
> >
>
>
> --
> ____________________________
> Johan Kindgren
> Acrend AB
> Phone: +46 (0) 733-58 36 60
> E-mail: johan.kindgren@acrend.se
> www.acrend.se
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
Re: Invoker vs. Verifier?
Posted by Johan Kindgren <jo...@acrend.se>.
I created a few integration-tests for the jar-plugin using the Maven
Embedder as described in
http://maven.apache.org/developers/committer-testing-plugins.html
under both maven-it-plugin and maven-plugin-test-plugin (I didn't use
any testing plugins, just the install-plugin). Add the Embedder to
your list of testing utilities.
I think a lot of things could be improved in the testing strategies.
The bugs that I fixed in the jar-plugin took me less than an hour to
complete, but creating the testcases to me about three days before I
finally found the above mentioned page.
Maybe I've missunderstood something completely, but as far as I
understand some tests are auctually testing the result from the
previous build?
/Johan
2007/12/12, Brian E. Fox <br...@reply.infinity.nu>:
> You forgot the maven-plugin-testing-harness ;-)
>
>
>
> -----Original Message-----
> From: Dan Fabulich [mailto:dan@fabulich.com]
> Sent: Tuesday, December 11, 2007 9:07 PM
> To: Maven Developers List
> Subject: Re: Invoker vs. Verifier?
>
> John Casey wrote:
>
> > What you're seeing as overlap is a mixture of concerns in the invoker
> > plugin. The verifications beanshell really needs to be migrated out to
>
> > some sort of proper integration-testing plugin (or, even better, a
> > plugin that unites invoker and verifier under a common
> > configuration...then extend the verifier with the invoker's beanshell
> > functionality). Regardless, the invoker plugin can be used for any
> sort
> > of scenario where you need to fork a new maven process. I've
> personally
> > used it to proxy secondary builds in some sticky client use cases. You
>
> > don't have to use the beanshell script to verify the build, it's just
> an
> > [admittedly confusing] option.
>
> As I've remarked before, I find it weird that various Maven developers
> have gone and written _plugins_ to do Maven integration testing.
>
> Integration tests are just tests; we know how to write/run tests using
> real test frameworks like JUnit and TestNG. Those frameworks are pretty
>
> cool; you can do stuff like rerun failures-only, graph results over
> time,
> write data-driven tests, etc. You can even use them to write tests in
> scripting languages like Groovy, BeanShell, etc. All that AND you get
> excellent IDE integration.
>
> More generally, while I certainly see the value of a
> maven-invoker-plugin,
> I don't expect that you'd want that to be the "normal" way people would
> write Maven integration tests.
>
> Right now there are four things: maven-verifier, maven-verifier-plugin
> (no
> relation!), maven-invoker, and maven-invoker-plugin.
>
> I think I'd like to advocate ripping out the bulk of maven-verifier and
> make it depend entirely on maven-invoker. Since maven-verifier is so
> confusingly named, I think I'd want to take the good bits out and put
> them
> in maven-integration-test-helper (which is what maven-verifier really
> is,
> anyway).
>
> More controversially (?) I'd like to deprecate the idea of writing
> *tests*
> using the maven-invoker-plugin, instead preferring to write them in Java
>
> (or BeanShell, I'm easy!) running them using a "real" test framework.
> maven-invoker-plugin should still be used for spawning sub-builds in
> those
> delightful cases where that's necessary.
>
> Thoughts?
>
> -Dan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
--
____________________________
Johan Kindgren
Acrend AB
Phone: +46 (0) 733-58 36 60
E-mail: johan.kindgren@acrend.se
www.acrend.se
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
RE: Invoker vs. Verifier?
Posted by "Brian E. Fox" <br...@reply.infinity.nu>.
You forgot the maven-plugin-testing-harness ;-)
-----Original Message-----
From: Dan Fabulich [mailto:dan@fabulich.com]
Sent: Tuesday, December 11, 2007 9:07 PM
To: Maven Developers List
Subject: Re: Invoker vs. Verifier?
John Casey wrote:
> What you're seeing as overlap is a mixture of concerns in the invoker
> plugin. The verifications beanshell really needs to be migrated out to
> some sort of proper integration-testing plugin (or, even better, a
> plugin that unites invoker and verifier under a common
> configuration...then extend the verifier with the invoker's beanshell
> functionality). Regardless, the invoker plugin can be used for any
sort
> of scenario where you need to fork a new maven process. I've
personally
> used it to proxy secondary builds in some sticky client use cases. You
> don't have to use the beanshell script to verify the build, it's just
an
> [admittedly confusing] option.
As I've remarked before, I find it weird that various Maven developers
have gone and written _plugins_ to do Maven integration testing.
Integration tests are just tests; we know how to write/run tests using
real test frameworks like JUnit and TestNG. Those frameworks are pretty
cool; you can do stuff like rerun failures-only, graph results over
time,
write data-driven tests, etc. You can even use them to write tests in
scripting languages like Groovy, BeanShell, etc. All that AND you get
excellent IDE integration.
More generally, while I certainly see the value of a
maven-invoker-plugin,
I don't expect that you'd want that to be the "normal" way people would
write Maven integration tests.
Right now there are four things: maven-verifier, maven-verifier-plugin
(no
relation!), maven-invoker, and maven-invoker-plugin.
I think I'd like to advocate ripping out the bulk of maven-verifier and
make it depend entirely on maven-invoker. Since maven-verifier is so
confusingly named, I think I'd want to take the good bits out and put
them
in maven-integration-test-helper (which is what maven-verifier really
is,
anyway).
More controversially (?) I'd like to deprecate the idea of writing
*tests*
using the maven-invoker-plugin, instead preferring to write them in Java
(or BeanShell, I'm easy!) running them using a "real" test framework.
maven-invoker-plugin should still be used for spawning sub-builds in
those
delightful cases where that's necessary.
Thoughts?
-Dan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Invoker vs. Verifier?
Posted by Mauro Talevi <ma...@aquilonia.org>.
Dan Fabulich wrote:
> John Casey wrote:
>
>> What you're seeing as overlap is a mixture of concerns in the invoker
>> plugin. The verifications beanshell really needs to be migrated out to
>> some sort of proper integration-testing plugin (or, even better, a
>> plugin that unites invoker and verifier under a common
>> configuration...then extend the verifier with the invoker's beanshell
>> functionality). Regardless, the invoker plugin can be used for any
>> sort of scenario where you need to fork a new maven process. I've
>> personally used it to proxy secondary builds in some sticky client use
>> cases. You don't have to use the beanshell script to verify the build,
>> it's just an [admittedly confusing] option.
>
> As I've remarked before, I find it weird that various Maven developers
> have gone and written _plugins_ to do Maven integration testing.
>
> Integration tests are just tests; we know how to write/run tests using
> real test frameworks like JUnit and TestNG. Those frameworks are pretty
> cool; you can do stuff like rerun failures-only, graph results over
> time, write data-driven tests, etc. You can even use them to write
> tests in scripting languages like Groovy, BeanShell, etc. All that AND
> you get excellent IDE integration.
>
> More generally, while I certainly see the value of a
> maven-invoker-plugin, I don't expect that you'd want that to be the
> "normal" way people would write Maven integration tests.
>
> Right now there are four things: maven-verifier, maven-verifier-plugin
> (no relation!), maven-invoker, and maven-invoker-plugin.
>
> I think I'd like to advocate ripping out the bulk of maven-verifier and
> make it depend entirely on maven-invoker. Since maven-verifier is so
> confusingly named, I think I'd want to take the good bits out and put
> them in maven-integration-test-helper (which is what maven-verifier
> really is, anyway).
>
> More controversially (?) I'd like to deprecate the idea of writing
> *tests* using the maven-invoker-plugin, instead preferring to write them
> in Java (or BeanShell, I'm easy!) running them using a "real" test
> framework. maven-invoker-plugin should still be used for spawning
> sub-builds in those delightful cases where that's necessary.
>
> Thoughts?
Dan,
I agree with you - integration tests would benefit from having a more
unified approach, just like unit tests are run by surefire. It should
support different languages, Java and scripting, and different runners.
I would still have a different plugin for ITs than for unit tests, as
they typically are run with different usecases, and I fear that surefire
would get overloaded.
Cheers
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Invoker vs. Verifier?
Posted by Dan Fabulich <da...@fabulich.com>.
John Casey wrote:
> What you're seeing as overlap is a mixture of concerns in the invoker
> plugin. The verifications beanshell really needs to be migrated out to
> some sort of proper integration-testing plugin (or, even better, a
> plugin that unites invoker and verifier under a common
> configuration...then extend the verifier with the invoker's beanshell
> functionality). Regardless, the invoker plugin can be used for any sort
> of scenario where you need to fork a new maven process. I've personally
> used it to proxy secondary builds in some sticky client use cases. You
> don't have to use the beanshell script to verify the build, it's just an
> [admittedly confusing] option.
As I've remarked before, I find it weird that various Maven developers
have gone and written _plugins_ to do Maven integration testing.
Integration tests are just tests; we know how to write/run tests using
real test frameworks like JUnit and TestNG. Those frameworks are pretty
cool; you can do stuff like rerun failures-only, graph results over time,
write data-driven tests, etc. You can even use them to write tests in
scripting languages like Groovy, BeanShell, etc. All that AND you get
excellent IDE integration.
More generally, while I certainly see the value of a maven-invoker-plugin,
I don't expect that you'd want that to be the "normal" way people would
write Maven integration tests.
Right now there are four things: maven-verifier, maven-verifier-plugin (no
relation!), maven-invoker, and maven-invoker-plugin.
I think I'd like to advocate ripping out the bulk of maven-verifier and
make it depend entirely on maven-invoker. Since maven-verifier is so
confusingly named, I think I'd want to take the good bits out and put them
in maven-integration-test-helper (which is what maven-verifier really is,
anyway).
More controversially (?) I'd like to deprecate the idea of writing *tests*
using the maven-invoker-plugin, instead preferring to write them in Java
(or BeanShell, I'm easy!) running them using a "real" test framework.
maven-invoker-plugin should still be used for spawning sub-builds in those
delightful cases where that's necessary.
Thoughts?
-Dan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Invoker vs. Verifier?
Posted by John Casey <jd...@commonjava.org>.
What you're seeing as overlap is a mixture of concerns in the invoker
plugin. The verifications beanshell really needs to be migrated out
to some sort of proper integration-testing plugin (or, even better, a
plugin that unites invoker and verifier under a common
configuration...then extend the verifier with the invoker's beanshell
functionality). Regardless, the invoker plugin can be used for any
sort of scenario where you need to fork a new maven process. I've
personally used it to proxy secondary builds in some sticky client
use cases. You don't have to use the beanshell script to verify the
build, it's just an [admittedly confusing] option.
-john
On Dec 11, 2007, at 6:28 PM, Dan Fabulich wrote:
>
> Forgive me if I'm picking at a sore spot, but can someone help me
> understand the difference/overlap between maven-invoker and maven-
> verifier?
>
> As I understand it, they both do roughly the same thing, except one
> of them is a Maven plugin where you write your test in a goals.txt
> file + beanshell, and the other is meant to be run programmatically
> from a JUnit test. Is that essentially correct?
>
> (Whereas maven-verifier-plugin is another beast entirely, simply
> checking for the presence/absence of certian files.)
>
> Am I right in thinking that, despite their substantial overlap,
> maven-verifier and maven-invoker share no code?
>
> -Dan
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
---
John Casey
Committer and PMC Member, Apache Maven
mail: jdcasey at commonjava dot org
blog: http://www.ejlife.net/blogs/john
rss: http://feeds.feedburner.com/ejlife/john