Measuring integration test code coverage for a Maven plugin

[Dan Fabulich <da...@fabulich.com>]

I recently added a bunch of integration tests for Surefire.  These 
integration tests automatically fork a separate Maven process to run real 
Maven builds, like the Maven core integration tests do.

This naturally led me to wonder: Does Surefire (now) have reasonable code 
coverage?  Specifically, which lines in Surefire were covered by unit 
tests, which by integration tests, and which weren't covered at all?

I know there's a variety of handy code coverage tools that work with 
Maven, allowing you to instrument classes for code coverage and run your 
unit tests against the instrumented classes.

The catch in this case is that I need to somehow convince Maven to use the 
instrumented version of my plugin, and not the regular "real" version of 
the plugin, when I go to run my integration tests.  The clover plugin, for 
example, doesn't seem to want to let me do that.

[On the other hand, maybe I should just use an instrumenting JVM 
instead...?  Java 1.5's new java.lang.instrument would probably do the 
trick, but I'm not aware of any code coverage tool that works with 
j.l.instrument, and anyway I'd have to fix SUREFIRE-179 just to get it to 
work... :-)]

Has anybody ever done this before?  More generally, I don't think I've 
ever seen an example of anyone using Maven to run multi-process 
integration tests (e.g. cargo tests) and also measuring code coverage on 
those integration tests.  Has anyone seen a good example of this that I 
could reuse?

-Dan

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org

Re: Measuring integration test code coverage for a Maven plugin

[Dan Fabulich <da...@fabulich.com>]

Raphaël Piéroni wrote:

> My guess is that you should deploy your instrumented plugin in a test 
> repository.

That's basically what I wound up doing.

For the record, I started by specifying a special local repository in my 
setting directory.

Then I tried just using maven-clover-plugin 3.6, calling 
instrumentInternal directly (which you're not supposed to do because you 
risk getting cloverized stuff in your production output, though that's 
exactly what I wanted to do in my case).

That didn't quite work because instrumentInternal includes a function 
redirectArtifact() that changes the output jar to be blah-clover.jar; once 
I commented that out and added clover as a dependency to all of my code, I 
was able to run my integration tests and get a total view of my 
integration test coverage.

For the record, Surefire's unit tests covered 45% of Surefire; with the 
added integration tests we've covered 79.4% (87 classes, 3,323 / 4,187 
elements).  It also highlighted some key areas where we could add more 
tests, which was exactly what I wanted.

-Dan

2007/12/7, Dan Fabulich <da...@fabulich.com>:
>
> I recently added a bunch of integration tests for Surefire.  These
> integration tests automatically fork a separate Maven process to run real
> Maven builds, like the Maven core integration tests do.
>
> This naturally led me to wonder: Does Surefire (now) have reasonable code
> coverage?  Specifically, which lines in Surefire were covered by unit
> tests, which by integration tests, and which weren't covered at all?
>
> I know there's a variety of handy code coverage tools that work with
> Maven, allowing you to instrument classes for code coverage and run your
> unit tests against the instrumented classes.
>
> The catch in this case is that I need to somehow convince Maven to use the
> instrumented version of my plugin, and not the regular "real" version of
> the plugin, when I go to run my integration tests.  The clover plugin, for
> example, doesn't seem to want to let me do that.
>
> [On the other hand, maybe I should just use an instrumenting JVM
> instead...?  Java 1.5's new java.lang.instrument would probably do the
> trick, but I'm not aware of any code coverage tool that works with
> j.l.instrument, and anyway I'd have to fix SUREFIRE-179 just to get it to
> work... :-)]
>
> Has anybody ever done this before?  More generally, I don't think I've
> ever seen an example of anyone using Maven to run multi-process
> integration tests (e.g. cargo tests) and also measuring code coverage on
> those integration tests.  Has anyone seen a good example of this that I
> could reuse?
>
> -Dan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>

Re: Measuring integration test code coverage for a Maven plugin

[Raphaël Piéroni <ra...@gmail.com>]

Hi,
I never yet seen such thing.
But i would be very interested in the answer.

My guess is that you should deploy your
instrumented plugin in a test repository.


Regards

Raphaël

2007/12/7, Dan Fabulich <da...@fabulich.com>:
>
> I recently added a bunch of integration tests for Surefire.  These
> integration tests automatically fork a separate Maven process to run real
> Maven builds, like the Maven core integration tests do.
>
> This naturally led me to wonder: Does Surefire (now) have reasonable code
> coverage?  Specifically, which lines in Surefire were covered by unit
> tests, which by integration tests, and which weren't covered at all?
>
> I know there's a variety of handy code coverage tools that work with
> Maven, allowing you to instrument classes for code coverage and run your
> unit tests against the instrumented classes.
>
> The catch in this case is that I need to somehow convince Maven to use the
> instrumented version of my plugin, and not the regular "real" version of
> the plugin, when I go to run my integration tests.  The clover plugin, for
> example, doesn't seem to want to let me do that.
>
> [On the other hand, maybe I should just use an instrumenting JVM
> instead...?  Java 1.5's new java.lang.instrument would probably do the
> trick, but I'm not aware of any code coverage tool that works with
> j.l.instrument, and anyway I'd have to fix SUREFIRE-179 just to get it to
> work... :-)]
>
> Has anybody ever done this before?  More generally, I don't think I've
> ever seen an example of anyone using Maven to run multi-process
> integration tests (e.g. cargo tests) and also measuring code coverage on
> those integration tests.  Has anyone seen a good example of this that I
> could reuse?
>
> -Dan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>