Re: [M2] clover

[Brett Porter <br...@apache.org>]

(sorry for mashing these together, my email client decided to download
and make them vanish, so I've pulled these from a digest request)

>Funny that you should say this when I've been unable to find a single unit
>test in all the plugins that currently exist for m2! :-)
>  
>
Yes, this is true. Some of them started out very simple, and some have
integration tests. There were some tests, but they were problematic for
the bootstrap (this should no longer be the case).

>That said I would be happy if you can show me how to write a significant
>unit test for http://rafb.net/paste/results/u2dtsf30.html
>
>  
>
You've addressed some of this below, but for comments on this itself:

#project.build.directory instead of #basedir/target
and try not to use #project directly if you can avoid it (this removes a
dependency on maven-core and makes it more reusable outside of Maven).

>What is needed is a test project using the plugin because I need source dirs
>and output dirs to test that Clover works. 
>
Surely you can do that without an integration test, since you can feed
in any parameters?

>Of course I also need to validate
>that all the plugin's magic (all the javadoc tags) works. 
>  
>
This is true...

> 
>I think we need to have a separate m2 project that tests the plugin
>(functional tests). I can already do this but in the same manner that we
>have asserts in the m1 plugin plugin we'll probably need helper APIs. I
>thought this was what you were doing in the it tests with the
>expected-results.txt and goals.txt files. The problem is that it was easy in
>m1 to have a maven.xml file to define the test setup, decide which goals to
>run and perform the assertions. Here's it's not doable easily (correct me if
>it is) so we'll probably need support for this (this is what you're doing
>with the txt files it seems).
>  
>
Well, I did say that I wanted to turn the verifier into an m2 plugin...
so if it does what you want, it can be used for that. I guess we can
make the "fork" of m2 be configurable. It's good for the integration
tests, to be sure all of m2 is working, but individual plugins might not
be so strict.

>Yep, almost :-)
>
>I have spoken with Jason on IRC yesterday and he said I should speak to you
>too but you were sleeping so I thought I might as well start something...
>:-)
>
>My first ideas are:
>1) Bind the plugin to the generate-sources phase
>  
>
Right, but you need to only activate it if clover is going to be used.
The configuration triggering is probably too much here - we might need
to look at an alternative.

>2) Modify the project source roots to replace the main source root with the
>one generated by Clover
>
>I have not thought at all about the reports, just about executing Clover for
>now.
>  
>
Ok, but how do you see the user invoking this? I'm worried you are
jumping down the implementation a little too fast... we need to think
about how it will be used.

Maybe the clover:on approach is right, but I'd like to think of
alternatives first - does it make more sense to just run "clover:clover"
and have that do the binding and run the tests (or configure a goal in
the pom), or do you really want to have clover:on?

>
>I'm planning to use the following directory structure:
>
>maven-clover-plugin/
>  |_ plugin/
>    |_ pom.xml
>    |_ [this is the plugin itself]
>  |_ tests/
>    |_ testAAAA
>      |_ pom.xml
>    |_ [...]
>    |_ testNNNN
>      |_ pom.xml
>  |_ pom.xml
>
>So that to build the plugin you would go at the top level and type "m2
>install" and it will run the integration tests too.
>
>What do you think?
>
>  
>
I'm personally still in favour of single-artifact integration tests
remaining in the same project, what do others think?

maven-clover-plugin/
  |_ src/
    |_ integration-tests/
      |_ testAAAA
        |_ pom.xml
      |_ [...]
      |_ testNNNN
        |_ pom.xml
  |_ [this is the plugin itself]
  |_ pom.xml

>Reporting is only a second step. The first step is that you need to
>instrument sources, get them to be compiled and executed. You don't want to
>execute the report right away. I think it should really be a second step not
>linked to the first step of getting the clover database ready (because the
>clover db can be enriched in different manners, including running manual
>tests). This is really how Clover works and this is why you have different
>Ant tasks of CLI tools in Clover (cloverInstr and *Reporters).
>  
>
Makes sense.

>Option 2 fits nicely for doing this but this is going in the direction of
>clover:on as it's exactly what it was doing.
>  
>
Maybe it's just the naming that is problematic for me. There's a couple
of things that I feel uncomfortable with:
- the need to specify it all the time when you are using it
- the ability to get the database out of sync when you request the
compile without it, which could then generate an incorrect report.
- doesn't gel with the lifecycle idea.

Now, if we're not doing a clover report, we must be:
a) building a database
b) erroring out if a certain coverage level is not hit?

without b or a report, I don't see the point unless I'm missing something.

So if it's (b), you'd want to do it every time? So you would wire it
into the normal lifecycle (though not into the produced jar).

> 
>True but that's the user's choice. He may want to have the instrumented code
>deployed (I do for example because I want acceptance tests to run with
>Clover so that I can check how much they cover of the code). Of course,
>there will also be some runs without clover but that's me the user who
>decides this.
>  
>
Sure, but you need to build different WARs with different names (this is
the intention of the "classifier" to build the same artifact differently
and be able to recognise it). We need to do everything possible to
prevent accidents.

>>I'm still working on the idea of alternate lifecycles (there is a JIRA
>>ticket regarding idea:idea and generated sources that covers the same
>>idea, and this applies to all reporting). What this means is that
>>clover:report would fork a lifecycle instance to run everything under
>>its own parameters. 
>>    
>>
>
>More precisely, what parameters are you talking about?
>
>  
>
the test inputs (or whatever goal is run, perhaps integration tests),
would be the clover classes instead of just the normal classes.

Thinking beyond clover, something might make a modification, not just an
addition - so it needs to be in a separate area.

>It is very tricky. I don't see how you could easily achieve this isolation.
>I don't think running the normal lifeycle with clovered sources is bad at
>all. If the user runs:
>
>m2 clover:on .... 
>
>he knows he'll have clovered sources. In any case he should rebuild the full
>thing with an automated build before sending to production so there's no
>chance he'll make a mistake.
>
>  
>
Yes, but I'm trying not to only think of clover here. There are other
things that modify the codebase (eg filtering) that can really mess with
you when you make a mistake.

>I'm open to all suggestions but it has to remain simple I think...
>  
>
The complexity is certainly going to be in m2, not in the plugin or user
interface...

> 
>
>I'm currently testing the modification of the source roots in order to get a
>clover db when running something like:
>
>m2 clover:on surefire:test
>
>And I want to write a m2 project for performing functional tests of this.
>This is where my other emails about testing m2 plugins comes into the
>picture.
>  
>
Ok, well its a good start... I'll have to try and understand better what
you are trying to achieve and whether there is a better alternative.
This doesn't feel quite right (and certainly, surefire:test won't work -
you need to run the "test" phase).

Cheers,
Brett

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org

RE: [M2] clover

[Vincent Massol <vm...@pivolis.com>]

> -----Original Message-----
> From: Brett Porter [mailto:brett@apache.org]
> Sent: dimanche 17 avril 2005 15:23
> To: dev@maven.apache.org
> Subject: Re: [M2] clover

[snip]

> #project.build.directory instead of #basedir/target

Makes sense. (If you want to fix it, I had copied it from
ModelloJavaMojo.java in maven-modello-plugin in the modello src tree).

> and try not to use #project directly if you can avoid it (this removes a
> dependency on maven-core and makes it more reusable outside of Maven).

The reason I have it is because I'd like to use something like:

project.addCompileSourceRoot( outputDirectory );
 
> >What is needed is a test project using the plugin because I need source
> dirs
> >and output dirs to test that Clover works.
> >
> Surely you can do that without an integration test, since you can feed
> in any parameters?

I do need a real physical src tree with real java files in there. I guess I
could use the test/resources tree for that but it sounds a bit awkward.

[snip]
 
> >My first ideas are:
> >1) Bind the plugin to the generate-sources phase
> >
> >
> Right, but you need to only activate it if clover is going to be used.
> The configuration triggering is probably too much here - we might need
> to look at an alternative.

I though that the trigger would be clover:on. Why wouldn't it work as a
trigger?

Also, there's something I'm not sure I understand. Where do you tell m2 that
such plugin is to be called when a lifecycle goal is invoked. For example,
invoking "m2 install" will invoke the maven-install-plugin because I guess
it's bound by default. What if I wanted to write another installer plugin
and let it be called instead of the default maven-install-plugin. How would
I do that?

Also, what if I wanted to have a plugin that is triggered just before (or
after) the maven-install-plugin. I guess I would bind it to the install
phase but how do I tell maven to execute it when "m2 install" is called?
 
> >2) Modify the project source roots to replace the main source root with
> the
> >one generated by Clover
> >
> >I have not thought at all about the reports, just about executing Clover
> for
> >now.
> >
> >
> Ok, but how do you see the user invoking this? I'm worried you are
> jumping down the implementation a little too fast... we need to think
> about how it will be used.

clover:on?
 
> Maybe the clover:on approach is right, but I'd like to think of
> alternatives first - does it make more sense to just run "clover:clover"
> and have that do the binding and run the tests (or configure a goal in
> the pom), or do you really want to have clover:on?

I would really prefer that clover:clover does not run the tests because the
type of test to be executed (unit tests, integration tests, manual tests,
performance tests, etc) are to be decided by the user. If we start linking
them together we would need as many goals (or configs) as there are
possibilities.

I think clover:on is the option that is the most flexible in term of clover
use cases.
 
> >
> >I'm planning to use the following directory structure:
> >
> >maven-clover-plugin/
> >  |_ plugin/
> >    |_ pom.xml
> >    |_ [this is the plugin itself]
> >  |_ tests/
> >    |_ testAAAA
> >      |_ pom.xml
> >    |_ [...]
> >    |_ testNNNN
> >      |_ pom.xml
> >  |_ pom.xml
> >
> >So that to build the plugin you would go at the top level and type "m2
> >install" and it will run the integration tests too.
> >
> >What do you think?
> >
> >
> >
> I'm personally still in favour of single-artifact integration tests
> remaining in the same project, what do others think?
> 
> maven-clover-plugin/
>   |_ src/
>     |_ integration-tests/
>       |_ testAAAA
>         |_ pom.xml
>       |_ [...]
>       |_ testNNNN
>         |_ pom.xml
>   |_ [this is the plugin itself]
>   |_ pom.xml

Oh that's very cool with me. Actually it solves a few questions I've had to
solve with the other solution. I'll change my structure to this.

I still have the issue of how to trigger the execution of the "verifier"
when building the top level plugin project. See other email.
 
[snip]

> >Option 2 fits nicely for doing this but this is going in the direction of
> >clover:on as it's exactly what it was doing.
> >
> >
> Maybe it's just the naming that is problematic for me. There's a couple
> of things that I feel uncomfortable with:
> - the need to specify it all the time when you are using it

Hmm... The problem is that you want to build your code both with and without
it so it cannot be a configuration option. 

I believe we need goal aliasing (or something equivalent) so we can define
project goal chains such as: 

cargo:clover --> clover:on test clover:report (or something similar)

> - the ability to get the database out of sync when you request the
> compile without it, which could then generate an incorrect report.

I don't understand this. If you run the compile without clover:on, then the
code will not get clovered and thus there'll be no clover db generated  when
the tests are executed. Thus if you run a clover:report afterwards, Clover
will say that the db is empty.

> - doesn't gel with the lifecycle idea.

I guess I don't know enough the lifecycle idea yet to really comment on
this. It does use the lifecycle by binding to the generate-sources phase.
 
> Now, if we're not doing a clover report, we must be:
> a) building a database
> b) erroring out if a certain coverage level is not hit?

Yes, that would be a goal that verifies the coverage level like
clover:check. So typically you would run: "m2 clover:on test clover:check"

> without b or a report, I don't see the point unless I'm missing something.
> 
> So if it's (b), you'd want to do it every time? So you would wire it
> into the normal lifecycle (though not into the produced jar).

I think I understand what you mean. You would want to have clover be
configuration option in the POM (for example) so that clover will get run
when running the normal lifecycle.

I like but I think there are 2 use cases:
1/ when the user wants to run clover for checking on coverage level
2/ when the user simply runs clover to get a report

For 1/, yes it would be nice to run clover automatically if clover is on in
the configuration. Thus the chain when the user type "m2 install" would be:

a/ somehow clover is triggered and instrumented sources are put in
target/clover/src

b/ when the compiler plugins executes, it compiles both the main sources in
target/classes and the instrumented sourcesi in say target/clover/classes.

c/ when the test phase executes, it runs the unit tests but also runs the
clover:check goal (after tests have executed - somehow) to verify the
coverage level

For 2/, it would be a pity to run clover every time you make a "m2 install"
as it'll take too much time. You'll only want it to be run when generating
reports which you don't execute as often as running "m2 install" (for
example, at work we run the build every hour and reports are only generated
once a day at night). You really want the "main" build to go as fast as
possible.

To summarize:

- I think that having a clover config in the POM to tell the build to always
run clover might be a good idea for projects that want to always ensure that
test coverage levels are checked. I have no idea how it can be triggered
automatically though

- We still need to run clover on demand, both the report builds and also for
projects where verifying the coverage level is not something you run at
every single build but only during integration builds.

> >True but that's the user's choice. He may want to have the instrumented
> code
> >deployed (I do for example because I want acceptance tests to run with
> >Clover so that I can check how much they cover of the code). Of course,
> >there will also be some runs without clover but that's me the user who
> >decides this.
> >
> >
> Sure, but you need to build different WARs with different names (this is
> the intention of the "classifier" to build the same artifact differently
> and be able to recognise it). We need to do everything possible to
> prevent accidents.

What is this "classifier"? Is that something available in m2-alpha-1?
 
> >>I'm still working on the idea of alternate lifecycles (there is a JIRA
> >>ticket regarding idea:idea and generated sources that covers the same
> >>idea, and this applies to all reporting). What this means is that
> >>clover:report would fork a lifecycle instance to run everything under
> >>its own parameters.
> >>
> >>
> >
> >More precisely, what parameters are you talking about?
> >
> >
> >
> the test inputs (or whatever goal is run, perhaps integration tests),
> would be the clover classes instead of just the normal classes.
> 
> Thinking beyond clover, something might make a modification, not just an
> addition - so it needs to be in a separate area.

I'm still not sure I fully understand what you mean. Clover makes
modifications not addition.

[snip]

> >I'm currently testing the modification of the source roots in order to
> get a
> >clover db when running something like:
> >
> >m2 clover:on surefire:test
> >
> >And I want to write a m2 project for performing functional tests of this.
> >This is where my other emails about testing m2 plugins comes into the
> >picture.
> >
> >
> Ok, well its a good start... I'll have to try and understand better what
> you are trying to achieve and whether there is a better alternative.
> This doesn't feel quite right (and certainly, surefire:test won't work -
> you need to run the "test" phase).

Ok for the test phase.

For the rest, I agree with you that we'll probably discover a generic
concept that is not there yet. I still need to understand better the
phases/lifecycle and the mapping of plugins to them (see my questions
above).

One thing that I'm still struggling with is whether you want all goals to be
called by using phases (and make it mandatory) or not. And if so, how do
users bind plugins to phases. In some cases I agree they can be bound by
default using the <packaging> type. But I don't think it covers all use
cases. Clover may be an example where it doesn't.

... some more thinking....

As I mentioned above, I think there are several build types that users want
to run. I can think of 3:

- "main" build: it builds the main artifacts
- "integration" build: it builds the main artifacts and perform extra
verifications like executing functional tests, checking clover coverage
levels, etc.
- "report" build: generate reports

Each of these build types will not require the same bindings on the phases.
Thus, I think bindings should be build-type dependent. That's if you want to
keep the same phase names for all possibly build types. 

Somehow the user needs to say "m2 <buildtype> <phase>" (it could default to
the "main" build type if the user only says "m2 <phase>").

The alternative is what is there in m1, i.e. the ability for the user to
define himself the ordering of goals by using custom top level goals. For
example, in cargo I have a cargo:build goal that does different thing for
the different subprojects. I have another cargo:clover goal, etc (see
http://svn.cargo.codehaus.org/cargo/trunk/maven.xml?rev=225&view=auto).

I think this is what is troubling me in m2 ATM: the ability to call either a
phase or a goal and the "magic" associated with a phase (how do I know what
is called when I run a phase?).

Thanks
-Vincent



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org