[m2] Clover plugin use cases (was RE: [m2] "Verification" phase required?)

[Vincent Massol <vm...@pivolis.com>]

> -----Original Message-----
> From: Brett Porter [mailto:brett@apache.org]
> Sent: lundi 13 juin 2005 02:05
> To: Maven Developers List
> Subject: Re: [m2] "Verification" phase required?
> 
> Are you now doing these checks as part of a normal "m2 install" build?

Yes

> That is probably a bit heavier than I'd have thought it should be, but
> probably a valid use case.

It's a test so I guess it's "normal" you may wish to run it in the "release"
lifecycle as part of "m2 install".

> Previously, you had talked about running it as "m2 clover:clover" and
> doing all of those checks in there.

Hmm... I had forgotten about that. I think that there is a valid use case
for having clover:check as part of "m2 install" even though it's a bit heavy
as you say. But it'd be optional of course. The same applies for checkstyle,
pmd, simian, etc.
 
> Do you think you could have a go at designing how you think this will be
> used, so we can compare it to the lifecycle design and propose
> solutions? Some of this still needs to be implemented, which I'll be
> working on this week.
> 
> Please enumerate all the use cases (including reports, as even if you
> aren't doing it now it is important), how the user will configure it and
> how the user will run it.

Yep, this is what I've done yesterday while implementing the clover plugin.
I've found only 2 use cases so far:

Use case 1:
===========

- Run "m2 site:site"
- It generates the clover report by calling the clover:report goal
- Still open: whether we spawn a "m2 install" lifecycle or not. Maybe we
shouldn't by default so that "m2 site:site" generates a report of what is
currently in the existing clover database. The user would run whatever
he/she wants before that (m2 install site). Or maybe we should check the
freshness of the clover database file against the source files (main and
test) and regenerate only if a source is newer that the clover db. I'm open
here. The same question will arise for the junit-report plugin in the
future.
- If the project has modules, clover:report will merge the clover databases
of the modules to generate an aggregated clover report. I guess this could
be turned on/off by a configuration property.

Use case 2:
===========

- Run "m2 install"
- It runs clover:instrument in the generate-sources phase and then the other
phase depending on the packaging type
- It runs clover:check as part of the "verification" phase (I've currently
bound it to the integration-tests phase while we discuss it). This is run
only if it is activated:

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-clover-plugin</artifactId>
        <version>1.0-SNAPSHOT</version>
        <configuration>
          <activated>true</activated>
          <targetPercentage>50</targetPercentage>
        </configuration>
      </plugin>
    </plugins>
  </build>

- It does NOT generate any report. For that you'll need to explicitely call
"m2 clover:report" or run the site generation (see use case 1).
- If the "activated" property is false, "m2 install" will not clover
anything.

Caveats:
--------

- Should we have 2 sets of sources and should we make sure that the clover
plugin does not affect the main sources so that we cannot mistakenly package
for production clovered sources? If so, we'll need to spawn a full new "m2
install" lifecycle from within the clover plugin. I guess we need this. I
haven't implemented it yet. ATM my clover plugin clovers the main sources
and not a copy. I'll need some help for how to implement this.

Use case 3:
===========

- Run "m2 clover:check" or "m2 clover:report"
- This would work even if the "activated" config property is false.
- Users of this use case will typically set "activated" to false so that
clover doesn't run automatically as part of "m2 install".

WDYT?

[snip]

Thanks
-Vincent


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org

RE: [m2] Clover plugin use cases (was RE: [m2] "Verification" phase required?)

[Vincent Massol <vm...@pivolis.com>]

> -----Original Message-----
> From: Brett Porter [mailto:brett@apache.org]
> Sent: mardi 14 juin 2005 07:58
> To: Maven Developers List
> Subject: Re: [m2] Clover plugin use cases (was RE: [m2] "Verification"
> phase required?)

[snip]

> >It's a test so I guess it's "normal" you may wish to run it in the
> "release" lifecycle as part of "m2 install".
> >
> We don't actually have a release lifecycle, so we need to think about
> how this applies. 

Ok, when I said "release" lifecycle I wasn't referring to a special phase. I
was referring to the main lifecycle. The reason I called it the "release"
lifecycle was to differentiate it from the site generation for example but
as it's not a lifecycle in itself I should simply have said "...you may wish
to run it in the lifecycle as part of..." :-)

[snip]

> >Use case 1:
> >===========
> >
> >- Run "m2 site:site"
> >- It generates the clover report by calling the clover:report goal
> >- Still open: whether we spawn a "m2 install" lifecycle or not.
> >
> I think we have to. It gets quite confusing which belongs to which
> otherwise, and you go through steps in the "wrong order" - eg compile,
> clover:compile, test, clover:test, ...
> 
> >Maybe we
> >shouldn't by default so that "m2 site:site" generates a report of what is
> >currently in the existing clover database. The user would run whatever
> >he/she wants before that (m2 install site).
> >
> I'd prefer that the information could be encapsulated in the POM, which
> the "forking" could do by specifying to the POM whether to run
> integration tests or normal tests to the clover plugin.

Ok let me rephrase it to see if I understand right. You're saying that
instead of us deciding what goal to run (like "install"), we would leave
that to the user who will define it as a config element in the POM. In that
manner, the user will be free to either specify no goal or any other "test"
or "integration-tests". Good idea, I like it. We can default it to the
"integration-tests" goal if not specified.

BTW, is there a Java API to programmatically call a goal/phase?

> >Or maybe we should check the
> >freshness of the clover database file against the source files (main and
> >test) and regenerate only if a source is newer that the clover db.
> >
> This is the long term goal, and I think we can do this. The best thing
> about this is that it works as designed now, and gets more efficient
> when we implement more timestamp checking later. And that checking can
> apply to anything taking a set of sources and a set of outputs.

yep

[snip]

> >Use case 2:
> >===========
> >
> >- Run "m2 install"
> >- It runs clover:instrument in the generate-sources phase and then the
> > other phase depending on the packaging type
> >
> >
> not sure what you mean by "and then the other phase depending on the
> packaging type".

s/and then the other phase depending on the packaging type/and then the
other goals till the install phase/
 
> >- It runs clover:check as part of the "verification" phase (I've
> currently
> >bound it to the integration-tests phase while we discuss it).
> >
> I agree with a verification phase, but I think the goals in here should
> fork the goals they require each time, rerunning things when the inputs
> have changed and relying on timestamp checking to avoid doing too much
> work.
> 
> The reason I suggest this is that if you have something like jcoverage
> and clover both configured (which you might do if they produce different
> metrics), and the classes they produce are not compatible with each
> other, you really need to have two sets of output classes, and run the
> tests twice.

So you're saying that when the lifecycle reaches the verification phase, the
clover plugin (which is bound this verification phase) is called and
internally it sets different source roots and output dirs and runs again the
goal/phase defined in the POM a clover plugin config element (as we've
agreed above)?

Sounds cool to me.

BTW, for the record, one other use case that I've met in real life and that
the current plugins in m1 do not support is the mix between AspectJ and
Clover.
 
> I guess the downside is always running the tests twice when you really
> just want clover enabled on the normal run. I don't see any other way
> presently, though - a fundamental issue here is, "should you package a
> set of classes, different to what you have just tested?" I know it is
> theoretically ok for clover, it just sounds like dangerous ground.
> 
> Let's look at this operate in the real world, then aim to improve it in
> Maven 2.1. 

I agree.

> So clover:check gets bound to a verification phase, executes
> the new lifecycle, configured by an xml file to do the binding.

I don't understand the "configured by an xml file to do the binding" but I
guess that's the way to define a custom lifecycle. But do we need a new
lifecycle? Cannot we just rerun the existing one using the goal/phase
defined in the POM?
 
> I'm still not confident in this design. It seems that if verification is
> registered it should actually happen in test (like the test plugin
> itself does),

Why not let the user decide as we've agreed above by letting him/her define
the goal/phase to run in the POM? Or are you suggesting that we don't bind
the CloverCheckMojo to a verification phase but rather to the test phase? If
so, I don't see the rationale. I still hold that the clover check is akin to
a functional test and should happen after the integration-tests phase.

>  and everything funnels through the one lifecycle, but then
> if that goal is executed standalone it does the fork. I will think about
> it some more.

I'm probably not understanding something here.
 
> >This is run
> >only if it is activated:
> >
> >
> This only seems necessary to workaround the fact it may or may not be
> included in the packaging, so I wouldn't think it is really necessary.
> Activation should be done by the way the plugin is bound to the lifecycle.

I agree that if the user specifies the goal/phase in the POM, it's not
required as I was envisioning it.

I think we still need an "activated" config element but for a different
reason: simply to tell the clover plugin if a check should be performed on
percentage coverage. Users may want to generate reports but not perform a
check. We could name the config element "activateCheck" rather than
"activate".

[snip]
 
> >- Should we have 2 sets of sources and should we make sure that the
> clover
> >plugin does not affect the main sources so that we cannot mistakenly
> package
> >for production clovered sources? If so, we'll need to spawn a full new
> "m2
> >install" lifecycle from within the clover plugin. I guess we need this. I
> >haven't implemented it yet. ATM my clover plugin clovers the main sources
> >and not a copy. I'll need some help for how to implement this.
> >
> >
> You should be spitting out the clovered sources to a second directory,
> and IIUC you compile both and test with both.
> 
> I think where this gets tricky is packaging, which is only a problem
> when you want to clover something in integration tests. I'm confident we
> can handle this using the existing lifecycle and attached artifacts, but
> we'd need to flesh it out some more. I'd prefer we let this particular
> case slide to Maven 2.1, however.

What we would really need is a full parallel target tree. 

Alternatively we could possibly make clover part of the profile. So you
would have a profile that builds the project with clover on, using the main
tree and another profile (for production) that builds the project without
clover. As the generated artifacts go in the local repository we should also
separate them according to profiles. For example if you build a jar with JDK
1.4 and with JDK 1.5 you should name them differently. Something like:
<artifactId>-<version>-<profilename>.<extension>.

It sounds to me it would be a better way rather than duplicating the source
and buildt trees.

In term of continuous build you would run the main profile often and then,
say every few builds, you would run the clover profile (or vice-versa). Of
course that means that people building your project would use the main
profile by default and would need to specify the extra flag on the command
line to run the clover/jcoverage/etc profile.

The advantage would be speed. The downside would be several build profiles
that are not executed together. But that's already true for different JDKs,
etc.

WDYT?

> >Use case 3:
> >===========
> >
> >- Run "m2 clover:check" or "m2 clover:report"
> >- This would work even if the "activated" config property is false.
> >- Users of this use case will typically set "activated" to false so that
> >clover doesn't run automatically as part of "m2 install".
> >
> >
> Activation is by inclusion in the POM. I think execution of these goals
> would either have no configuration in the POM, or at least no executions
> listed (just using the plugin configuration to set what would be used
> from the command line).

You're talking about the goal/phase config here, right?
 
> So, to summarise:
> - add a verification phase, so it goes ..., test, ..., package,
> integration-test, verify, install, ... (though this is quite late -
> verify after packaging?)

It depends I guess. If you're only interested in clovering unit tests then
yes, probably. Otherwise no.

> - existing lifecycle design sufficies for each use case, as detailed
> below:
> 
> Use case 1: site:site
> - execution of clover report operates in either aggregation mode (top
> level), or execute mode (individual project).
> - execution of report in execute mode will first run m2 test or
> appropriately configured goal, using an overlaid lifecycle mapping and
> appropriately modified classes directories

You'll need to give me some pointers for how to do this "overlaid lifecycle
mapping". Again, is this required? Cannot we simply rerun the existing
lifecycle?

> - report then takes those results and produces the report from the
> database
> - works the same for clover:report standalone
> 
> Use case 2: clover:check in verification phase
> - execution of check works as above
> - takes those results and performs the check from the database
> 
> Use case 3: clover:check standalone
> - execution of check works as above
> - takes those results and performs the check from the database

Ok

Thanks for your feedback!
-Vincent


	

	
		
___________________________________________________________________________ 
Appel audio GRATUIT partout dans le monde avec le nouveau Yahoo! Messenger 
T�l�chargez cette version sur http://fr.messenger.yahoo.com

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org

Re: [m2] Clover plugin use cases (was RE: [m2] "Verification" phase required?)

[Brett Porter <br...@apache.org>]

Ok, responses inline.

Vincent Massol wrote:

>It's a test so I guess it's "normal" you may wish to run it in the "release"
>lifecycle as part of "m2 install".
>  
>
We don't actually have a release lifecycle, so we need to think about 
how this applies. I would like to keep it a little separate though. I 
think we might use a specific profile for a release - as you can affect 
the plugins bound there, as well as configuration (eg, set a higher 
standard for release, turn off debugging, add in a class obfuscation 
plugin, etc)

>Hmm... I had forgotten about that. I think that there is a valid use case
>for having clover:check as part of "m2 install" even though it's a bit heavy
>as you say. But it'd be optional of course. The same applies for checkstyle,
>pmd, simian, etc.
>  
>
I agree.

> 
>
>Yep, this is what I've done yesterday while implementing the clover plugin.
>I've found only 2 use cases so far:
>  
>
Thanks for this, it's much easier to get a whole view this way.

>Use case 1:
>===========
>
>- Run "m2 site:site"
>- It generates the clover report by calling the clover:report goal
>- Still open: whether we spawn a "m2 install" lifecycle or not. 
>
I think we have to. It gets quite confusing which belongs to which 
otherwise, and you go through steps in the "wrong order" - eg compile, 
clover:compile, test, clover:test, ...

>Maybe we
>shouldn't by default so that "m2 site:site" generates a report of what is
>currently in the existing clover database. The user would run whatever
>he/she wants before that (m2 install site). 
>
I'd prefer that the information could be encapsulated in the POM, which 
the "forking" could do by specifying to the POM whether to run 
integration tests or normal tests to the clover plugin.

>Or maybe we should check the
>freshness of the clover database file against the source files (main and
>test) and regenerate only if a source is newer that the clover db. 
>
This is the long term goal, and I think we can do this. The best thing 
about this is that it works as designed now, and gets more efficient 
when we implement more timestamp checking later. And that checking can 
apply to anything taking a set of sources and a set of outputs.

>- If the project has modules, clover:report will merge the clover databases
>of the modules to generate an aggregated clover report. I guess this could
>be turned on/off by a configuration property.
>  
>
Yep, we already have aggregation proposals lying around somewhere, and 
this is how it should work. The clover plugin needs to operate contextually.

>Use case 2:
>===========
>
>- Run "m2 install"
>- It runs clover:instrument in the generate-sources phase and then the other
>phase depending on the packaging type
>  
>
not sure what you mean by "and then the other phase depending on the 
packaging type".

>- It runs clover:check as part of the "verification" phase (I've currently
>bound it to the integration-tests phase while we discuss it). 
>
I agree with a verification phase, but I think the goals in here should 
fork the goals they require each time, rerunning things when the inputs 
have changed and relying on timestamp checking to avoid doing too much work.

The reason I suggest this is that if you have something like jcoverage 
and clover both configured (which you might do if they produce different 
metrics), and the classes they produce are not compatible with each 
other, you really need to have two sets of output classes, and run the 
tests twice.

I guess the downside is always running the tests twice when you really 
just want clover enabled on the normal run. I don't see any other way 
presently, though - a fundamental issue here is, "should you package a 
set of classes, different to what you have just tested?" I know it is 
theoretically ok for clover, it just sounds like dangerous ground.

Let's look at this operate in the real world, then aim to improve it in 
Maven 2.1. So clover:check gets bound to a verification phase, executes 
the new lifecycle, configured by an xml file to do the binding.

I'm still not confident in this design. It seems that if verification is 
registered it should actually happen in test (like the test plugin 
itself does), and everything funnels through the one lifecycle, but then 
if that goal is executed standalone it does the fork. I will think about 
it some more.

>This is run
>only if it is activated:
>  
>
This only seems necessary to workaround the fact it may or may not be 
included in the packaging, so I wouldn't think it is really necessary. 
Activation should be done by the way the plugin is bound to the lifecycle.

>- It does NOT generate any report. For that you'll need to explicitely call
>"m2 clover:report" or run the site generation (see use case 1).
>  
>
Agreed.

>- Should we have 2 sets of sources and should we make sure that the clover
>plugin does not affect the main sources so that we cannot mistakenly package
>for production clovered sources? If so, we'll need to spawn a full new "m2
>install" lifecycle from within the clover plugin. I guess we need this. I
>haven't implemented it yet. ATM my clover plugin clovers the main sources
>and not a copy. I'll need some help for how to implement this.
>  
>
You should be spitting out the clovered sources to a second directory, 
and IIUC you compile both and test with both.

I think where this gets tricky is packaging, which is only a problem 
when you want to clover something in integration tests. I'm confident we 
can handle this using the existing lifecycle and attached artifacts, but 
we'd need to flesh it out some more. I'd prefer we let this particular 
case slide to Maven 2.1, however.

>Use case 3:
>===========
>
>- Run "m2 clover:check" or "m2 clover:report"
>- This would work even if the "activated" config property is false.
>- Users of this use case will typically set "activated" to false so that
>clover doesn't run automatically as part of "m2 install".
>  
>
Activation is by inclusion in the POM. I think execution of these goals 
would either have no configuration in the POM, or at least no executions 
listed (just using the plugin configuration to set what would be used 
from the command line).

So, to summarise:
- add a verification phase, so it goes ..., test, ..., package, 
integration-test, verify, install, ... (though this is quite late - 
verify after packaging?)
- existing lifecycle design sufficies for each use case, as detailed below:

Use case 1: site:site
- execution of clover report operates in either aggregation mode (top 
level), or execute mode (individual project).
- execution of report in execute mode will first run m2 test or 
appropriately configured goal, using an overlaid lifecycle mapping and 
appropriately modified classes directories
- report then takes those results and produces the report from the database
- works the same for clover:report standalone

Use case 2: clover:check in verification phase
- execution of check works as above
- takes those results and performs the check from the database

Use case 3: clover:check standalone
- execution of check works as above
- takes those results and performs the check from the database

- Brett


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org