unit testing vs integration testing plugins

[Brett Porter <br...@apache.org>]

Hi,

I wanted to get some feedback from other devs on this topic.
Historically, I think we've done very little of both and would like to
change that. But most of the tests cropping up seem to be integration
tests, because they are probably an easier mechanism than to setup the
preconditions of the plugin. However, they are much larger in terms of
creation, checkout space and time to run.

I'd like to see that we "unit test" to the greatest extent possible.
IMO, the only reason to go to integration tests is to test lifecycle
interactions and interactions with other plugins.

What we probably require to do this is convenience methods to construct
a decent project and settings object and an expression populator based
on the project and settings.

Thoughts?

- Brett

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org

Re: unit testing vs integration testing plugins

[Jason van Zyl <ja...@maven.org>]

Brett Porter wrote:
> Hi,
> 
> I wanted to get some feedback from other devs on this topic.
> Historically, I think we've done very little of both and would like to
> change that. But most of the tests cropping up seem to be integration
> tests, because they are probably an easier mechanism than to setup the
> preconditions of the plugin. However, they are much larger in terms of
> creation, checkout space and time to run.

We have turned to the integration in the recent past because they more 
closely represent what the user will experience. Done more out of a 
sense of expediency and because it is somewhat difficult right now to 
quickly because of some of things people have mentioned in that we don't 
have an easy harness for many of the components.

> I'd like to see that we "unit test" to the greatest extent possible.
> IMO, the only reason to go to integration tests is to test lifecycle
> interactions and interactions with other plugins.

We should have plenty of both but to me the integration is the final 
word on what a user should experience. The symptom here is the decline 
of unit testing but in a lot of places for me the cause is that the 
components I'm trying to test are so course that in a short time span it 
is hard to test a component for which the source is ~1200, ~1400, ~1600 
lines long. Namely the plugin manager, project builder and life cycle 
executor. These are some of the most important components and they are 
not easy to follow and therefore not easy to test. I feel that's one of 
the root causes of why more unit testing is not happening. I ran out of 
time during the last release because I've not been in the core for a 
while and the project builder is just so big so I punted and made an 
integration test.

> What we probably require to do this is convenience methods to construct
> a decent project and settings object and an expression populator based
> on the project and settings.

The convenience methods and harnesses will certainly help and I think 
we've got quite a bit of code for that. What would also be cool  is to 
align the components we have in MNG with some archetypes for unit 
testing components of that particular flavour. Constructed in such a way 
that they can be isolated and easily attached to a JIRA issue while at 
the same time be easy to integrate in the existing suite of tests.

> - Brett

-- 

jvz.

Jason van Zyl
jason at maven.org
http://maven.apache.org

First, the taking in of scattered particulars under one Idea,
so that everyone understands what is being talked about ... Second,
the separation of the Idea into parts, by dividing it at the joints,
as nature directs, not breaking any limb in half as a bad carver might.

   -- Plato, Phaedrus (Notes on the Synthesis of Form by C. Alexander)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org

RE: unit testing vs integration testing plugins

[Vincent Massol <vm...@pivolis.com>]

I've been writing unit tests for the Cargo m2 plugin and it works quite
nice. I didn't feel any need for a framework. In some cases I've used jmock
to mock things.

You can check it out here:
http://svn.cargo.codehaus.org/cargo/trunk/extensions/maven2/src/test/

I agree with Brett about having a healthy balance of unit vs
integration/functional tests (in favor of unit tests in number), but there
must be functional tests as they are the only type of tests that prove that
it works (and it usually doesn't work).

Thanks
-Vincent

> -----Original Message-----
> From: carlossg@gmail.com [mailto:carlossg@gmail.com] On Behalf Of Carlos
> Sanchez
> Sent: mardi 31 janvier 2006 01:57
> To: Maven Developers List
> Subject: Re: unit testing vs integration testing plugins
> 
> We probably need to provide some plugin test infrastructure like mock
> objects and utilities to do the common test tasks.
> 
> On 1/30/06, Brett Porter <br...@apache.org> wrote:
> > Hi,
> >
> > I wanted to get some feedback from other devs on this topic.
> > Historically, I think we've done very little of both and would like to
> > change that. But most of the tests cropping up seem to be integration
> > tests, because they are probably an easier mechanism than to setup the
> > preconditions of the plugin. However, they are much larger in terms of
> > creation, checkout space and time to run.
> >
> > I'd like to see that we "unit test" to the greatest extent possible.
> > IMO, the only reason to go to integration tests is to test lifecycle
> > interactions and interactions with other plugins.
> >
> > What we probably require to do this is convenience methods to construct
> > a decent project and settings object and an expression populator based
> > on the project and settings.
> >
> > Thoughts?
> >
> > - Brett
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > For additional commands, e-mail: dev-help@maven.apache.org
> >
> >
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org

Re: unit testing vs integration testing plugins

[Carlos Sanchez <ca...@apache.org>]

We probably need to provide some plugin test infrastructure like mock
objects and utilities to do the common test tasks.

On 1/30/06, Brett Porter <br...@apache.org> wrote:
> Hi,
>
> I wanted to get some feedback from other devs on this topic.
> Historically, I think we've done very little of both and would like to
> change that. But most of the tests cropping up seem to be integration
> tests, because they are probably an easier mechanism than to setup the
> preconditions of the plugin. However, they are much larger in terms of
> creation, checkout space and time to run.
>
> I'd like to see that we "unit test" to the greatest extent possible.
> IMO, the only reason to go to integration tests is to test lifecycle
> interactions and interactions with other plugins.
>
> What we probably require to do this is convenience methods to construct
> a decent project and settings object and an expression populator based
> on the project and settings.
>
> Thoughts?
>
> - Brett
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org

Re: unit testing vs integration testing plugins

[John Casey <jd...@yahoo.com>]

It might be nice to have a harness which would allow unit testing 
plugins. The only problem is you have to test the test harness...this is 
a problem we've run into time and again with all of the test harnesses 
we have for Maven. I've personally fixed failures based on assumptions 
in the artifact test harness, the project testing harness, and the 
PlexusTestCase test harness.

My big worry here is that we'd be building a test harness that didn't 
accurately reflect reality, or which would have to be manually 
resynchronized with the reality of Maven as it changes. This may 
introduce subtle bugs...

Just my 2c.

-john

Jesse McConnell wrote:
> I have always kinda figured the rule of thumb regarding tests and artifacts
> is unit tests would test anything inside of the artifact and integration
> tests would encompass extra-artifact scope testing
> 
> and I totally agree, I think unit testing within mojo projects would be a
> lot more prelavent with a simple mechanism of bootstraping the plugin
> requirements..
> 
> jesse
> 
> On 1/30/06, Brett Porter <br...@apache.org> wrote:
>> Hi,
>>
>> I wanted to get some feedback from other devs on this topic.
>> Historically, I think we've done very little of both and would like to
>> change that. But most of the tests cropping up seem to be integration
>> tests, because they are probably an easier mechanism than to setup the
>> preconditions of the plugin. However, they are much larger in terms of
>> creation, checkout space and time to run.
>>
>> I'd like to see that we "unit test" to the greatest extent possible.
>> IMO, the only reason to go to integration tests is to test lifecycle
>> interactions and interactions with other plugins.
>>
>> What we probably require to do this is convenience methods to construct
>> a decent project and settings object and an expression populator based
>> on the project and settings.
>>
>> Thoughts?
>>
>> - Brett
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>>
> 
> 
> --
> jesse mcconnell
> jesseDOTmcconnellATgmailDOTcom
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org

Re: unit testing vs integration testing plugins

[Jesse McConnell <je...@gmail.com>]

I have always kinda figured the rule of thumb regarding tests and artifacts
is unit tests would test anything inside of the artifact and integration
tests would encompass extra-artifact scope testing

and I totally agree, I think unit testing within mojo projects would be a
lot more prelavent with a simple mechanism of bootstraping the plugin
requirements..

jesse

On 1/30/06, Brett Porter <br...@apache.org> wrote:
>
> Hi,
>
> I wanted to get some feedback from other devs on this topic.
> Historically, I think we've done very little of both and would like to
> change that. But most of the tests cropping up seem to be integration
> tests, because they are probably an easier mechanism than to setup the
> preconditions of the plugin. However, they are much larger in terms of
> creation, checkout space and time to run.
>
> I'd like to see that we "unit test" to the greatest extent possible.
> IMO, the only reason to go to integration tests is to test lifecycle
> interactions and interactions with other plugins.
>
> What we probably require to do this is convenience methods to construct
> a decent project and settings object and an expression populator based
> on the project and settings.
>
> Thoughts?
>
> - Brett
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>


--
jesse mcconnell
jesseDOTmcconnellATgmailDOTcom