Creating IT Tests

[Shane Isbell <sh...@gmail.com>]

On the current NMaven trunk, there are a number of IT tests using
maven-core-it-verifier, most of these are going to be invalid and need to be
rewritten.

Under the Maven 2.1 trunk, I see there are a lot of new core integration
testing plugins. Does anyone on the list have experience on using these? And
possibly be able to help out by setting up a few tests to get things
started?

Thanks,
Shane

Re: Creating IT Tests

[Jason van Zyl <ja...@maven.org>]

On 12 Dec 07, at 7:18 PM 12 Dec 07, Shane Isbell wrote:

> On Dec 12, 2007 3:49 PM, Brett Porter <br...@apache.org> wrote:
>
>> Are you looking to write ITs for the plugins, or the system as a
>> whole? As you can see from the dev@ discussion about the invoker, the
>> testing situation is still a bit of an inconsistent mess.
>
>
> I was thinking of ITs for the system as a whole.
>
>>
>>
>> The core-integration-testing module for the Maven system tests works
>> well and under the current structure that might be the best way to  
>> go.
>> I think Dan has replicated such an environment for plugin testing
>> which sounds perfect - though I have to admit I haven't looked at the
>> surefire tests just yet to see how that works.
>>
>> This actually brings me to a bigger question though - in Maven we
>> realised early on that each plugin would need to be decoupled from  
>> the
>> core release cycle. NMaven is still in the stage where all the  
>> plugins
>> are in sync with the larger system - is that expected to continue, or
>> is it likely the NMaven core libraries will be one thing released
>> together, and then the plugins folded into /plugins/ ?
>
>
> I would expect that as long as we remain in the incubator to release  
> the
> core and plugins together. When we graduate, decoupling the releases  
> makes
> sense, provided that the NMaven core API is stable.
>

Sounds like a reasonable plan. Decoupling too early just makes it hard  
to change when you discover you do need to make sweeping changes and  
NMaven is still pretty young.

> Shane
>
>>
>>
>> Cheers,
>> Brett
>>
>> On 13/12/2007, at 6:05 AM, Shane Isbell wrote:
>>
>>> On the current NMaven trunk, there are a number of IT tests using
>>> maven-core-it-verifier, most of these are going to be invalid and
>>> need to be
>>> rewritten.
>>>
>>> Under the Maven 2.1 trunk, I see there are a lot of new core
>>> integration
>>> testing plugins. Does anyone on the list have experience on using
>>> these? And
>>> possibly be able to help out by setting up a few tests to get things
>>> started?
>>>
>>> Thanks,
>>> Shane
>>
>>

Thanks,

Jason

----------------------------------------------------------
Jason van Zyl
Founder,  Apache Maven
jason at sonatype dot com
----------------------------------------------------------

the course of true love never did run smooth ...

-- Shakespeare 

Re: Creating IT Tests

[Brett Porter <br...@apache.org>]

On 13/12/2007, at 2:18 PM, Shane Isbell wrote:

> I would expect that as long as we remain in the incubator to release  
> the
> core and plugins together. When we graduate, decoupling the releases  
> makes
> sense, provided that the NMaven core API is stable.

Sounds reasonable to me - something to think towards in the future  
though.

One thing I do want to clarify though: while they are in some ways  
linked, NMaven is in the incubator to build a user-base and developer  
community, not finish the code. Plenty of projects have graduated  
without a release, and a stable release won't be the criteria for  
graduation either :)

I would generally be against sweeping changes using a "we're  
incubating" excuse, since that's the worst thing for community growth.  
However, I do think in the case of the move back to using more of  
Maven's code it's the right thing to do, since that will make it  
easier to get cross-community work and a smaller learning curve for  
contributors and should be a positive change.

- Brett

Re: Creating IT Tests

[Shane Isbell <sh...@gmail.com>]

On Dec 12, 2007 3:49 PM, Brett Porter <br...@apache.org> wrote:

> Are you looking to write ITs for the plugins, or the system as a
> whole? As you can see from the dev@ discussion about the invoker, the
> testing situation is still a bit of an inconsistent mess.


I was thinking of ITs for the system as a whole.

>
>
> The core-integration-testing module for the Maven system tests works
> well and under the current structure that might be the best way to go.
> I think Dan has replicated such an environment for plugin testing
> which sounds perfect - though I have to admit I haven't looked at the
> surefire tests just yet to see how that works.
>
> This actually brings me to a bigger question though - in Maven we
> realised early on that each plugin would need to be decoupled from the
> core release cycle. NMaven is still in the stage where all the plugins
> are in sync with the larger system - is that expected to continue, or
> is it likely the NMaven core libraries will be one thing released
> together, and then the plugins folded into /plugins/ ?


I would expect that as long as we remain in the incubator to release the
core and plugins together. When we graduate, decoupling the releases makes
sense, provided that the NMaven core API is stable.

Shane

>
>
> Cheers,
> Brett
>
> On 13/12/2007, at 6:05 AM, Shane Isbell wrote:
>
> > On the current NMaven trunk, there are a number of IT tests using
> > maven-core-it-verifier, most of these are going to be invalid and
> > need to be
> > rewritten.
> >
> > Under the Maven 2.1 trunk, I see there are a lot of new core
> > integration
> > testing plugins. Does anyone on the list have experience on using
> > these? And
> > possibly be able to help out by setting up a few tests to get things
> > started?
> >
> > Thanks,
> > Shane
>
>

Re: Creating IT Tests

[Dan Fabulich <da...@fabulich.com>]

Brett Porter wrote:

> The core-integration-testing module for the Maven system tests works well and 
> under the current structure that might be the best way to go. I think Dan has 
> replicated such an environment for plugin testing which sounds perfect - 
> though I have to admit I haven't looked at the surefire tests just yet to see 
> how that works.

My integration tests are basically just a copy of the 
core-integration-testing tests, which basically just use maven-verifier to 
launch instances of Maven and assert on the results.  There's a Wiki 
article on them here:

http://docs.codehaus.org/display/MAVEN/Creating+a+Maven+Integration+Test

Note that you should ignore the bit about using the archetype and using 
archetypeNG... the wiki says so, but the remark is small and easily 
overlooked.

Finally, in all honesty, I must point out that it is controversial whether 
the is the best on the market.  (Though, IMO, it is.)

Certainly there are plenty of other integration testing frameworks out 
there right now that people have advocated for, including 
maven-invoker-plugin, maven-plugin-testing-harness, shitty-maven-plugin 
and probably at least one more I'm forgetting.  There's a couple of 
threads on this very topic running on maven-dev.

-Dan

Re: Creating IT Tests

[Brett Porter <br...@apache.org>]

Are you looking to write ITs for the plugins, or the system as a  
whole? As you can see from the dev@ discussion about the invoker, the  
testing situation is still a bit of an inconsistent mess.

The core-integration-testing module for the Maven system tests works  
well and under the current structure that might be the best way to go.  
I think Dan has replicated such an environment for plugin testing  
which sounds perfect - though I have to admit I haven't looked at the  
surefire tests just yet to see how that works.

This actually brings me to a bigger question though - in Maven we  
realised early on that each plugin would need to be decoupled from the  
core release cycle. NMaven is still in the stage where all the plugins  
are in sync with the larger system - is that expected to continue, or  
is it likely the NMaven core libraries will be one thing released  
together, and then the plugins folded into /plugins/ ?

Cheers,
Brett

On 13/12/2007, at 6:05 AM, Shane Isbell wrote:

> On the current NMaven trunk, there are a number of IT tests using
> maven-core-it-verifier, most of these are going to be invalid and  
> need to be
> rewritten.
>
> Under the Maven 2.1 trunk, I see there are a lot of new core  
> integration
> testing plugins. Does anyone on the list have experience on using  
> these? And
> possibly be able to help out by setting up a few tests to get things
> started?
>
> Thanks,
> Shane