Re: [discussion] Integration testing location

[Steve Loughran <st...@apache.org>]

Vincent Massol wrote:
> 
>> -----Original Message-----
>> From: Jesse McConnell [mailto:jesse.mcconnell@gmail.com]
>> Sent: mercredi 28 décembre 2005 20:44
>> To: Maven Developers List
>> Subject: Re: [discussion] Integration testing location
>>
>> I worry a bit about mixing unit and integration tests generally...
>>
>> maybe we have the recommended case for them go into
>> src/integration-test/java or something along those lines...
>>
>> logically the structure src/test/java and src/test/it doesn't do it for me
>> since it tests are probably written in java anyway, so that kinda breaks
>> the
>> spirit of 'src'/'type'/'language' convention we kind of have going..
> 
> Right. I hadn't seen it this way (I thought src/test was for all tests and
> that src/main was for all runtime sources) but I think you're right. I'm
> fine with src/it/java.
>  
>> as for the rest of it, I like adding the phases...
> 
> Sure but there's a very common use case for integration testing: the need to
> have environment setup before the test and to clean it after the tests. Of
> course you could write all sort of plugin to that the plugin support doing
> this itself but then you're no longer flexible and you're not providing a
> solution to lots of other use cases.
> 
> For example cargo could define a cargo:test goal which would start the
> container, run the tests and stop the containers. But this doesn't lead to
> all variations like:
> - the user simply wants to redeploy the new artifacts in the container but
> not start the container again
> - the user want to have his/her tests written using tetstng and not junit
> (or any other test framework)


Sounds more like functional testing to me...anything that needs a 
deployed system is a v. more complex situation.

I am working with a PhD student at CERN on distributed testing,  and 
there is a project gridunit that does some good stuff already, running 
junit tests across a farm of nodes, collecting and presenting the 
results. Clearly presenting the results gets more complex once you have 
tested on 20 boxes; you want to know what failed everywhere, what failed 
somewhere, and if there is any commonality for the partial failures.

The perspective we are taking is that a test run is just another thing 
to deploy; you have a test listener to collect results and logs from 
across the machines, then test runners on different machines running 
different tests. The listener collects the results, post-processes them 
and then you can act on the outcome (report failures, host the reports, 
etc).

In this view, functional testing of a deployment is just another 
deployment. Its different from a production deployment, but not very; 
just a different deployment descriptor to process.

-Steve

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org

Re: [discussion] Integration testing location

[Jason van Zyl <ja...@maven.org>]

Steve Loughran wrote:
> Vincent Massol wrote:
>  
> what we actually do at work is just run cruise control on a single box 
> and collect the results. Its that result collection which becomes 
> critical and hard on a big distributed system.

Just some other things to note FYI.

Continuum recently gained an addition from David Blevins (from the 
Geronimo team) which allows Continuum to run across several machines. So 
David essentially created a distributed test environment for the 
Geronimo TCK. It's still in the early stages but is working quite well: 
it uses ActiveMQ to move build definitions and information around.

There is also a tool called SysUnit at Codeahaus which Bob McWhirter 
created for distributed testing. Specifically for distributed caching 
products.

I have also recently talked to the guy behind ControlTier (Alex Honor), 
which has been recently open sourced, and would like to see where we can 
go with that (open.controltier.com). I talked with Vincent briefly about 
ControlTier the other day. Alex has a vast wealth of knowledge regarding 
deployment which for us is beneficial as Maven is the preferred tool to 
use in conjunction with ControlTier which is a plus in terms of 
integration with Maven.

-- 

jvz.

Jason van Zyl
jason at maven.org
http://maven.apache.org

A man enjoys hsi work when he understands the whole and when he
is responsible for the quality of the whole

  -- Christopher Alexander, A Pattern Language

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org

Re: [discussion] Integration testing location

[Steve Loughran <st...@apache.org>]

Vincent Massol wrote:

>  
>> I am working with a PhD student at CERN on distributed testing,  and
>> there is a project gridunit that does some good stuff already, running
>> junit tests across a farm of nodes, collecting and presenting the
>> results. 
> 
> Wow, that sounds really cool! Especially as there's a grid already in place
> (http://www.lsd.ufcg.edu.br/~alex/doku.php?id=gridunit).

yes, the grid is there. current gridunit can run a jar full of junit 
tests across many machines, I/we want to extend that to do testing of 
distributed systems, where you bring up some machines with your app then 
run the tests from a long way away.

There is also planetlab (http://planetlab.org), which is a large soup of 
random boxes out there, with random network QoS. Its there to let 
students/researchers test on a real network, not a simulation.

> 
> Is there an integration with a CI already in place? (I'd love to try it out
> for building Cargo - It has lots of tests and this would help a lot).

oh, no, nothing yet from my work. Talk to Alex about what he has done. I 
am part way through a cargo component for smartfrog though, also on my 
todo list is something to deploy cruise control itself.

what we actually do at work is just run cruise control on a single box 
and collect the results. Its that result collection which becomes 
critical and hard on a big distributed system.

> 
> Some time back I had blogged about distributed build
> (http://tinyurl.com/42zc7). This is still something I'd love to do. I agree
> that distributed test is a different topic than distributed build but
> combining the two would be even greater... ;-)

I havent looked at distributed builds much. Smartfrog can actually 
deploy any Ant task, so you could write your build in the language, 
though nobody does; we just use the tasks to avoid writing so many 
utilities ourselves, though when we do do the latter we get to implement 
cleanup on undeploy.

-steve

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org

RE: [discussion] Integration testing location

[Vincent Massol <vm...@pivolis.com>]

Hi Steve,

> -----Original Message-----
> From: Steve Loughran [mailto:stevel@apache.org]
> Sent: mercredi 4 janvier 2006 14:37
> To: Maven Developers List
> Subject: Re: [discussion] Integration testing location

[snip]

> > For example cargo could define a cargo:test goal which would start the
> > container, run the tests and stop the containers. But this doesn't lead
> to
> > all variations like:
> > - the user simply wants to redeploy the new artifacts in the container
> but
> > not start the container again
> > - the user want to have his/her tests written using tetstng and not
> junit
> > (or any other test framework)
> 
> 
> Sounds more like functional testing to me...anything that needs a
> deployed system is a v. more complex situation.

Ah, I guess it depends on the definition of it vs functional tests. For me
it's only a question of context. For example, I have a project with wars,
ejbs, ears. I want to be able to do two things:

- Inside the ejb project, I want to be able to run unit tests but also
integration tests. I don't want to run many its. Just one or two that proves
that my ejb can be deployed and works. For that I want to use cargo without
having to define any configuration. Cargo is perfect for this use case as it
provides a default config for me. Still I call this IT and not functional
testing.

- I want to have a system/ project where I'll run functional tests (aka
system tests in this case) where I'll have all the different components
deployed (the ear and all). For this functional test, I want to run with the
real configuration I'll use for deployment.
 
> I am working with a PhD student at CERN on distributed testing,  and
> there is a project gridunit that does some good stuff already, running
> junit tests across a farm of nodes, collecting and presenting the
> results. 

Wow, that sounds really cool! Especially as there's a grid already in place
(http://www.lsd.ufcg.edu.br/~alex/doku.php?id=gridunit).

Is there an integration with a CI already in place? (I'd love to try it out
for building Cargo - It has lots of tests and this would help a lot).

Some time back I had blogged about distributed build
(http://tinyurl.com/42zc7). This is still something I'd love to do. I agree
that distributed test is a different topic than distributed build but
combining the two would be even greater... ;-)

> Clearly presenting the results gets more complex once you have
> tested on 20 boxes; you want to know what failed everywhere, what failed
> somewhere, and if there is any commonality for the partial failures.
> 
> The perspective we are taking is that a test run is just another thing
> to deploy; you have a test listener to collect results and logs from
> across the machines, then test runners on different machines running
> different tests. The listener collects the results, post-processes them
> and then you can act on the outcome (report failures, host the reports,
> etc).
> 
> In this view, functional testing of a deployment is just another
> deployment. Its different from a production deployment, but not very;
> just a different deployment descriptor to process.

Cool

-Vincent


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org