Integration Tests + Unit Tests in the same project?

[Peter Donald <pe...@realityforge.org>]

Hi,

I am in the process of converting a bunch of builds based on
ant/maven/rake to buildr. One of the patterns that I can not see how
to directly translate into buildr is the single project with multiple
test types within it. i.e. A project has integration and unit tests
both written in java and both stored in test hierarchy. It also has
ruby based integration tests sitting side by side the java test code.
AFAICS buildr is not really set up for this scenario. It seems that
for each project there can only be one test invocation, one test
framework and can not run parts at both integration testing time and
unit testing time. Is this correct?

Any suggestions on how to deal with this?

-- 
Cheers,

Peter Donald

Re: Integration Tests + Unit Tests in the same project?

[Peter Donald <pe...@realityforge.org>]

Hi,

On Wed, Sep 29, 2010 at 12:24 PM, Derrick Schneider
<de...@gmail.com> wrote:
> I got around it with a bit of a hack. I have a "longtests" environment in
> profile.yaml, and unless I'm in that environment, I exclude any test file
> named *LongTest. This, of course, means that developers have to know which
> is which. But it does allow me to have a mode where I do DBUnit
> setup/teardown, Spring config loading, and so on and so forth. (The build
> machine runs them at every check-in, but I assume developers will get
> frustrated if they have to run the time-consuming tests all the time.)

Right this is the approach that I ended up using. Tests that run
against a DB / Message Broker / etc are guarded against (with an
environment setting in my case that is set on the CI).

On Wed, Sep 29, 2010 at 1:20 PM, Rhett Sutphin
<rh...@detailedbalance.net> wrote:
> Yes.  Each buildr project can only have one test type (either integration or unit) and one test framework (junit, rspec, etc.).

I guess I see the first as a limitation of buildr and the other as a
rasonable assumption ;-)

>> Any suggestions on how to deal with this?
>
> I use a separate subproject for integration tests -- i.e., a subproject that only has tests in it and which depends on whatever module(s) I'm trying to integratedly test.  It might even be possible to have a subproject definition which, through the :base_dir option, executed tests which are physically alongside the unit tests in the parent project.  I haven't tried that, though.

Certainly an option but I can imagine that almost every bundle that we
write should have both unit and integration tests and it does not seem
entirely reasonable to double the number of projects to accomodate
this. Dooable - especially as we already wrap the define() method and
customize the project definition anyhoo but a bit ugly.

On Wed, Sep 29, 2010 at 3:21 PM, Alex Boisvert <al...@gmail.com> wrote:
> type1 = TestTask.define_task('test-type1')
> type1.include [...]
>
> type2 = TestTask.define_task('test-type2')
> type2.include [...]
>
> ...
>
> test.enhance [type1, type2]
>
> However as it stands, the code makes several assumptions that there's only
> one TestTask per project.   It's certainly doable but may require
> non-trivial changes to the code.

I have a stop gap measure in place but I guess what I would like to
see when/if I get a chance is something like

test.include 'org.example.SimpleTestCase'

integration.test.include 'org.example.MyLongRunningDbTest'

Something I may look at down the track.
-- 
Cheers,

Peter Donald

Re: Integration Tests + Unit Tests in the same project?

[Derrick Schneider <de...@gmail.com>]

I found the same thing (or at least the alternate wasn't obvious) when I
recently set up the same functionality.

I got around it with a bit of a hack. I have a "longtests" environment in
profile.yaml, and unless I'm in that environment, I exclude any test file
named *LongTest. This, of course, means that developers have to know which
is which. But it does allow me to have a mode where I do DBUnit
setup/teardown, Spring config loading, and so on and so forth. (The build
machine runs them at every check-in, but I assume developers will get
frustrated if they have to run the time-consuming tests all the time.)

Derrick

On Tue, Sep 28, 2010 at 7:15 PM, Antoine Toulme <an...@lunar-ocean.com>wrote:

> I think that's the case right now, indeed. Buildr was trying to be more
> granular and say, we're either building java, or building a big war. The
> big
> assembly project at the end is the one that should require integration
> tests.
>
> At least I think that was the idea.
>
> On Tue, Sep 28, 2010 at 18:16, Peter Donald <pe...@realityforge.org>
> wrote:
>
> > Hi,
> >
> > I am in the process of converting a bunch of builds based on
> > ant/maven/rake to buildr. One of the patterns that I can not see how
> > to directly translate into buildr is the single project with multiple
> > test types within it. i.e. A project has integration and unit tests
> > both written in java and both stored in test hierarchy. It also has
> > ruby based integration tests sitting side by side the java test code.
> > AFAICS buildr is not really set up for this scenario. It seems that
> > for each project there can only be one test invocation, one test
> > framework and can not run parts at both integration testing time and
> > unit testing time. Is this correct?
> >
> > Any suggestions on how to deal with this?
> >
> > --
> > Cheers,
> >
> > Peter Donald
> >
>



-- 
Writer. Programmer. Puzzle Designer.
http://www.obsessionwithfood.com

Re: Integration Tests + Unit Tests in the same project?

[Antoine Toulme <an...@lunar-ocean.com>]

I think that's the case right now, indeed. Buildr was trying to be more
granular and say, we're either building java, or building a big war. The big
assembly project at the end is the one that should require integration
tests.

At least I think that was the idea.

On Tue, Sep 28, 2010 at 18:16, Peter Donald <pe...@realityforge.org> wrote:

> Hi,
>
> I am in the process of converting a bunch of builds based on
> ant/maven/rake to buildr. One of the patterns that I can not see how
> to directly translate into buildr is the single project with multiple
> test types within it. i.e. A project has integration and unit tests
> both written in java and both stored in test hierarchy. It also has
> ruby based integration tests sitting side by side the java test code.
> AFAICS buildr is not really set up for this scenario. It seems that
> for each project there can only be one test invocation, one test
> framework and can not run parts at both integration testing time and
> unit testing time. Is this correct?
>
> Any suggestions on how to deal with this?
>
> --
> Cheers,
>
> Peter Donald
>

Re: Integration Tests + Unit Tests in the same project?

[Alex Boisvert <al...@gmail.com>]

Hi Peter,

In an ideal world, setting up multiple test types would be a simple as
writing,

type1 = TestTask.define_task('test-type1')
type1.include [...]

type2 = TestTask.define_task('test-type2')
type2.include [...]

...

test.enhance [type1, type2]

However as it stands, the code makes several assumptions that there's only
one TestTask per project.   It's certainly doable but may require
non-trivial changes to the code.

Rhett's suggestion to use multiple projects and rebase them with :base_dir
works to a certain extent but may not work entirely if using the same test
framework several times because the same file ends up being used to check if
tests have been run.

My suggestion if you want to get things working quickly is to separate out
the different test types into different physical (sub)projects.  I don't
think you need to set up a full directory structure (e.g.
$subproject/src/test/...), you could have it such that only $subproject is
needed by setting up test.compile.from to ".".

If you can provide an example (ideally with simple/fake sources) that
illustrates your directory structure and needs, I can help getting multiple
test types working in a single project.

alex

On Tue, Sep 28, 2010 at 8:20 PM, Rhett Sutphin <rh...@detailedbalance.net>wrote:

> Hi Peter,
>
> On Sep 28, 2010, at 8:16 PM, Peter Donald wrote:
>
> > Hi,
> >
> > I am in the process of converting a bunch of builds based on
> > ant/maven/rake to buildr. One of the patterns that I can not see how
> > to directly translate into buildr is the single project with multiple
> > test types within it. i.e. A project has integration and unit tests
> > both written in java and both stored in test hierarchy. It also has
> > ruby based integration tests sitting side by side the java test code.
> > AFAICS buildr is not really set up for this scenario. It seems that
> > for each project there can only be one test invocation, one test
> > framework and can not run parts at both integration testing time and
> > unit testing time. Is this correct?
>
> Yes.  Each buildr project can only have one test type (either integration
> or unit) and one test framework (junit, rspec, etc.).
>
> > Any suggestions on how to deal with this?
>
> I use a separate subproject for integration tests -- i.e., a subproject
> that only has tests in it and which depends on whatever module(s) I'm trying
> to integratedly test.  It might even be possible to have a subproject
> definition which, through the :base_dir option, executed tests which are
> physically alongside the unit tests in the parent project.  I haven't tried
> that, though.
>
> Rhett
>
> >
> > --
> > Cheers,
> >
> > Peter Donald
>
>

Re: Integration Tests + Unit Tests in the same project?

[Rhett Sutphin <rh...@detailedbalance.net>]

Hi Peter,

On Sep 28, 2010, at 8:16 PM, Peter Donald wrote:

> Hi,
> 
> I am in the process of converting a bunch of builds based on
> ant/maven/rake to buildr. One of the patterns that I can not see how
> to directly translate into buildr is the single project with multiple
> test types within it. i.e. A project has integration and unit tests
> both written in java and both stored in test hierarchy. It also has
> ruby based integration tests sitting side by side the java test code.
> AFAICS buildr is not really set up for this scenario. It seems that
> for each project there can only be one test invocation, one test
> framework and can not run parts at both integration testing time and
> unit testing time. Is this correct?

Yes.  Each buildr project can only have one test type (either integration or unit) and one test framework (junit, rspec, etc.).

> Any suggestions on how to deal with this?

I use a separate subproject for integration tests -- i.e., a subproject that only has tests in it and which depends on whatever module(s) I'm trying to integratedly test.  It might even be possible to have a subproject definition which, through the :base_dir option, executed tests which are physically alongside the unit tests in the parent project.  I haven't tried that, though.

Rhett

> 
> -- 
> Cheers,
> 
> Peter Donald