You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@buildr.apache.org by Peter Schröder <ps...@blau.de> on 2009/10/21 09:32:23 UTC

unpredictable behavior of test.using :integration

hi there,

we are experiencing somewhat strange behavior when using integration  
tests.

we have a setup where a project has n subprojects and only one of them  
has tests that should be executed as integration-tests.

we are using environment-variables to let buildr include some test- 
classes that are only executed on our system-test-servers.

this works very well: compiles, package artifacts, calls integration- 
setup, performs-tests, integration-teardown

on the other hand we have the same setup but with a single project  
without subprojects, where the integration setup is missing, so the  
system-tests fail without a system to test. in that project i have to  
call the integration-target directly to have the desired behavior.

the builds are quite complex so i wont include them here, but i played  
around a little bit with a dummy project (see below).

the things i dont understand are:

1. why are setup and teardown called for all projects even though they  
are never configured with :integration
2. why are all tests executed as :integration if you just configure it  
in the parent project
3. under wich circumstances will the integration-target get invoked


module SystemTests
   include Extension

   after_define do |project|
     info "after define for #{project}"
     #project.test.using :integration
     project.integration.setup do
       info "-" * 50 + " integeration setup for #{project}"
     end

     project.integration.teardown do
       info "-" * 50 + " integeration teardown for #{project}"
     end
   end

end

class Buildr::Project
   include SystemTests
end

repositories.remote << 'http://www.ibiblio.org/maven2/'

info "define all"
define 'all' do

   info "define subA"
   define 'subA' do
     info "subA goes here"
     #test.using :integration
   end

   info "define subB"
   define 'subB' do
     info "subB goes here"
     #test.using :junit
   end

   info "all goes here"

   test.using :integration

end

Re: unpredictable behavior of test.using :integration

Posted by Shane Witbeck <sh...@digitalsanctum.com>.
I've had some frustration in this area too. In order to run
integration tests you have to call from the top level project. I don't
believe you need the test.using :integration defined in the top level
project because integration tests get run automatically if called from
top level. I'm not 100% sure about this so someone correct me if I'm
wrong.

I've asked for a finer grained approach but was told to either
reorganize where my tests were located or create a project
specifically to contain integration tests.


-Shane



On Wed, Oct 21, 2009 at 3:32 AM, Peter Schröder <ps...@blau.de> wrote:
> hi there,
>
> we are experiencing somewhat strange behavior when using integration
> tests.
>
> we have a setup where a project has n subprojects and only one of them
> has tests that should be executed as integration-tests.
>
> we are using environment-variables to let buildr include some test-
> classes that are only executed on our system-test-servers.
>
> this works very well: compiles, package artifacts, calls integration-
> setup, performs-tests, integration-teardown
>
> on the other hand we have the same setup but with a single project
> without subprojects, where the integration setup is missing, so the
> system-tests fail without a system to test. in that project i have to
> call the integration-target directly to have the desired behavior.
>
> the builds are quite complex so i wont include them here, but i played
> around a little bit with a dummy project (see below).
>
> the things i dont understand are:
>
> 1. why are setup and teardown called for all projects even though they
> are never configured with :integration
> 2. why are all tests executed as :integration if you just configure it
> in the parent project
> 3. under wich circumstances will the integration-target get invoked
>
>
> module SystemTests
>   include Extension
>
>   after_define do |project|
>     info "after define for #{project}"
>     #project.test.using :integration
>     project.integration.setup do
>       info "-" * 50 + " integeration setup for #{project}"
>     end
>
>     project.integration.teardown do
>       info "-" * 50 + " integeration teardown for #{project}"
>     end
>   end
>
> end
>
> class Buildr::Project
>   include SystemTests
> end
>
> repositories.remote << 'http://www.ibiblio.org/maven2/'
>
> info "define all"
> define 'all' do
>
>   info "define subA"
>   define 'subA' do
>     info "subA goes here"
>     #test.using :integration
>   end
>
>   info "define subB"
>   define 'subB' do
>     info "subB goes here"
>     #test.using :junit
>   end
>
>   info "all goes here"
>
>   test.using :integration
>
> end
>

Re: helper for directories

Posted by Assaf Arkin <as...@labnotes.org>.
On Fri, Oct 23, 2009 at 1:02 AM, Peter Schröder <ps...@blau.de> wrote:

> hi there,
>
> i am looking for the way that buildr is creating directories on
> demand. i would like to use some magic like "if the directory is
> missing, just create it!" so that i dont have to create the
> directories myself.
>

Rake has a nice way of handling this using dependencies:

  directory "foo/bar"

  file "foo/bar/baz.h"=>"foo/bar" do
    # create baz.h, directory exists ...
  end

Personally, I use mkdir_p, which will create parent directories as
necessary, and not complain if directory already exists:

  file "foo/bar/baz.h" do
    mkdir_p "foo/bar"
    # create baz.h, directory exists ...
  end

Assaf



>
> kind regards,
> peter
>

AW: helper for directories

Posted by Klaas Prause <kp...@blau.de>.
As far as I have seen, buildr uses FileUtils for this stuff, so there is no magic think as far as I know. For all custom tasks I have used FileUtils to create missing dirs recursively.

Regards Klaas

-----Ursprüngliche Nachricht-----
Von: Peter Schröder [mailto:ps@blau.de] 
Gesendet: Freitag, 23. Oktober 2009 10:02
An: users@buildr.apache.org
Betreff: helper for directories

hi there,

i am looking for the way that buildr is creating directories on  
demand. i would like to use some magic like "if the directory is  
missing, just create it!" so that i dont have to create the  
directories myself.

kind regards,
peter

helper for directories

Posted by Peter Schröder <ps...@blau.de>.
hi there,

i am looking for the way that buildr is creating directories on  
demand. i would like to use some magic like "if the directory is  
missing, just create it!" so that i dont have to create the  
directories myself.

kind regards,
peter

Re: unpredictable behavior of test.using :integration

Posted by Peter Schröder <ps...@blau.de>.
ok, i'll have a look into the spec.

my biggest problem is, that buildr is not my everyday work. i just use  
it from time to time to move a project from ant to buildr... so it is  
kind of hard to digg really deep into the internals of buildr every  
time.

Am 23.10.2009 um 11:27 schrieb Alex Boisvert:

> Just in passing... if the documentation doesn't provide enough  
> information,
> the second place to look are the specs.  The specs are our 'reference
> documentation' in the sense that they specify actual expected runtime
> behavior.
>
> And this being said, I'll second you on the need for more  
> documentation on
> integration testing.
>
> alex
>
> On Fri, Oct 23, 2009 at 12:59 AM, Peter Schröder <ps...@blau.de> wrote:
>
>> the documentation on this topic is very limited...
>>
>> my expectations were the following:
>>
>> 1. integration setup and teardown are only executed if i explicitly
>> use :integration
>> 2. i can configure it for each subproject
>> 3. i am able to configure it from the outside via a custom task
>>
>> i dont think that it is neccessary to support all my needs, but i
>> think a clear instruction what you can do and what you cant would  
>> help!
>>
>> kind Regards,
>> Peter
>>
>> Am 21.10.2009 um 20:40 schrieb Assaf Arkin:
>>
>>> On Wed, Oct 21, 2009 at 2:10 AM, Peter Schröder <ps...@blau.de> wrote:
>>>
>>>> another thing i noticed, is that calling
>>>>
>>>> project.test.using :integration in a post-define task
>>>>
>>>> seems to work differntly form calling it directly in the project  
>>>> via
>>>>
>>>> test.using :integration
>>>>
>>>> any ideas?
>>>>
>>>
>>> Flipping integration bit on simply changes when tests are run: after
>>> packaging (integration), instead of before packaging (regular). All
>>> other
>>> behavior is the same (setup, teardown, frameworks, etc).
>>>
>>> Test options are inherited, options you define in a parent project  
>>> are
>>> inherited by its sub-project, and that should include the  
>>> integration
>>> option.
>>>
>>> Changing the order in which tests are run involving setting up task
>>> dependencies, which happens at the very end of the project
>>> definition, after
>>> evaluating the block. If you then (post-define) change the setting,
>>> nothing
>>> much happens, as task are already wired to run in non-integration
>>> order.
>>>
>>> Assaf
>>>
>>>
>>>>
>>>> Am 21.10.2009 um 09:32 schrieb Peter Schröder:
>>>>
>>>>> hi there,
>>>>>
>>>>> we are experiencing somewhat strange behavior when using  
>>>>> integration
>>>>> tests.
>>>>>
>>>>> we have a setup where a project has n subprojects and only one of
>>>>> them
>>>>> has tests that should be executed as integration-tests.
>>>>>
>>>>> we are using environment-variables to let buildr include some  
>>>>> test-
>>>>> classes that are only executed on our system-test-servers.
>>>>>
>>>>> this works very well: compiles, package artifacts, calls
>>>>> integration-
>>>>> setup, performs-tests, integration-teardown
>>>>>
>>>>> on the other hand we have the same setup but with a single project
>>>>> without subprojects, where the integration setup is missing, so  
>>>>> the
>>>>> system-tests fail without a system to test. in that project i have
>>>>> to
>>>>> call the integration-target directly to have the desired behavior.
>>>>>
>>>>> the builds are quite complex so i wont include them here, but i
>>>>> played
>>>>> around a little bit with a dummy project (see below).
>>>>>
>>>>> the things i dont understand are:
>>>>>
>>>>> 1. why are setup and teardown called for all projects even though
>>>>> they
>>>>> are never configured with :integration
>>>>> 2. why are all tests executed as :integration if you just
>>>>> configure it
>>>>> in the parent project
>>>>> 3. under wich circumstances will the integration-target get  
>>>>> invoked
>>>>>
>>>>>
>>>>> module SystemTests
>>>>> include Extension
>>>>>
>>>>> after_define do |project|
>>>>>   info "after define for #{project}"
>>>>>   #project.test.using :integration
>>>>>   project.integration.setup do
>>>>>     info "-" * 50 + " integeration setup for #{project}"
>>>>>   end
>>>>>
>>>>>   project.integration.teardown do
>>>>>     info "-" * 50 + " integeration teardown for #{project}"
>>>>>   end
>>>>> end
>>>>>
>>>>> end
>>>>>
>>>>> class Buildr::Project
>>>>> include SystemTests
>>>>> end
>>>>>
>>>>> repositories.remote << 'http://www.ibiblio.org/maven2/'
>>>>>
>>>>> info "define all"
>>>>> define 'all' do
>>>>>
>>>>> info "define subA"
>>>>> define 'subA' do
>>>>>   info "subA goes here"
>>>>>   #test.using :integration
>>>>> end
>>>>>
>>>>> info "define subB"
>>>>> define 'subB' do
>>>>>   info "subB goes here"
>>>>>   #test.using :junit
>>>>> end
>>>>>
>>>>> info "all goes here"
>>>>>
>>>>> test.using :integration
>>>>>
>>>>> end
>>>>
>>>>
>>
>>


Re: unpredictable behavior of test.using :integration

Posted by Alex Boisvert <al...@gmail.com>.
Just in passing... if the documentation doesn't provide enough information,
the second place to look are the specs.  The specs are our 'reference
documentation' in the sense that they specify actual expected runtime
behavior.

And this being said, I'll second you on the need for more documentation on
integration testing.

alex

On Fri, Oct 23, 2009 at 12:59 AM, Peter Schröder <ps...@blau.de> wrote:

> the documentation on this topic is very limited...
>
> my expectations were the following:
>
> 1. integration setup and teardown are only executed if i explicitly
> use :integration
> 2. i can configure it for each subproject
> 3. i am able to configure it from the outside via a custom task
>
> i dont think that it is neccessary to support all my needs, but i
> think a clear instruction what you can do and what you cant would help!
>
> kind Regards,
> Peter
>
> Am 21.10.2009 um 20:40 schrieb Assaf Arkin:
>
> > On Wed, Oct 21, 2009 at 2:10 AM, Peter Schröder <ps...@blau.de> wrote:
> >
> >> another thing i noticed, is that calling
> >>
> >> project.test.using :integration in a post-define task
> >>
> >> seems to work differntly form calling it directly in the project via
> >>
> >> test.using :integration
> >>
> >> any ideas?
> >>
> >
> > Flipping integration bit on simply changes when tests are run: after
> > packaging (integration), instead of before packaging (regular). All
> > other
> > behavior is the same (setup, teardown, frameworks, etc).
> >
> > Test options are inherited, options you define in a parent project are
> > inherited by its sub-project, and that should include the integration
> > option.
> >
> > Changing the order in which tests are run involving setting up task
> > dependencies, which happens at the very end of the project
> > definition, after
> > evaluating the block. If you then (post-define) change the setting,
> > nothing
> > much happens, as task are already wired to run in non-integration
> > order.
> >
> > Assaf
> >
> >
> >>
> >> Am 21.10.2009 um 09:32 schrieb Peter Schröder:
> >>
> >>> hi there,
> >>>
> >>> we are experiencing somewhat strange behavior when using integration
> >>> tests.
> >>>
> >>> we have a setup where a project has n subprojects and only one of
> >>> them
> >>> has tests that should be executed as integration-tests.
> >>>
> >>> we are using environment-variables to let buildr include some test-
> >>> classes that are only executed on our system-test-servers.
> >>>
> >>> this works very well: compiles, package artifacts, calls
> >>> integration-
> >>> setup, performs-tests, integration-teardown
> >>>
> >>> on the other hand we have the same setup but with a single project
> >>> without subprojects, where the integration setup is missing, so the
> >>> system-tests fail without a system to test. in that project i have
> >>> to
> >>> call the integration-target directly to have the desired behavior.
> >>>
> >>> the builds are quite complex so i wont include them here, but i
> >>> played
> >>> around a little bit with a dummy project (see below).
> >>>
> >>> the things i dont understand are:
> >>>
> >>> 1. why are setup and teardown called for all projects even though
> >>> they
> >>> are never configured with :integration
> >>> 2. why are all tests executed as :integration if you just
> >>> configure it
> >>> in the parent project
> >>> 3. under wich circumstances will the integration-target get invoked
> >>>
> >>>
> >>> module SystemTests
> >>>  include Extension
> >>>
> >>>  after_define do |project|
> >>>    info "after define for #{project}"
> >>>    #project.test.using :integration
> >>>    project.integration.setup do
> >>>      info "-" * 50 + " integeration setup for #{project}"
> >>>    end
> >>>
> >>>    project.integration.teardown do
> >>>      info "-" * 50 + " integeration teardown for #{project}"
> >>>    end
> >>>  end
> >>>
> >>> end
> >>>
> >>> class Buildr::Project
> >>>  include SystemTests
> >>> end
> >>>
> >>> repositories.remote << 'http://www.ibiblio.org/maven2/'
> >>>
> >>> info "define all"
> >>> define 'all' do
> >>>
> >>>  info "define subA"
> >>>  define 'subA' do
> >>>    info "subA goes here"
> >>>    #test.using :integration
> >>>  end
> >>>
> >>>  info "define subB"
> >>>  define 'subB' do
> >>>    info "subB goes here"
> >>>    #test.using :junit
> >>>  end
> >>>
> >>>  info "all goes here"
> >>>
> >>>  test.using :integration
> >>>
> >>> end
> >>
> >>
>
>

Re: unpredictable behavior of test.using :integration

Posted by Peter Schröder <ps...@blau.de>.
the documentation on this topic is very limited...

my expectations were the following:

1. integration setup and teardown are only executed if i explicitly  
use :integration
2. i can configure it for each subproject
3. i am able to configure it from the outside via a custom task

i dont think that it is neccessary to support all my needs, but i  
think a clear instruction what you can do and what you cant would help!

kind Regards,
Peter

Am 21.10.2009 um 20:40 schrieb Assaf Arkin:

> On Wed, Oct 21, 2009 at 2:10 AM, Peter Schröder <ps...@blau.de> wrote:
>
>> another thing i noticed, is that calling
>>
>> project.test.using :integration in a post-define task
>>
>> seems to work differntly form calling it directly in the project via
>>
>> test.using :integration
>>
>> any ideas?
>>
>
> Flipping integration bit on simply changes when tests are run: after
> packaging (integration), instead of before packaging (regular). All  
> other
> behavior is the same (setup, teardown, frameworks, etc).
>
> Test options are inherited, options you define in a parent project are
> inherited by its sub-project, and that should include the integration
> option.
>
> Changing the order in which tests are run involving setting up task
> dependencies, which happens at the very end of the project  
> definition, after
> evaluating the block. If you then (post-define) change the setting,  
> nothing
> much happens, as task are already wired to run in non-integration  
> order.
>
> Assaf
>
>
>>
>> Am 21.10.2009 um 09:32 schrieb Peter Schröder:
>>
>>> hi there,
>>>
>>> we are experiencing somewhat strange behavior when using integration
>>> tests.
>>>
>>> we have a setup where a project has n subprojects and only one of  
>>> them
>>> has tests that should be executed as integration-tests.
>>>
>>> we are using environment-variables to let buildr include some test-
>>> classes that are only executed on our system-test-servers.
>>>
>>> this works very well: compiles, package artifacts, calls  
>>> integration-
>>> setup, performs-tests, integration-teardown
>>>
>>> on the other hand we have the same setup but with a single project
>>> without subprojects, where the integration setup is missing, so the
>>> system-tests fail without a system to test. in that project i have  
>>> to
>>> call the integration-target directly to have the desired behavior.
>>>
>>> the builds are quite complex so i wont include them here, but i  
>>> played
>>> around a little bit with a dummy project (see below).
>>>
>>> the things i dont understand are:
>>>
>>> 1. why are setup and teardown called for all projects even though  
>>> they
>>> are never configured with :integration
>>> 2. why are all tests executed as :integration if you just  
>>> configure it
>>> in the parent project
>>> 3. under wich circumstances will the integration-target get invoked
>>>
>>>
>>> module SystemTests
>>>  include Extension
>>>
>>>  after_define do |project|
>>>    info "after define for #{project}"
>>>    #project.test.using :integration
>>>    project.integration.setup do
>>>      info "-" * 50 + " integeration setup for #{project}"
>>>    end
>>>
>>>    project.integration.teardown do
>>>      info "-" * 50 + " integeration teardown for #{project}"
>>>    end
>>>  end
>>>
>>> end
>>>
>>> class Buildr::Project
>>>  include SystemTests
>>> end
>>>
>>> repositories.remote << 'http://www.ibiblio.org/maven2/'
>>>
>>> info "define all"
>>> define 'all' do
>>>
>>>  info "define subA"
>>>  define 'subA' do
>>>    info "subA goes here"
>>>    #test.using :integration
>>>  end
>>>
>>>  info "define subB"
>>>  define 'subB' do
>>>    info "subB goes here"
>>>    #test.using :junit
>>>  end
>>>
>>>  info "all goes here"
>>>
>>>  test.using :integration
>>>
>>> end
>>
>>


Re: unpredictable behavior of test.using :integration

Posted by Assaf Arkin <as...@labnotes.org>.
On Wed, Oct 21, 2009 at 2:10 AM, Peter Schröder <ps...@blau.de> wrote:

> another thing i noticed, is that calling
>
> project.test.using :integration in a post-define task
>
> seems to work differntly form calling it directly in the project via
>
> test.using :integration
>
> any ideas?
>

Flipping integration bit on simply changes when tests are run: after
packaging (integration), instead of before packaging (regular). All other
behavior is the same (setup, teardown, frameworks, etc).

Test options are inherited, options you define in a parent project are
inherited by its sub-project, and that should include the integration
option.

Changing the order in which tests are run involving setting up task
dependencies, which happens at the very end of the project definition, after
evaluating the block. If you then (post-define) change the setting, nothing
much happens, as task are already wired to run in non-integration order.

Assaf


>
> Am 21.10.2009 um 09:32 schrieb Peter Schröder:
>
> > hi there,
> >
> > we are experiencing somewhat strange behavior when using integration
> > tests.
> >
> > we have a setup where a project has n subprojects and only one of them
> > has tests that should be executed as integration-tests.
> >
> > we are using environment-variables to let buildr include some test-
> > classes that are only executed on our system-test-servers.
> >
> > this works very well: compiles, package artifacts, calls integration-
> > setup, performs-tests, integration-teardown
> >
> > on the other hand we have the same setup but with a single project
> > without subprojects, where the integration setup is missing, so the
> > system-tests fail without a system to test. in that project i have to
> > call the integration-target directly to have the desired behavior.
> >
> > the builds are quite complex so i wont include them here, but i played
> > around a little bit with a dummy project (see below).
> >
> > the things i dont understand are:
> >
> > 1. why are setup and teardown called for all projects even though they
> > are never configured with :integration
> > 2. why are all tests executed as :integration if you just configure it
> > in the parent project
> > 3. under wich circumstances will the integration-target get invoked
> >
> >
> > module SystemTests
> >   include Extension
> >
> >   after_define do |project|
> >     info "after define for #{project}"
> >     #project.test.using :integration
> >     project.integration.setup do
> >       info "-" * 50 + " integeration setup for #{project}"
> >     end
> >
> >     project.integration.teardown do
> >       info "-" * 50 + " integeration teardown for #{project}"
> >     end
> >   end
> >
> > end
> >
> > class Buildr::Project
> >   include SystemTests
> > end
> >
> > repositories.remote << 'http://www.ibiblio.org/maven2/'
> >
> > info "define all"
> > define 'all' do
> >
> >   info "define subA"
> >   define 'subA' do
> >     info "subA goes here"
> >     #test.using :integration
> >   end
> >
> >   info "define subB"
> >   define 'subB' do
> >     info "subB goes here"
> >     #test.using :junit
> >   end
> >
> >   info "all goes here"
> >
> >   test.using :integration
> >
> > end
>
>

Re: unpredictable behavior of test.using :integration

Posted by Peter Schröder <ps...@blau.de>.
another thing i noticed, is that calling

project.test.using :integration in a post-define task

seems to work differntly form calling it directly in the project via

test.using :integration

any ideas?

Am 21.10.2009 um 09:32 schrieb Peter Schröder:

> hi there,
>
> we are experiencing somewhat strange behavior when using integration
> tests.
>
> we have a setup where a project has n subprojects and only one of them
> has tests that should be executed as integration-tests.
>
> we are using environment-variables to let buildr include some test-
> classes that are only executed on our system-test-servers.
>
> this works very well: compiles, package artifacts, calls integration-
> setup, performs-tests, integration-teardown
>
> on the other hand we have the same setup but with a single project
> without subprojects, where the integration setup is missing, so the
> system-tests fail without a system to test. in that project i have to
> call the integration-target directly to have the desired behavior.
>
> the builds are quite complex so i wont include them here, but i played
> around a little bit with a dummy project (see below).
>
> the things i dont understand are:
>
> 1. why are setup and teardown called for all projects even though they
> are never configured with :integration
> 2. why are all tests executed as :integration if you just configure it
> in the parent project
> 3. under wich circumstances will the integration-target get invoked
>
>
> module SystemTests
>   include Extension
>
>   after_define do |project|
>     info "after define for #{project}"
>     #project.test.using :integration
>     project.integration.setup do
>       info "-" * 50 + " integeration setup for #{project}"
>     end
>
>     project.integration.teardown do
>       info "-" * 50 + " integeration teardown for #{project}"
>     end
>   end
>
> end
>
> class Buildr::Project
>   include SystemTests
> end
>
> repositories.remote << 'http://www.ibiblio.org/maven2/'
>
> info "define all"
> define 'all' do
>
>   info "define subA"
>   define 'subA' do
>     info "subA goes here"
>     #test.using :integration
>   end
>
>   info "define subB"
>   define 'subB' do
>     info "subB goes here"
>     #test.using :junit
>   end
>
>   info "all goes here"
>
>   test.using :integration
>
> end


error executing tests with no default test-folder

Posted by Peter Schröder <ps...@blau.de>.
hi,

i have a subproject that holds only system-tests. for that reason it  
contains only of one class in one folder src/test-system/java .

i add this folder using test.compile.from _("src/test-system/java")

wich results in a Class not found exception:

Compiling chief:systemtest:test into /home/phoet/workspace/chief/ 
systemtest/target/test/classes
NativeException: java.lang.ClassNotFoundException:  
de.blau.chief.ws.validation.v1_0.ValidationWebServiceImplSystemTest
Buildr aborted!
java.lang.ClassNotFoundException:  
de.blau.chief.ws.validation.v1_0.ValidationWebServiceImplSystemTest

if i add a default folder src/test/java with an empty class in it  
everything is fine!

this seems kind of strange to me...

kind regards
peter