You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@buildr.apache.org by Peter Schröder <ps...@blau.de> on 2009/10/21 09:32:23 UTC
unpredictable behavior of test.using :integration
hi there,
we are experiencing somewhat strange behavior when using integration
tests.
we have a setup where a project has n subprojects and only one of them
has tests that should be executed as integration-tests.
we are using environment-variables to let buildr include some test-
classes that are only executed on our system-test-servers.
this works very well: compiles, package artifacts, calls integration-
setup, performs-tests, integration-teardown
on the other hand we have the same setup but with a single project
without subprojects, where the integration setup is missing, so the
system-tests fail without a system to test. in that project i have to
call the integration-target directly to have the desired behavior.
the builds are quite complex so i wont include them here, but i played
around a little bit with a dummy project (see below).
the things i dont understand are:
1. why are setup and teardown called for all projects even though they
are never configured with :integration
2. why are all tests executed as :integration if you just configure it
in the parent project
3. under wich circumstances will the integration-target get invoked
module SystemTests
include Extension
after_define do |project|
info "after define for #{project}"
#project.test.using :integration
project.integration.setup do
info "-" * 50 + " integeration setup for #{project}"
end
project.integration.teardown do
info "-" * 50 + " integeration teardown for #{project}"
end
end
end
class Buildr::Project
include SystemTests
end
repositories.remote << 'http://www.ibiblio.org/maven2/'
info "define all"
define 'all' do
info "define subA"
define 'subA' do
info "subA goes here"
#test.using :integration
end
info "define subB"
define 'subB' do
info "subB goes here"
#test.using :junit
end
info "all goes here"
test.using :integration
end
Re: unpredictable behavior of test.using :integration
Posted by Shane Witbeck <sh...@digitalsanctum.com>.
I've had some frustration in this area too. In order to run
integration tests you have to call from the top level project. I don't
believe you need the test.using :integration defined in the top level
project because integration tests get run automatically if called from
top level. I'm not 100% sure about this so someone correct me if I'm
wrong.
I've asked for a finer grained approach but was told to either
reorganize where my tests were located or create a project
specifically to contain integration tests.
-Shane
On Wed, Oct 21, 2009 at 3:32 AM, Peter Schröder <ps...@blau.de> wrote:
> hi there,
>
> we are experiencing somewhat strange behavior when using integration
> tests.
>
> we have a setup where a project has n subprojects and only one of them
> has tests that should be executed as integration-tests.
>
> we are using environment-variables to let buildr include some test-
> classes that are only executed on our system-test-servers.
>
> this works very well: compiles, package artifacts, calls integration-
> setup, performs-tests, integration-teardown
>
> on the other hand we have the same setup but with a single project
> without subprojects, where the integration setup is missing, so the
> system-tests fail without a system to test. in that project i have to
> call the integration-target directly to have the desired behavior.
>
> the builds are quite complex so i wont include them here, but i played
> around a little bit with a dummy project (see below).
>
> the things i dont understand are:
>
> 1. why are setup and teardown called for all projects even though they
> are never configured with :integration
> 2. why are all tests executed as :integration if you just configure it
> in the parent project
> 3. under wich circumstances will the integration-target get invoked
>
>
> module SystemTests
> include Extension
>
> after_define do |project|
> info "after define for #{project}"
> #project.test.using :integration
> project.integration.setup do
> info "-" * 50 + " integeration setup for #{project}"
> end
>
> project.integration.teardown do
> info "-" * 50 + " integeration teardown for #{project}"
> end
> end
>
> end
>
> class Buildr::Project
> include SystemTests
> end
>
> repositories.remote << 'http://www.ibiblio.org/maven2/'
>
> info "define all"
> define 'all' do
>
> info "define subA"
> define 'subA' do
> info "subA goes here"
> #test.using :integration
> end
>
> info "define subB"
> define 'subB' do
> info "subB goes here"
> #test.using :junit
> end
>
> info "all goes here"
>
> test.using :integration
>
> end
>
Re: helper for directories
Posted by Assaf Arkin <as...@labnotes.org>.
On Fri, Oct 23, 2009 at 1:02 AM, Peter Schröder <ps...@blau.de> wrote:
> hi there,
>
> i am looking for the way that buildr is creating directories on
> demand. i would like to use some magic like "if the directory is
> missing, just create it!" so that i dont have to create the
> directories myself.
>
Rake has a nice way of handling this using dependencies:
directory "foo/bar"
file "foo/bar/baz.h"=>"foo/bar" do
# create baz.h, directory exists ...
end
Personally, I use mkdir_p, which will create parent directories as
necessary, and not complain if directory already exists:
file "foo/bar/baz.h" do
mkdir_p "foo/bar"
# create baz.h, directory exists ...
end
Assaf
>
> kind regards,
> peter
>
AW: helper for directories
Posted by Klaas Prause <kp...@blau.de>.
As far as I have seen, buildr uses FileUtils for this stuff, so there is no magic think as far as I know. For all custom tasks I have used FileUtils to create missing dirs recursively.
Regards Klaas
-----Ursprüngliche Nachricht-----
Von: Peter Schröder [mailto:ps@blau.de]
Gesendet: Freitag, 23. Oktober 2009 10:02
An: users@buildr.apache.org
Betreff: helper for directories
hi there,
i am looking for the way that buildr is creating directories on
demand. i would like to use some magic like "if the directory is
missing, just create it!" so that i dont have to create the
directories myself.
kind regards,
peter
helper for directories
Posted by Peter Schröder <ps...@blau.de>.
hi there,
i am looking for the way that buildr is creating directories on
demand. i would like to use some magic like "if the directory is
missing, just create it!" so that i dont have to create the
directories myself.
kind regards,
peter
Re: unpredictable behavior of test.using :integration
Posted by Peter Schröder <ps...@blau.de>.
ok, i'll have a look into the spec.
my biggest problem is, that buildr is not my everyday work. i just use
it from time to time to move a project from ant to buildr... so it is
kind of hard to digg really deep into the internals of buildr every
time.
Am 23.10.2009 um 11:27 schrieb Alex Boisvert:
> Just in passing... if the documentation doesn't provide enough
> information,
> the second place to look are the specs. The specs are our 'reference
> documentation' in the sense that they specify actual expected runtime
> behavior.
>
> And this being said, I'll second you on the need for more
> documentation on
> integration testing.
>
> alex
>
> On Fri, Oct 23, 2009 at 12:59 AM, Peter Schröder <ps...@blau.de> wrote:
>
>> the documentation on this topic is very limited...
>>
>> my expectations were the following:
>>
>> 1. integration setup and teardown are only executed if i explicitly
>> use :integration
>> 2. i can configure it for each subproject
>> 3. i am able to configure it from the outside via a custom task
>>
>> i dont think that it is neccessary to support all my needs, but i
>> think a clear instruction what you can do and what you cant would
>> help!
>>
>> kind Regards,
>> Peter
>>
>> Am 21.10.2009 um 20:40 schrieb Assaf Arkin:
>>
>>> On Wed, Oct 21, 2009 at 2:10 AM, Peter Schröder <ps...@blau.de> wrote:
>>>
>>>> another thing i noticed, is that calling
>>>>
>>>> project.test.using :integration in a post-define task
>>>>
>>>> seems to work differntly form calling it directly in the project
>>>> via
>>>>
>>>> test.using :integration
>>>>
>>>> any ideas?
>>>>
>>>
>>> Flipping integration bit on simply changes when tests are run: after
>>> packaging (integration), instead of before packaging (regular). All
>>> other
>>> behavior is the same (setup, teardown, frameworks, etc).
>>>
>>> Test options are inherited, options you define in a parent project
>>> are
>>> inherited by its sub-project, and that should include the
>>> integration
>>> option.
>>>
>>> Changing the order in which tests are run involving setting up task
>>> dependencies, which happens at the very end of the project
>>> definition, after
>>> evaluating the block. If you then (post-define) change the setting,
>>> nothing
>>> much happens, as task are already wired to run in non-integration
>>> order.
>>>
>>> Assaf
>>>
>>>
>>>>
>>>> Am 21.10.2009 um 09:32 schrieb Peter Schröder:
>>>>
>>>>> hi there,
>>>>>
>>>>> we are experiencing somewhat strange behavior when using
>>>>> integration
>>>>> tests.
>>>>>
>>>>> we have a setup where a project has n subprojects and only one of
>>>>> them
>>>>> has tests that should be executed as integration-tests.
>>>>>
>>>>> we are using environment-variables to let buildr include some
>>>>> test-
>>>>> classes that are only executed on our system-test-servers.
>>>>>
>>>>> this works very well: compiles, package artifacts, calls
>>>>> integration-
>>>>> setup, performs-tests, integration-teardown
>>>>>
>>>>> on the other hand we have the same setup but with a single project
>>>>> without subprojects, where the integration setup is missing, so
>>>>> the
>>>>> system-tests fail without a system to test. in that project i have
>>>>> to
>>>>> call the integration-target directly to have the desired behavior.
>>>>>
>>>>> the builds are quite complex so i wont include them here, but i
>>>>> played
>>>>> around a little bit with a dummy project (see below).
>>>>>
>>>>> the things i dont understand are:
>>>>>
>>>>> 1. why are setup and teardown called for all projects even though
>>>>> they
>>>>> are never configured with :integration
>>>>> 2. why are all tests executed as :integration if you just
>>>>> configure it
>>>>> in the parent project
>>>>> 3. under wich circumstances will the integration-target get
>>>>> invoked
>>>>>
>>>>>
>>>>> module SystemTests
>>>>> include Extension
>>>>>
>>>>> after_define do |project|
>>>>> info "after define for #{project}"
>>>>> #project.test.using :integration
>>>>> project.integration.setup do
>>>>> info "-" * 50 + " integeration setup for #{project}"
>>>>> end
>>>>>
>>>>> project.integration.teardown do
>>>>> info "-" * 50 + " integeration teardown for #{project}"
>>>>> end
>>>>> end
>>>>>
>>>>> end
>>>>>
>>>>> class Buildr::Project
>>>>> include SystemTests
>>>>> end
>>>>>
>>>>> repositories.remote << 'http://www.ibiblio.org/maven2/'
>>>>>
>>>>> info "define all"
>>>>> define 'all' do
>>>>>
>>>>> info "define subA"
>>>>> define 'subA' do
>>>>> info "subA goes here"
>>>>> #test.using :integration
>>>>> end
>>>>>
>>>>> info "define subB"
>>>>> define 'subB' do
>>>>> info "subB goes here"
>>>>> #test.using :junit
>>>>> end
>>>>>
>>>>> info "all goes here"
>>>>>
>>>>> test.using :integration
>>>>>
>>>>> end
>>>>
>>>>
>>
>>
Re: unpredictable behavior of test.using :integration
Posted by Alex Boisvert <al...@gmail.com>.
Just in passing... if the documentation doesn't provide enough information,
the second place to look are the specs. The specs are our 'reference
documentation' in the sense that they specify actual expected runtime
behavior.
And this being said, I'll second you on the need for more documentation on
integration testing.
alex
On Fri, Oct 23, 2009 at 12:59 AM, Peter Schröder <ps...@blau.de> wrote:
> the documentation on this topic is very limited...
>
> my expectations were the following:
>
> 1. integration setup and teardown are only executed if i explicitly
> use :integration
> 2. i can configure it for each subproject
> 3. i am able to configure it from the outside via a custom task
>
> i dont think that it is neccessary to support all my needs, but i
> think a clear instruction what you can do and what you cant would help!
>
> kind Regards,
> Peter
>
> Am 21.10.2009 um 20:40 schrieb Assaf Arkin:
>
> > On Wed, Oct 21, 2009 at 2:10 AM, Peter Schröder <ps...@blau.de> wrote:
> >
> >> another thing i noticed, is that calling
> >>
> >> project.test.using :integration in a post-define task
> >>
> >> seems to work differntly form calling it directly in the project via
> >>
> >> test.using :integration
> >>
> >> any ideas?
> >>
> >
> > Flipping integration bit on simply changes when tests are run: after
> > packaging (integration), instead of before packaging (regular). All
> > other
> > behavior is the same (setup, teardown, frameworks, etc).
> >
> > Test options are inherited, options you define in a parent project are
> > inherited by its sub-project, and that should include the integration
> > option.
> >
> > Changing the order in which tests are run involving setting up task
> > dependencies, which happens at the very end of the project
> > definition, after
> > evaluating the block. If you then (post-define) change the setting,
> > nothing
> > much happens, as task are already wired to run in non-integration
> > order.
> >
> > Assaf
> >
> >
> >>
> >> Am 21.10.2009 um 09:32 schrieb Peter Schröder:
> >>
> >>> hi there,
> >>>
> >>> we are experiencing somewhat strange behavior when using integration
> >>> tests.
> >>>
> >>> we have a setup where a project has n subprojects and only one of
> >>> them
> >>> has tests that should be executed as integration-tests.
> >>>
> >>> we are using environment-variables to let buildr include some test-
> >>> classes that are only executed on our system-test-servers.
> >>>
> >>> this works very well: compiles, package artifacts, calls
> >>> integration-
> >>> setup, performs-tests, integration-teardown
> >>>
> >>> on the other hand we have the same setup but with a single project
> >>> without subprojects, where the integration setup is missing, so the
> >>> system-tests fail without a system to test. in that project i have
> >>> to
> >>> call the integration-target directly to have the desired behavior.
> >>>
> >>> the builds are quite complex so i wont include them here, but i
> >>> played
> >>> around a little bit with a dummy project (see below).
> >>>
> >>> the things i dont understand are:
> >>>
> >>> 1. why are setup and teardown called for all projects even though
> >>> they
> >>> are never configured with :integration
> >>> 2. why are all tests executed as :integration if you just
> >>> configure it
> >>> in the parent project
> >>> 3. under wich circumstances will the integration-target get invoked
> >>>
> >>>
> >>> module SystemTests
> >>> include Extension
> >>>
> >>> after_define do |project|
> >>> info "after define for #{project}"
> >>> #project.test.using :integration
> >>> project.integration.setup do
> >>> info "-" * 50 + " integeration setup for #{project}"
> >>> end
> >>>
> >>> project.integration.teardown do
> >>> info "-" * 50 + " integeration teardown for #{project}"
> >>> end
> >>> end
> >>>
> >>> end
> >>>
> >>> class Buildr::Project
> >>> include SystemTests
> >>> end
> >>>
> >>> repositories.remote << 'http://www.ibiblio.org/maven2/'
> >>>
> >>> info "define all"
> >>> define 'all' do
> >>>
> >>> info "define subA"
> >>> define 'subA' do
> >>> info "subA goes here"
> >>> #test.using :integration
> >>> end
> >>>
> >>> info "define subB"
> >>> define 'subB' do
> >>> info "subB goes here"
> >>> #test.using :junit
> >>> end
> >>>
> >>> info "all goes here"
> >>>
> >>> test.using :integration
> >>>
> >>> end
> >>
> >>
>
>
Re: unpredictable behavior of test.using :integration
Posted by Peter Schröder <ps...@blau.de>.
the documentation on this topic is very limited...
my expectations were the following:
1. integration setup and teardown are only executed if i explicitly
use :integration
2. i can configure it for each subproject
3. i am able to configure it from the outside via a custom task
i dont think that it is neccessary to support all my needs, but i
think a clear instruction what you can do and what you cant would help!
kind Regards,
Peter
Am 21.10.2009 um 20:40 schrieb Assaf Arkin:
> On Wed, Oct 21, 2009 at 2:10 AM, Peter Schröder <ps...@blau.de> wrote:
>
>> another thing i noticed, is that calling
>>
>> project.test.using :integration in a post-define task
>>
>> seems to work differntly form calling it directly in the project via
>>
>> test.using :integration
>>
>> any ideas?
>>
>
> Flipping integration bit on simply changes when tests are run: after
> packaging (integration), instead of before packaging (regular). All
> other
> behavior is the same (setup, teardown, frameworks, etc).
>
> Test options are inherited, options you define in a parent project are
> inherited by its sub-project, and that should include the integration
> option.
>
> Changing the order in which tests are run involving setting up task
> dependencies, which happens at the very end of the project
> definition, after
> evaluating the block. If you then (post-define) change the setting,
> nothing
> much happens, as task are already wired to run in non-integration
> order.
>
> Assaf
>
>
>>
>> Am 21.10.2009 um 09:32 schrieb Peter Schröder:
>>
>>> hi there,
>>>
>>> we are experiencing somewhat strange behavior when using integration
>>> tests.
>>>
>>> we have a setup where a project has n subprojects and only one of
>>> them
>>> has tests that should be executed as integration-tests.
>>>
>>> we are using environment-variables to let buildr include some test-
>>> classes that are only executed on our system-test-servers.
>>>
>>> this works very well: compiles, package artifacts, calls
>>> integration-
>>> setup, performs-tests, integration-teardown
>>>
>>> on the other hand we have the same setup but with a single project
>>> without subprojects, where the integration setup is missing, so the
>>> system-tests fail without a system to test. in that project i have
>>> to
>>> call the integration-target directly to have the desired behavior.
>>>
>>> the builds are quite complex so i wont include them here, but i
>>> played
>>> around a little bit with a dummy project (see below).
>>>
>>> the things i dont understand are:
>>>
>>> 1. why are setup and teardown called for all projects even though
>>> they
>>> are never configured with :integration
>>> 2. why are all tests executed as :integration if you just
>>> configure it
>>> in the parent project
>>> 3. under wich circumstances will the integration-target get invoked
>>>
>>>
>>> module SystemTests
>>> include Extension
>>>
>>> after_define do |project|
>>> info "after define for #{project}"
>>> #project.test.using :integration
>>> project.integration.setup do
>>> info "-" * 50 + " integeration setup for #{project}"
>>> end
>>>
>>> project.integration.teardown do
>>> info "-" * 50 + " integeration teardown for #{project}"
>>> end
>>> end
>>>
>>> end
>>>
>>> class Buildr::Project
>>> include SystemTests
>>> end
>>>
>>> repositories.remote << 'http://www.ibiblio.org/maven2/'
>>>
>>> info "define all"
>>> define 'all' do
>>>
>>> info "define subA"
>>> define 'subA' do
>>> info "subA goes here"
>>> #test.using :integration
>>> end
>>>
>>> info "define subB"
>>> define 'subB' do
>>> info "subB goes here"
>>> #test.using :junit
>>> end
>>>
>>> info "all goes here"
>>>
>>> test.using :integration
>>>
>>> end
>>
>>
Re: unpredictable behavior of test.using :integration
Posted by Assaf Arkin <as...@labnotes.org>.
On Wed, Oct 21, 2009 at 2:10 AM, Peter Schröder <ps...@blau.de> wrote:
> another thing i noticed, is that calling
>
> project.test.using :integration in a post-define task
>
> seems to work differntly form calling it directly in the project via
>
> test.using :integration
>
> any ideas?
>
Flipping integration bit on simply changes when tests are run: after
packaging (integration), instead of before packaging (regular). All other
behavior is the same (setup, teardown, frameworks, etc).
Test options are inherited, options you define in a parent project are
inherited by its sub-project, and that should include the integration
option.
Changing the order in which tests are run involving setting up task
dependencies, which happens at the very end of the project definition, after
evaluating the block. If you then (post-define) change the setting, nothing
much happens, as task are already wired to run in non-integration order.
Assaf
>
> Am 21.10.2009 um 09:32 schrieb Peter Schröder:
>
> > hi there,
> >
> > we are experiencing somewhat strange behavior when using integration
> > tests.
> >
> > we have a setup where a project has n subprojects and only one of them
> > has tests that should be executed as integration-tests.
> >
> > we are using environment-variables to let buildr include some test-
> > classes that are only executed on our system-test-servers.
> >
> > this works very well: compiles, package artifacts, calls integration-
> > setup, performs-tests, integration-teardown
> >
> > on the other hand we have the same setup but with a single project
> > without subprojects, where the integration setup is missing, so the
> > system-tests fail without a system to test. in that project i have to
> > call the integration-target directly to have the desired behavior.
> >
> > the builds are quite complex so i wont include them here, but i played
> > around a little bit with a dummy project (see below).
> >
> > the things i dont understand are:
> >
> > 1. why are setup and teardown called for all projects even though they
> > are never configured with :integration
> > 2. why are all tests executed as :integration if you just configure it
> > in the parent project
> > 3. under wich circumstances will the integration-target get invoked
> >
> >
> > module SystemTests
> > include Extension
> >
> > after_define do |project|
> > info "after define for #{project}"
> > #project.test.using :integration
> > project.integration.setup do
> > info "-" * 50 + " integeration setup for #{project}"
> > end
> >
> > project.integration.teardown do
> > info "-" * 50 + " integeration teardown for #{project}"
> > end
> > end
> >
> > end
> >
> > class Buildr::Project
> > include SystemTests
> > end
> >
> > repositories.remote << 'http://www.ibiblio.org/maven2/'
> >
> > info "define all"
> > define 'all' do
> >
> > info "define subA"
> > define 'subA' do
> > info "subA goes here"
> > #test.using :integration
> > end
> >
> > info "define subB"
> > define 'subB' do
> > info "subB goes here"
> > #test.using :junit
> > end
> >
> > info "all goes here"
> >
> > test.using :integration
> >
> > end
>
>
Re: unpredictable behavior of test.using :integration
Posted by Peter Schröder <ps...@blau.de>.
another thing i noticed, is that calling
project.test.using :integration in a post-define task
seems to work differntly form calling it directly in the project via
test.using :integration
any ideas?
Am 21.10.2009 um 09:32 schrieb Peter Schröder:
> hi there,
>
> we are experiencing somewhat strange behavior when using integration
> tests.
>
> we have a setup where a project has n subprojects and only one of them
> has tests that should be executed as integration-tests.
>
> we are using environment-variables to let buildr include some test-
> classes that are only executed on our system-test-servers.
>
> this works very well: compiles, package artifacts, calls integration-
> setup, performs-tests, integration-teardown
>
> on the other hand we have the same setup but with a single project
> without subprojects, where the integration setup is missing, so the
> system-tests fail without a system to test. in that project i have to
> call the integration-target directly to have the desired behavior.
>
> the builds are quite complex so i wont include them here, but i played
> around a little bit with a dummy project (see below).
>
> the things i dont understand are:
>
> 1. why are setup and teardown called for all projects even though they
> are never configured with :integration
> 2. why are all tests executed as :integration if you just configure it
> in the parent project
> 3. under wich circumstances will the integration-target get invoked
>
>
> module SystemTests
> include Extension
>
> after_define do |project|
> info "after define for #{project}"
> #project.test.using :integration
> project.integration.setup do
> info "-" * 50 + " integeration setup for #{project}"
> end
>
> project.integration.teardown do
> info "-" * 50 + " integeration teardown for #{project}"
> end
> end
>
> end
>
> class Buildr::Project
> include SystemTests
> end
>
> repositories.remote << 'http://www.ibiblio.org/maven2/'
>
> info "define all"
> define 'all' do
>
> info "define subA"
> define 'subA' do
> info "subA goes here"
> #test.using :integration
> end
>
> info "define subB"
> define 'subB' do
> info "subB goes here"
> #test.using :junit
> end
>
> info "all goes here"
>
> test.using :integration
>
> end
error executing tests with no default test-folder
Posted by Peter Schröder <ps...@blau.de>.
hi,
i have a subproject that holds only system-tests. for that reason it
contains only of one class in one folder src/test-system/java .
i add this folder using test.compile.from _("src/test-system/java")
wich results in a Class not found exception:
Compiling chief:systemtest:test into /home/phoet/workspace/chief/
systemtest/target/test/classes
NativeException: java.lang.ClassNotFoundException:
de.blau.chief.ws.validation.v1_0.ValidationWebServiceImplSystemTest
Buildr aborted!
java.lang.ClassNotFoundException:
de.blau.chief.ws.validation.v1_0.ValidationWebServiceImplSystemTest
if i add a default folder src/test/java with an empty class in it
everything is fine!
this seems kind of strange to me...
kind regards
peter