Functional testing harness

[Antoine Toulme <an...@lunar-ocean.com>]

I'm looking for volunteers to take a quick look at the patch attached to:
https://issues.apache.org/jira/browse/BUILDR-456

And tell me what they think.

The goal is to see Buildr in action. We have spec coverage but we have had
bad surprises on each of our releases because we lack functional testing.

Functional testing imo is a requirement to make the project rock solid.

Please comment on the bug if you like the code or not, and I'll commit or
will amend.

Thanks !

Antoine

Re: Functional testing harness

[Alex Boisvert <al...@gmail.com>]

I like it.  Previous comments already cover what I would have noted.  We can
start small and grow from there.

alex


On Wed, Jul 21, 2010 at 9:45 AM, Antoine Toulme <an...@lunar-ocean.com>wrote:

> I'm looking for volunteers to take a quick look at the patch attached to:
> https://issues.apache.org/jira/browse/BUILDR-456
>
> And tell me what they think.
>
> The goal is to see Buildr in action. We have spec coverage but we have had
> bad surprises on each of our releases because we lack functional testing.
>
> Functional testing imo is a requirement to make the project rock solid.
>
> Please comment on the bug if you like the code or not, and I'll commit or
> will amend.
>
> Thanks !
>
> Antoine
>

Re: Functional testing harness

[Antoine Toulme <an...@lunar-ocean.com>]

Thanks for correcting me. Yes, integration tests it is.

And yes, we can't do anything serious without them.

On Thu, Jul 22, 2010 at 14:01, Assaf Arkin <as...@labnotes.org> wrote:

> On Wed, Jul 21, 2010 at 10:09 PM, Antoine Toulme <an...@lunar-ocean.com>wrote:
>
>> There actually will be crucial differences between specs and functional
>> tests. Mainly specs are a free for all to mock, squeeze, open wide open
>> internals to check the code as close as possible.
>>
>> Functional tests should run with the stock buildr, no changes.
>>
>> Your points are valid: we still need to run tests in a safe environment,
>> where we make sure to dispose of artifacts created.
>> So definitely a before and an after are in order.
>> The two tests added are peculiar - they don't create files just yet.
>>
>> However you think we're better off writing the Buildfile every time ? In
>> rake I see they kept the Buildfile in their own folders. This way I can
>> picture myself pointing people to "real" examples, real files they can see
>> and try for themselves to reproduce problems.
>>
>> Even if we keep the Buildfiles, we can still copy them dynamically in a
>> testbed. So your idea stands sound to me.
>>
>> At this point, I'd like to commit what I have, and refine to do a testbed
>> as
>> you suggest. Believe it or not, this is looking pretty exciting to me :)
>>
>
> I think you're looking at integration tests here.
>
> At the granularity of a Buildfile and running a full build (how else can
> you test packaging),  you're going to be testing multiple things in one go.
>
> Functional tests look at one function at a time. And you want to quickly
> run a small set of them to test the code as you're writing it (TDD), so
> functional tests contain a lot of mocking and stubbing.
>
> A lot of problems don't surface until you run end-to-end integration tests.
> You definitely want directory for each test, with the initial set of files.
> Besides being much easier to maintain, and like you said, can point people
> there to look at actual examples.
>
> Assaf
>
>
>>
>> On Wed, Jul 21, 2010 at 21:48, Peter Donald <pe...@realityforge.org>
>> wrote:
>>
>> > Hi,
>> >
>> > On Thu, Jul 22, 2010 at 2:45 AM, Antoine Toulme <
>> antoine@lunar-ocean.com>
>> > wrote:
>> > > I'm looking for volunteers to take a quick look at the patch attached
>> to:
>> > > https://issues.apache.org/jira/browse/BUILDR-456
>> > >
>> > > And tell me what they think.
>> >
>> > As a first cut it looks not bad. I would expect to see a library of
>> > assertions built up over time to make it easy to check the output and
>> > results of the builds.
>> >
>> > One thing I would like to see is a consistency between the spec tests
>> > and the functional tests in how they setup the environment. In both
>> > cases you are typically going to need to delete then create  a scratch
>> > directory structure at start of test, configure (and populate) the
>> > maven repository. Both tests also may create some dirs and setup a
>> > build file (although in the spec tests it is done in memory while in
>> > the functional tests you are likely to be copying around files). At
>> > the end of the tests the scratch directory should be nuked.
>> >
>> > So extracting this into a common api would be great. It would also
>> > mean that it could be more easily used by outside people that are
>> > writing buildr plugins. I could imagine an api like
>> >
>> > # Put in test_helper/sepc_helper
>> > Buildr::Test.scratch_directory = File.basedir(__FILE__) + "/../tmp"
>> > Buildr::Test.clean_scratch_on_destroy = ENV["TEST_CLEAN_SCRATCH"].nil?
>> > ? true : ENV["TEST_CLEAN_SCRATCH"] == 'true'
>> >
>> > # called from setup
>> > Buildr::Test.init #
>> >
>> > # Called to copy artifact from local maven repo to maven repo in test
>> > Buildr::Test.install_artifact 'com.biz:my-widget:jar:2.0'
>> > Buildr::Test.install_artifact 'com.biz:my-other-widget:jar:2.0'
>> >
>> > # Example code to setup a functional test
>> > FileUtils.cp_r  File.basedir(__FILE__) + "/../functional/" +
>> > test_name, Buildr::Test.project_directory
>> >
>> > # called from teardown
>> > Buildr::Test.destroy
>> >
>> > Thoughts?
>> >
>> > --
>> > Cheers,
>> >
>> > Peter Donald
>> >
>>
>
>

Re: Functional testing harness

[Assaf Arkin <as...@labnotes.org>]

On Wed, Jul 21, 2010 at 10:09 PM, Antoine Toulme <an...@lunar-ocean.com>wrote:

> There actually will be crucial differences between specs and functional
> tests. Mainly specs are a free for all to mock, squeeze, open wide open
> internals to check the code as close as possible.
>
> Functional tests should run with the stock buildr, no changes.
>
> Your points are valid: we still need to run tests in a safe environment,
> where we make sure to dispose of artifacts created.
> So definitely a before and an after are in order.
> The two tests added are peculiar - they don't create files just yet.
>
> However you think we're better off writing the Buildfile every time ? In
> rake I see they kept the Buildfile in their own folders. This way I can
> picture myself pointing people to "real" examples, real files they can see
> and try for themselves to reproduce problems.
>
> Even if we keep the Buildfiles, we can still copy them dynamically in a
> testbed. So your idea stands sound to me.
>
> At this point, I'd like to commit what I have, and refine to do a testbed
> as
> you suggest. Believe it or not, this is looking pretty exciting to me :)
>

I think you're looking at integration tests here.

At the granularity of a Buildfile and running a full build (how else can you
test packaging),  you're going to be testing multiple things in one go.

Functional tests look at one function at a time. And you want to quickly run
a small set of them to test the code as you're writing it (TDD), so
functional tests contain a lot of mocking and stubbing.

A lot of problems don't surface until you run end-to-end integration tests.
You definitely want directory for each test, with the initial set of files.
Besides being much easier to maintain, and like you said, can point people
there to look at actual examples.

Assaf


>
> On Wed, Jul 21, 2010 at 21:48, Peter Donald <pe...@realityforge.org>
> wrote:
>
> > Hi,
> >
> > On Thu, Jul 22, 2010 at 2:45 AM, Antoine Toulme <antoine@lunar-ocean.com
> >
> > wrote:
> > > I'm looking for volunteers to take a quick look at the patch attached
> to:
> > > https://issues.apache.org/jira/browse/BUILDR-456
> > >
> > > And tell me what they think.
> >
> > As a first cut it looks not bad. I would expect to see a library of
> > assertions built up over time to make it easy to check the output and
> > results of the builds.
> >
> > One thing I would like to see is a consistency between the spec tests
> > and the functional tests in how they setup the environment. In both
> > cases you are typically going to need to delete then create  a scratch
> > directory structure at start of test, configure (and populate) the
> > maven repository. Both tests also may create some dirs and setup a
> > build file (although in the spec tests it is done in memory while in
> > the functional tests you are likely to be copying around files). At
> > the end of the tests the scratch directory should be nuked.
> >
> > So extracting this into a common api would be great. It would also
> > mean that it could be more easily used by outside people that are
> > writing buildr plugins. I could imagine an api like
> >
> > # Put in test_helper/sepc_helper
> > Buildr::Test.scratch_directory = File.basedir(__FILE__) + "/../tmp"
> > Buildr::Test.clean_scratch_on_destroy = ENV["TEST_CLEAN_SCRATCH"].nil?
> > ? true : ENV["TEST_CLEAN_SCRATCH"] == 'true'
> >
> > # called from setup
> > Buildr::Test.init #
> >
> > # Called to copy artifact from local maven repo to maven repo in test
> > Buildr::Test.install_artifact 'com.biz:my-widget:jar:2.0'
> > Buildr::Test.install_artifact 'com.biz:my-other-widget:jar:2.0'
> >
> > # Example code to setup a functional test
> > FileUtils.cp_r  File.basedir(__FILE__) + "/../functional/" +
> > test_name, Buildr::Test.project_directory
> >
> > # called from teardown
> > Buildr::Test.destroy
> >
> > Thoughts?
> >
> > --
> > Cheers,
> >
> > Peter Donald
> >
>

Re: Functional testing harness

[Antoine Toulme <an...@lunar-ocean.com>]

There actually will be crucial differences between specs and functional
tests. Mainly specs are a free for all to mock, squeeze, open wide open
internals to check the code as close as possible.

Functional tests should run with the stock buildr, no changes.

Your points are valid: we still need to run tests in a safe environment,
where we make sure to dispose of artifacts created.
So definitely a before and an after are in order.
The two tests added are peculiar - they don't create files just yet.

However you think we're better off writing the Buildfile every time ? In
rake I see they kept the Buildfile in their own folders. This way I can
picture myself pointing people to "real" examples, real files they can see
and try for themselves to reproduce problems.

Even if we keep the Buildfiles, we can still copy them dynamically in a
testbed. So your idea stands sound to me.

At this point, I'd like to commit what I have, and refine to do a testbed as
you suggest. Believe it or not, this is looking pretty exciting to me :)

On Wed, Jul 21, 2010 at 21:48, Peter Donald <pe...@realityforge.org> wrote:

> Hi,
>
> On Thu, Jul 22, 2010 at 2:45 AM, Antoine Toulme <an...@lunar-ocean.com>
> wrote:
> > I'm looking for volunteers to take a quick look at the patch attached to:
> > https://issues.apache.org/jira/browse/BUILDR-456
> >
> > And tell me what they think.
>
> As a first cut it looks not bad. I would expect to see a library of
> assertions built up over time to make it easy to check the output and
> results of the builds.
>
> One thing I would like to see is a consistency between the spec tests
> and the functional tests in how they setup the environment. In both
> cases you are typically going to need to delete then create  a scratch
> directory structure at start of test, configure (and populate) the
> maven repository. Both tests also may create some dirs and setup a
> build file (although in the spec tests it is done in memory while in
> the functional tests you are likely to be copying around files). At
> the end of the tests the scratch directory should be nuked.
>
> So extracting this into a common api would be great. It would also
> mean that it could be more easily used by outside people that are
> writing buildr plugins. I could imagine an api like
>
> # Put in test_helper/sepc_helper
> Buildr::Test.scratch_directory = File.basedir(__FILE__) + "/../tmp"
> Buildr::Test.clean_scratch_on_destroy = ENV["TEST_CLEAN_SCRATCH"].nil?
> ? true : ENV["TEST_CLEAN_SCRATCH"] == 'true'
>
> # called from setup
> Buildr::Test.init #
>
> # Called to copy artifact from local maven repo to maven repo in test
> Buildr::Test.install_artifact 'com.biz:my-widget:jar:2.0'
> Buildr::Test.install_artifact 'com.biz:my-other-widget:jar:2.0'
>
> # Example code to setup a functional test
> FileUtils.cp_r  File.basedir(__FILE__) + "/../functional/" +
> test_name, Buildr::Test.project_directory
>
> # called from teardown
> Buildr::Test.destroy
>
> Thoughts?
>
> --
> Cheers,
>
> Peter Donald
>

Re: Functional testing harness

[Peter Donald <pe...@realityforge.org>]

Hi,

On Thu, Jul 22, 2010 at 2:45 AM, Antoine Toulme <an...@lunar-ocean.com> wrote:
> I'm looking for volunteers to take a quick look at the patch attached to:
> https://issues.apache.org/jira/browse/BUILDR-456
>
> And tell me what they think.

As a first cut it looks not bad. I would expect to see a library of
assertions built up over time to make it easy to check the output and
results of the builds.

One thing I would like to see is a consistency between the spec tests
and the functional tests in how they setup the environment. In both
cases you are typically going to need to delete then create  a scratch
directory structure at start of test, configure (and populate) the
maven repository. Both tests also may create some dirs and setup a
build file (although in the spec tests it is done in memory while in
the functional tests you are likely to be copying around files). At
the end of the tests the scratch directory should be nuked.

So extracting this into a common api would be great. It would also
mean that it could be more easily used by outside people that are
writing buildr plugins. I could imagine an api like

# Put in test_helper/sepc_helper
Buildr::Test.scratch_directory = File.basedir(__FILE__) + "/../tmp"
Buildr::Test.clean_scratch_on_destroy = ENV["TEST_CLEAN_SCRATCH"].nil?
? true : ENV["TEST_CLEAN_SCRATCH"] == 'true'

# called from setup
Buildr::Test.init #

# Called to copy artifact from local maven repo to maven repo in test
Buildr::Test.install_artifact 'com.biz:my-widget:jar:2.0'
Buildr::Test.install_artifact 'com.biz:my-other-widget:jar:2.0'

# Example code to setup a functional test
FileUtils.cp_r  File.basedir(__FILE__) + "/../functional/" +
test_name, Buildr::Test.project_directory

# called from teardown
Buildr::Test.destroy

Thoughts?

-- 
Cheers,

Peter Donald