You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@buildr.apache.org by Assaf Arkin <ar...@intalio.com> on 2008/02/21 21:45:51 UTC
Where do we put specs: src/test or src/spec?
Here's an interesting issue brought up on JIRA, and I like to move it
to the mailing list, since not all of you are following issue 29 [1].
Daniel Roop:
Not sure where you guys stand on this issue, but I was going to
suggest
src/spec/java
src/spec/scala
etc...
This seems to be a compromise between maven convention and rails
convention.
In a rails project when you write specs instead of a
/test
you have a
/spec
so in maven instead of src/test you have src/spec
Victor:
As you can see, regarding directories for ruby based tests/specs, I
was thinking of using a single directory src/test/ruby
*_test would be rubyUnit
*_spec would be rspec examples.
A few observations of my own.
The conventions we use in Buildr are descendants of many generations
of working with various open source projects, mostly at Apache, and
date back to the early days of Ant. I picked those up because I think
they work very well and had good experience with them on projects big
and small. Compatibility with Maven 2 was an important feature, but
not the top priority.
Something unique to Buildr is the wide language diversity and some new
development practices that we're adopting as first class citizens. So
we have to accept that at some point we'll be making and following our
own conventions.
Unfortunately the best conventions are revealed over time, making them
in advance is always a tricky proposition. That's why I think we a
show of hands before deciding either way.
Personally, I like to make a distinction between unit tests and specs,
even though I only use one or the other on any given projects [2].
That convention comes from using RSpec, which also happens to be a
source of influence to many other BDD test frameworks. Another one to
look at is JBehave [3], although I can't tell if they have any
conventions for directory structure.
I happen to think it's useful to keep that distinction, so I would
like to see us using src/test for unit tests and src/spec for specs.
I don't know if everybody would like that. What do you think?
Assaf
[1] https://issues.apache.org/jira/browse/BUILDR-29
[2] Early on Buildr actually had a test directory (now spec), but that
was following a common Gem packaging convention and ignoring the RSpec
convention.
[3] http://jbehave.org/documentation/the-story-framework/
Re: Where do we put specs: src/test or src/spec?
Posted by Assaf Arkin <ar...@intalio.com>.
On Feb 22, 2008, at 12:01 PM, Matthieu Riou wrote:
>
> I don't see that. They are different in the way you write them as
> BDD is
> much better at expressing the intended behavior. However down the
> road,
> they're both used to test your code and nothing else. So what is the
> difference in purpose?
If you're looking at assert vs should, then those are two different
ways to formulate test cases. That's separate from how they're being
used.
If you're writing specs using asserts (which RSpec allows), those are
still specs. If you're writing tests using should (which can be done
with some difficulty using RUnit), those are still tests. You're
mixing APIs, but not changing the manner in which they are used.
Let's say in Buildr 1.3 the build task no longer executes on the
project in the current directory. If the test fails (on any
framework) then we know the code is broken and needs to be fixed.
That's what tests do.
But what if the test passes? Then clearly the code is correct, it
complies with the test. But it doesn't comply with the spec, and it's
the spec that tells us the code is incorrect and in addition the test
is broken.
Here's what the specification says:
build task
- should execute task for project in current directory
- should not execute task for projects in other directory
Note that the specs don't have to test anything, I just think it's
much more convenient to always use RSpec for the tests, given that I
would like to have a spec of anything before we have a test for it
(spec->test->code). We could in theory have unit tests in /test and
specs in /specs. But clearly only one of those would provide the
formal specification for what the software does, and that's a
fundamental difference.
Assaf
Re: Where do we put specs: src/test or src/spec?
Posted by Matthieu Riou <ma...@offthelip.org>.
On Fri, Feb 22, 2008 at 11:33 AM, Assaf Arkin <ar...@intalio.com> wrote:
> On Feb 22, 2008, at 11:11 AM, Matthieu Riou wrote:
>
> > I agree about the significant difference although that's not really
> > my point
> > (maybe that wasn't too clear). There are all kind of sources as well
> > and
> > they all belong to src/main. The different between test and sources is
> > mostly that one is packaged and shipped and must be part of your
> > runtime,
> > not the other. In that light spec and tests are strictly identical.
>
> Buildr packages and ships both sources and specs. You can tell
> RubyGems to run tests during installation, make sure it works on your
> environment. You can consult the specs for things the docs/code are
> not clear about. You can patch it locally and run the test suite (gem
> check -t) to make sure you're not breaking anything.
>
> Likewise, I expect that in the future you'll be able to do similar
> things with specs, e.g. generate spec report alongside rdoc for your
> Gems, and I consider both essential part of the runtime. I spend more
> time reading those than coding against them.
>
> The distinction between main, test and spec is that conceptual:
> they're different parts of your codebase that serve different purpose
> and are used in different way.
>
I don't see that. They are different in the way you write them as BDD is
much better at expressing the intended behavior. However down the road,
they're both used to test your code and nothing else. So what is the
difference in purpose?
>
> The packaging distinction exists in Java for various reasons, but Java
> is not the only model we're trying to promote here.
>
> >
> > Now if you wanted to do test/spec/java and test/junit/java that'd be
> > something else, which would make sense even if it would probably be
> > easy.
>
> We allow that, but I don't see why we would promote that by
> convention. test/junit/java will also work on TestNG, so I wouldn't
> recommend coding the framework name in the path; likewise, specs
> written against RSpec can run on several test frameworks.
>
> On the other hand, specs are fundamentally different from test. If
> you have both in the same project you're still using them
> differently. If you're packaging them for a language that supports
> that distinction, they're packaged and installed separately.
>
> Again, not a distinction Java makes, but that distinction definitely
> exists in contexts that allow for it.
>
> Assaf
>
> >
> > Having spec outside of test would surprise me, specs are tests even
> > if they
> > are different kinds of tests, they obey to the same packaging rules.
> >
> > Matthieu
> >
> > On Fri, Feb 22, 2008 at 10:24 AM, Assaf Arkin <ar...@intalio.com>
> > wrote:
> >
> >> On Feb 22, 2008, at 8:50 AM, Matthieu Riou wrote:
> >>
> >>> I'd go for src/test because, even if specs are changing the way we
> >>> test,
> >>> it's still test. If tomorrow a new nice testing library comes up
> >>> with new
> >>> testing concepts are we going to create its own src/foo directory as
> >>> well?
> >>
> >> BDD is significant, it's not another test framework that does thing
> >> slightly different than the previous one, and these conceptual
> >> evolutions don't happen frequently. Not every other month, not even
> >> very other year, that I'm just not worried about potential namespace
> >> pollution.
> >>
> >>
> >> BDD is significantly different from TDD. First, you write a
> >> specification of how the software behaves. Then you fill it up with
> >> test case that run against the implementation. Then you write the
> >> implementation to pass the tests. TDD does only two out of these
> >> three.
> >>
> >> I can tell you that 'local task should execute task for project in
> >> local directory' because we have a specification that says exactly
> >> that (build_spec.rb, line 21). If the local task doesn't execute for
> >> project in local directory, then the code is wrong and the test is
> >> wrong (for not revealing it), but the spec is still right. So you
> >> patch the test, and you patch the code to pass the test, but you
> >> don't
> >> patch the spec. The spec is the behavior we agreed upon.
> >>
> >> That is a significant different from unit tests. Unit tests have
> >> code, which may be wrong, and tests which may also be wrong, and
> >> nothing that you can rely on to express what the right thing is.
> >>
> >> When 1.3 comes out, we'll have one page on the Web site with the
> >> formal Buildr specification generated from the files in the spec
> >> directory. If the specs, code, tests and docs are in conflict, the
> >> spec is always right. If we don't like the behavior, we change the
> >> spec, but until we do that, the spec is right even if the tests are
> >> wrong.
> >>
> >> So there's is a significant difference between TDD and BDD, they're
> >> not just two different ways to write test cases. One writes formal
> >> specifications, the other doesn't.
> >>
> >> Assaf
> >>
> >>>
> >>>
> >>> Matthieu
> >>>
> >>> On Thu, Feb 21, 2008 at 1:20 PM, Victor Hugo Borja <
> vic.borja@gmail.com
> >>>>
> >>> wrote:
> >>>
> >>>>> Another one to
> >>>>> look at is JBehave, although I can't tell if they have any
> >>>>> conventions for directory structure.
> >>>>
> >>>>
> >>>> Haven't used JBehave, IIRC the
> >>>> jbehave.rb[1]<
> >>>> https://issues.apache.org/jira/secure/attachment/12376069/
> >>>> jbehave.rb>from
> >>>> John Layton just searches for *
> >>>> Behaviour.class, compiled from src/test/java
> >>>>
> >>>> [1] https://issues.apache.org/jira/browse/BUILDR-49
> >>>>
> >>>> --
> >>>> vic
> >>>>
> >>>> Quaerendo invenietis.
> >>>>
> >>
> >>
>
>
Re: Where do we put specs: src/test or src/spec?
Posted by Assaf Arkin <ar...@intalio.com>.
On Feb 22, 2008, at 11:11 AM, Matthieu Riou wrote:
> I agree about the significant difference although that's not really
> my point
> (maybe that wasn't too clear). There are all kind of sources as well
> and
> they all belong to src/main. The different between test and sources is
> mostly that one is packaged and shipped and must be part of your
> runtime,
> not the other. In that light spec and tests are strictly identical.
Buildr packages and ships both sources and specs. You can tell
RubyGems to run tests during installation, make sure it works on your
environment. You can consult the specs for things the docs/code are
not clear about. You can patch it locally and run the test suite (gem
check -t) to make sure you're not breaking anything.
Likewise, I expect that in the future you'll be able to do similar
things with specs, e.g. generate spec report alongside rdoc for your
Gems, and I consider both essential part of the runtime. I spend more
time reading those than coding against them.
The distinction between main, test and spec is that conceptual:
they're different parts of your codebase that serve different purpose
and are used in different way.
The packaging distinction exists in Java for various reasons, but Java
is not the only model we're trying to promote here.
>
> Now if you wanted to do test/spec/java and test/junit/java that'd be
> something else, which would make sense even if it would probably be
> easy.
We allow that, but I don't see why we would promote that by
convention. test/junit/java will also work on TestNG, so I wouldn't
recommend coding the framework name in the path; likewise, specs
written against RSpec can run on several test frameworks.
On the other hand, specs are fundamentally different from test. If
you have both in the same project you're still using them
differently. If you're packaging them for a language that supports
that distinction, they're packaged and installed separately.
Again, not a distinction Java makes, but that distinction definitely
exists in contexts that allow for it.
Assaf
>
> Having spec outside of test would surprise me, specs are tests even
> if they
> are different kinds of tests, they obey to the same packaging rules.
>
> Matthieu
>
> On Fri, Feb 22, 2008 at 10:24 AM, Assaf Arkin <ar...@intalio.com>
> wrote:
>
>> On Feb 22, 2008, at 8:50 AM, Matthieu Riou wrote:
>>
>>> I'd go for src/test because, even if specs are changing the way we
>>> test,
>>> it's still test. If tomorrow a new nice testing library comes up
>>> with new
>>> testing concepts are we going to create its own src/foo directory as
>>> well?
>>
>> BDD is significant, it's not another test framework that does thing
>> slightly different than the previous one, and these conceptual
>> evolutions don't happen frequently. Not every other month, not even
>> very other year, that I'm just not worried about potential namespace
>> pollution.
>>
>>
>> BDD is significantly different from TDD. First, you write a
>> specification of how the software behaves. Then you fill it up with
>> test case that run against the implementation. Then you write the
>> implementation to pass the tests. TDD does only two out of these
>> three.
>>
>> I can tell you that 'local task should execute task for project in
>> local directory' because we have a specification that says exactly
>> that (build_spec.rb, line 21). If the local task doesn't execute for
>> project in local directory, then the code is wrong and the test is
>> wrong (for not revealing it), but the spec is still right. So you
>> patch the test, and you patch the code to pass the test, but you
>> don't
>> patch the spec. The spec is the behavior we agreed upon.
>>
>> That is a significant different from unit tests. Unit tests have
>> code, which may be wrong, and tests which may also be wrong, and
>> nothing that you can rely on to express what the right thing is.
>>
>> When 1.3 comes out, we'll have one page on the Web site with the
>> formal Buildr specification generated from the files in the spec
>> directory. If the specs, code, tests and docs are in conflict, the
>> spec is always right. If we don't like the behavior, we change the
>> spec, but until we do that, the spec is right even if the tests are
>> wrong.
>>
>> So there's is a significant difference between TDD and BDD, they're
>> not just two different ways to write test cases. One writes formal
>> specifications, the other doesn't.
>>
>> Assaf
>>
>>>
>>>
>>> Matthieu
>>>
>>> On Thu, Feb 21, 2008 at 1:20 PM, Victor Hugo Borja <vic.borja@gmail.com
>>>>
>>> wrote:
>>>
>>>>> Another one to
>>>>> look at is JBehave, although I can't tell if they have any
>>>>> conventions for directory structure.
>>>>
>>>>
>>>> Haven't used JBehave, IIRC the
>>>> jbehave.rb[1]<
>>>> https://issues.apache.org/jira/secure/attachment/12376069/
>>>> jbehave.rb>from
>>>> John Layton just searches for *
>>>> Behaviour.class, compiled from src/test/java
>>>>
>>>> [1] https://issues.apache.org/jira/browse/BUILDR-49
>>>>
>>>> --
>>>> vic
>>>>
>>>> Quaerendo invenietis.
>>>>
>>
>>
Re: Where do we put specs: src/test or src/spec?
Posted by Matthieu Riou <ma...@offthelip.org>.
I agree about the significant difference although that's not really my point
(maybe that wasn't too clear). There are all kind of sources as well and
they all belong to src/main. The different between test and sources is
mostly that one is packaged and shipped and must be part of your runtime,
not the other. In that light spec and tests are strictly identical.
Now if you wanted to do test/spec/java and test/junit/java that'd be
something else, which would make sense even if it would probably be easy.
Having spec outside of test would surprise me, specs are tests even if they
are different kinds of tests, they obey to the same packaging rules.
Matthieu
On Fri, Feb 22, 2008 at 10:24 AM, Assaf Arkin <ar...@intalio.com> wrote:
> On Feb 22, 2008, at 8:50 AM, Matthieu Riou wrote:
>
> > I'd go for src/test because, even if specs are changing the way we
> > test,
> > it's still test. If tomorrow a new nice testing library comes up
> > with new
> > testing concepts are we going to create its own src/foo directory as
> > well?
>
> BDD is significant, it's not another test framework that does thing
> slightly different than the previous one, and these conceptual
> evolutions don't happen frequently. Not every other month, not even
> very other year, that I'm just not worried about potential namespace
> pollution.
>
>
> BDD is significantly different from TDD. First, you write a
> specification of how the software behaves. Then you fill it up with
> test case that run against the implementation. Then you write the
> implementation to pass the tests. TDD does only two out of these three.
>
> I can tell you that 'local task should execute task for project in
> local directory' because we have a specification that says exactly
> that (build_spec.rb, line 21). If the local task doesn't execute for
> project in local directory, then the code is wrong and the test is
> wrong (for not revealing it), but the spec is still right. So you
> patch the test, and you patch the code to pass the test, but you don't
> patch the spec. The spec is the behavior we agreed upon.
>
> That is a significant different from unit tests. Unit tests have
> code, which may be wrong, and tests which may also be wrong, and
> nothing that you can rely on to express what the right thing is.
>
> When 1.3 comes out, we'll have one page on the Web site with the
> formal Buildr specification generated from the files in the spec
> directory. If the specs, code, tests and docs are in conflict, the
> spec is always right. If we don't like the behavior, we change the
> spec, but until we do that, the spec is right even if the tests are
> wrong.
>
> So there's is a significant difference between TDD and BDD, they're
> not just two different ways to write test cases. One writes formal
> specifications, the other doesn't.
>
> Assaf
>
> >
> >
> > Matthieu
> >
> > On Thu, Feb 21, 2008 at 1:20 PM, Victor Hugo Borja <vic.borja@gmail.com
> > >
> > wrote:
> >
> >>> Another one to
> >>> look at is JBehave, although I can't tell if they have any
> >>> conventions for directory structure.
> >>
> >>
> >> Haven't used JBehave, IIRC the
> >> jbehave.rb[1]<
> >> https://issues.apache.org/jira/secure/attachment/12376069/
> >> jbehave.rb>from
> >> John Layton just searches for *
> >> Behaviour.class, compiled from src/test/java
> >>
> >> [1] https://issues.apache.org/jira/browse/BUILDR-49
> >>
> >> --
> >> vic
> >>
> >> Quaerendo invenietis.
> >>
>
>
Re: Where do we put specs: src/test or src/spec?
Posted by Assaf Arkin <ar...@intalio.com>.
On Feb 22, 2008, at 8:50 AM, Matthieu Riou wrote:
> I'd go for src/test because, even if specs are changing the way we
> test,
> it's still test. If tomorrow a new nice testing library comes up
> with new
> testing concepts are we going to create its own src/foo directory as
> well?
BDD is significant, it's not another test framework that does thing
slightly different than the previous one, and these conceptual
evolutions don't happen frequently. Not every other month, not even
very other year, that I'm just not worried about potential namespace
pollution.
BDD is significantly different from TDD. First, you write a
specification of how the software behaves. Then you fill it up with
test case that run against the implementation. Then you write the
implementation to pass the tests. TDD does only two out of these three.
I can tell you that 'local task should execute task for project in
local directory' because we have a specification that says exactly
that (build_spec.rb, line 21). If the local task doesn't execute for
project in local directory, then the code is wrong and the test is
wrong (for not revealing it), but the spec is still right. So you
patch the test, and you patch the code to pass the test, but you don't
patch the spec. The spec is the behavior we agreed upon.
That is a significant different from unit tests. Unit tests have
code, which may be wrong, and tests which may also be wrong, and
nothing that you can rely on to express what the right thing is.
When 1.3 comes out, we'll have one page on the Web site with the
formal Buildr specification generated from the files in the spec
directory. If the specs, code, tests and docs are in conflict, the
spec is always right. If we don't like the behavior, we change the
spec, but until we do that, the spec is right even if the tests are
wrong.
So there's is a significant difference between TDD and BDD, they're
not just two different ways to write test cases. One writes formal
specifications, the other doesn't.
Assaf
>
>
> Matthieu
>
> On Thu, Feb 21, 2008 at 1:20 PM, Victor Hugo Borja <vic.borja@gmail.com
> >
> wrote:
>
>>> Another one to
>>> look at is JBehave, although I can't tell if they have any
>>> conventions for directory structure.
>>
>>
>> Haven't used JBehave, IIRC the
>> jbehave.rb[1]<
>> https://issues.apache.org/jira/secure/attachment/12376069/
>> jbehave.rb>from
>> John Layton just searches for *
>> Behaviour.class, compiled from src/test/java
>>
>> [1] https://issues.apache.org/jira/browse/BUILDR-49
>>
>> --
>> vic
>>
>> Quaerendo invenietis.
>>
Re: Where do we put specs: src/test or src/spec?
Posted by Assaf Arkin <ar...@intalio.com>.
On Feb 22, 2008, at 10:06 AM, Alex Boisvert wrote:
> Anyone knows what is the reasoning to separate unit tests and specs in
> Rails? Is there a good argument for separating them in
> Java/Scala/Groovy/...?
Specs are specs, tests are tests, not the same thing (see other e-mail
I just sent). The same argument applies to Java, Scala, Groovy, even
FORTRAN (yes, they do have a TDD framework for FORTRAN, and it's
compiled using Rake).
>
>
> And I'll throw in a twist. Do you run unit tests first, then
> specs? Or
> vice-versa? Does it matter?
Doesn't really matter. When we do a Buildr release we run them first
as tests on JRuby to make sure we didn't break anything, and a second
time as specs on Ruby to also produce the HTML specification.
Assaf
>
>
> alex
>
>
> On 2/22/08, Matthieu Riou <ma...@offthelip.org> wrote:
>>
>> I'd go for src/test because, even if specs are changing the way we
>> test,
>> it's still test. If tomorrow a new nice testing library comes up
>> with new
>> testing concepts are we going to create its own src/foo directory
>> as well?
>>
>> Matthieu
>>
>> On Thu, Feb 21, 2008 at 1:20 PM, Victor Hugo Borja <vic.borja@gmail.com
>> >
>> wrote:
>>
>>
>>>> Another one to
>>>> look at is JBehave, although I can't tell if they have any
>>>> conventions for directory structure.
>>>
>>>
>>> Haven't used JBehave, IIRC the
>>> jbehave.rb[1]<
>>> https://issues.apache.org/jira/secure/attachment/12376069/jbehave.rb
>>> from
>>> John Layton just searches for *
>>> Behaviour.class, compiled from src/test/java
>>>
>>> [1] https://issues.apache.org/jira/browse/BUILDR-49
>>>
>>> --
>>> vic
>>>
>>> Quaerendo invenietis.
>>>
>>
Re: Where do we put specs: src/test or src/spec?
Posted by Alex Boisvert <bo...@intalio.com>.
Anyone knows what is the reasoning to separate unit tests and specs in
Rails? Is there a good argument for separating them in
Java/Scala/Groovy/...?
And I'll throw in a twist. Do you run unit tests first, then specs? Or
vice-versa? Does it matter?
alex
On 2/22/08, Matthieu Riou <ma...@offthelip.org> wrote:
>
> I'd go for src/test because, even if specs are changing the way we test,
> it's still test. If tomorrow a new nice testing library comes up with new
> testing concepts are we going to create its own src/foo directory as well?
>
> Matthieu
>
> On Thu, Feb 21, 2008 at 1:20 PM, Victor Hugo Borja <vi...@gmail.com>
> wrote:
>
>
> > > Another one to
> > > look at is JBehave, although I can't tell if they have any
> > > conventions for directory structure.
> >
> >
> > Haven't used JBehave, IIRC the
> > jbehave.rb[1]<
> > https://issues.apache.org/jira/secure/attachment/12376069/jbehave.rb
> >from
> > John Layton just searches for *
> > Behaviour.class, compiled from src/test/java
> >
> > [1] https://issues.apache.org/jira/browse/BUILDR-49
> >
> > --
> > vic
> >
> > Quaerendo invenietis.
> >
>
Re: Where do we put specs: src/test or src/spec?
Posted by Matthieu Riou <ma...@offthelip.org>.
I'd go for src/test because, even if specs are changing the way we test,
it's still test. If tomorrow a new nice testing library comes up with new
testing concepts are we going to create its own src/foo directory as well?
Matthieu
On Thu, Feb 21, 2008 at 1:20 PM, Victor Hugo Borja <vi...@gmail.com>
wrote:
> > Another one to
> > look at is JBehave, although I can't tell if they have any
> > conventions for directory structure.
>
>
> Haven't used JBehave, IIRC the
> jbehave.rb[1]<
> https://issues.apache.org/jira/secure/attachment/12376069/jbehave.rb>from
> John Layton just searches for *
> Behaviour.class, compiled from src/test/java
>
> [1] https://issues.apache.org/jira/browse/BUILDR-49
>
> --
> vic
>
> Quaerendo invenietis.
>
Re: Where do we put specs: src/test or src/spec?
Posted by Victor Hugo Borja <vi...@gmail.com>.
> Another one to
> look at is JBehave, although I can't tell if they have any
> conventions for directory structure.
Haven't used JBehave, IIRC the
jbehave.rb[1]<https://issues.apache.org/jira/secure/attachment/12376069/jbehave.rb>from
John Layton just searches for *
Behaviour.class, compiled from src/test/java
[1] https://issues.apache.org/jira/browse/BUILDR-49
--
vic
Quaerendo invenietis.