Test Practices and Patterns (was Re: svn commit: r750296 - in /ofbiz/trunk/framework/testtools: dtd/test-suite.xsd src/org/ofbiz/testtools/ModelTestSuite.java src/org/ofbiz/testtools/TestListContainer.java)

[David E Jones <da...@hotwaxmedia.com>]

On Mar 4, 2009, at 8:03 PM, Adam Heath wrote:

> David E Jones wrote:
>>
>> Why not just use different test-suite elements for this? Why have an
>> hierarchy in this?
>>
>> Actually... I'm not even sure what "this" is, ie what is the point  
>> of a
>> group? Maybe we should discuss that first...
>
> The service engine tests define several different test cases.  The
> second test case does a data load.  Then each subsequent test modifies
> the data state of the previous test.  They all have to run in
> sequence, for them to pass.
>
> This means that they are *not* separate test cases, but a single test
> case.  A test case is supposed to be self-contained, not depending on
> the state of anything else.  It's supposed to start from empty, do
> whatever mutative changes it needs to, without any external
> dependencies.

Who says?

Why?

The change you made doesn't change this, it just creates something new  
called a "test group" that I'm guessing is supposed to group tests  
together than depend on eachother. However, it doesn't make the test  
cases independent.

Also, why introduce a test group concept when we already have the test- 
suite concept? Why not just use that? Run one suite at a time and only  
allow dependencies within that suite.

Still, this hasn't been discussed. The only thing we've discussed and  
somewhat decided on so far (including the ApacheCon discussions I was  
involved in, which was the same as what was discussed before) is to  
only allow dependencies between test cases within components.

We could restrict that more by only allowing dependencies between test  
cases within a suite (which I think is the more normal practice), and  
the terms "test case" and "test suite" are more known as those are  
used in JUnit and other things.

> The service test cases did not follow this.

Of course not. It's never been discussed or defined or established  
even as a recommended practice, let alone one we'll adopt... possibly  
because it doesn't make much sense?

Backing up a little bit, you can make the assertion that each test  
case should be 100% independent, but that is only the beginning of the  
discussion, not the end of it. We do need to decide on what  
dependencies will be allowed, but I'm pretty sure won't end up with an  
agreed on answer of "no dependencies period". That leads to excessive  
redundancy.

I guess one thing to keep in mind is that while we use JUnit  
underneath we're actually talking about something higher level than  
unit tests. These are general test cases and test suites, not  
necessarily just unit tests as those are really of limited utility. In  
fact, we're mostly testing processes with many steps and in the case  
of the applications tests these are meant to call a series of services  
that would normally be called through the UI, but we're dropping down  
to the service layer for them to test on that level.

> I've got a script that runs each defined test case separately; it
> stops/starts ofbiz for each individual test case, and restores a
> previous saved copy of the derby data folder between each test.

Yep, I saw that, looks cool.

Either way, unless there is a good reason to have these test-groups,  
let's just stick with test-suite. I still haven't seen anything you've  
written about what the test-group will be used for, so at least that  
would be helpful.

-David

Re: Test Practices and Patterns (was Re: svn commit: r750296 - in /ofbiz/trunk/framework/testtools: dtd/test-suite.xsd src/org/ofbiz/testtools/ModelTestSuite.java src/org/ofbiz/testtools/TestListContainer.java)

[guo weizhan <gu...@gmail.com>]

I'm fine with the test-group, I think one test suite can have some groups
and one group can contain some test cases. It's good for cases organization.

As my experience, the test cases can share some precondition and post
condition in group scope, also the groups can share precondition and post
condition too in current suite, that allows for code reuse as Adam say.

And it's the bad habit if one case depend on the other one, we should avoid
this.Test cases can share precondtion and post condition, but don't share
the test case code each other.



2009/3/5 Tim Ruppert <ti...@hotwaxmedia.com>

> Gentlemen, great debate.  I will provide a bit more information a bit later
> as I don't want to answer without fully running thru all of the scenarios
> that everyone lays out.  Adam - thanks for pushing on this - it's needed it
> for a long time.
>
> Cheers,
> Tim
> --
> Tim Ruppert
> HotWax Media
> http://www.hotwaxmedia.com
>
> o:801.649.6594
> f:801.649.6595
>
> ----- "Adam Heath" <do...@brainfood.com> wrote:
>
> > David E Jones wrote:
> > > *** First off Adam: it's great to see you working on this stuff,
> > the
> > > testing stuff in general needs a LOT more work and hasn't seen much
> > > attention, so yeah... this is great.
> >
> > Completely automatic tests with no side-effects is what I am striving
> > for.  I believe we are close.
> >
> > > On Mar 4, 2009, at 9:25 PM, Adam Heath wrote:
> > >
> > >>>> The service engine tests define several different test cases.
> > The
> > >>>> second test case does a data load.  Then each subsequent test
> > modifies
> > >>>> the data state of the previous test.  They all have to run in
> > >>>> sequence, for them to pass.
> > >>>>
> > >>>> This means that they are *not* separate test cases, but a single
> > test
> > >>>> case.  A test case is supposed to be self-contained, not
> > depending on
> > >>>> the state of anything else.  It's supposed to start from empty,
> > do
> > >>>> whatever mutative changes it needs to, without any external
> > >>>> dependencies.
> > >>>
> > >>> Who says?
> > >>>
> > >>> Why?
> > >>>
> > >>> The change you made doesn't change this, it just creates something
> > new
> > >>> called a "test group" that I'm guessing is supposed to group
> > tests
> > >>> together than depend on eachother. However, it doesn't make the
> > test
> > >>> cases independent.
> > >>>
> > >>> Also, why introduce a test group concept when we already have the
> > >>> test-suite concept? Why not just use that? Run one suite at a time
> > and
> > >>> only allow dependencies within that suite.
> > >>
> > >> Well, TestRunContainer supports running a singly-defined test case
> > >> from some testdef.xml file.  If done against service tests, then
> > >> you'll get false positives.
> > >>
> > >> I'm fine with not having this <test-group>, so long as we don't
> > allow
> > >> for running separate tests, that TestRunContainer currently
> > allows.
> > >>
> > >> Currently, we can run all tests suites inside a component, or one
> > >> separate test in a component, in whatever test suite it resides
> > in.
> > >> We have no way to restrict the lookup code to just a single test
> > suite.
> > >
> > > What does all of this have to do with it?
> > >
> > > Are you just saying that we need a way to run one test suite at a
> > time?
> > > If we're going to move toward the practice of having dependencies
> > only
> > > within a single test suite, then yes, I'd agree this is necessary.
> > >
> > > At the minute the practice is to allow dependencies within a
> > component.
> > >
> > >>> Still, this hasn't been discussed. The only thing we've discussed
> > and
> > >>> somewhat decided on so far (including the ApacheCon discussions I
> > was
> > >>> involved in, which was the same as what was discussed before) is
> > to only
> > >>> allow dependencies between test cases within components.
> > >>
> > >> Well, in some cases, multiple tests defined inside some
> > test-suite.xml
> > >> can *not* run in series, or in parallel.  Each defined test case
> > needs
> > >> a *clean* database.  In other cases, the test cases inside the
> > >> test-suite.xml *do* require running each in series.
> > >
> > > Oh yeah? Where?
> > >
> > > Here's the scary thing that I'm starting to believe to be the case:
> > > because you didn't read and research before starting to change
> > things
> > > you didn't try running the tests for a component all at once. You
> > made
> > > incorrect assumptions about how they were written and how they were
> > > meant to be run.
> >
> > But I did do some.
> >
> > The service tests require running everything in the test suite.  That
> > I am absolutely certain of.
> >
> > The other thing we know, is that running some tests together causes
> > problems.  I'm not entirely certain if these same bad tests together
> > exist in a single test suite, or if different test suites conflict.
> >
> > > Your rants and complaints about the tests may be incorrect, and the
> > > "fixes" you've been making for them may be unnecessary. They are
> > only
> > > true if you adopt your assertion that all test cases should be
> > > independent. Otherwise, they are not true and you're heading down a
> > road
> > > that others may not appreciate.
> > >
> > > Could we maybe step back and talk about this before you cruise along
> > too
> > > much more?
> >
> > I still stand by the <test-group> addition; it's a new feature, that
> > seems usefull.  It allows combining separately defined test cases
> > into
> > a single whole, without giving them each a separate name.  It allows
> > for code reuse, without having to write some java code, or a
> > simple-method, to combine them.
> >
> > However, what we are heading towards is making the granularity be a
> > test suite.  In that case, the service test definition should be
> > changed back, as well as changing my recent build.xml and
> > TestListContainer policy.
> >
> > Just to restate, I will change the policy to be each suite gets run
> > separately.
>

Re: Test Practices and Patterns (was Re: svn commit: r750296 - in /ofbiz/trunk/framework/testtools: dtd/test-suite.xsd src/org/ofbiz/testtools/ModelTestSuite.java src/org/ofbiz/testtools/TestListContainer.java)

[Tim Ruppert <ti...@hotwaxmedia.com>]

Gentlemen, great debate.  I will provide a bit more information a bit later as I don't want to answer without fully running thru all of the scenarios that everyone lays out.  Adam - thanks for pushing on this - it's needed it for a long time.

Cheers,
Tim
--
Tim Ruppert
HotWax Media
http://www.hotwaxmedia.com

o:801.649.6594
f:801.649.6595

----- "Adam Heath" <do...@brainfood.com> wrote:

> David E Jones wrote:
> > *** First off Adam: it's great to see you working on this stuff,
> the
> > testing stuff in general needs a LOT more work and hasn't seen much
> > attention, so yeah... this is great.
> 
> Completely automatic tests with no side-effects is what I am striving
> for.  I believe we are close.
> 
> > On Mar 4, 2009, at 9:25 PM, Adam Heath wrote:
> > 
> >>>> The service engine tests define several different test cases. 
> The
> >>>> second test case does a data load.  Then each subsequent test
> modifies
> >>>> the data state of the previous test.  They all have to run in
> >>>> sequence, for them to pass.
> >>>>
> >>>> This means that they are *not* separate test cases, but a single
> test
> >>>> case.  A test case is supposed to be self-contained, not
> depending on
> >>>> the state of anything else.  It's supposed to start from empty,
> do
> >>>> whatever mutative changes it needs to, without any external
> >>>> dependencies.
> >>>
> >>> Who says?
> >>>
> >>> Why?
> >>>
> >>> The change you made doesn't change this, it just creates something
> new
> >>> called a "test group" that I'm guessing is supposed to group
> tests
> >>> together than depend on eachother. However, it doesn't make the
> test
> >>> cases independent.
> >>>
> >>> Also, why introduce a test group concept when we already have the
> >>> test-suite concept? Why not just use that? Run one suite at a time
> and
> >>> only allow dependencies within that suite.
> >>
> >> Well, TestRunContainer supports running a singly-defined test case
> >> from some testdef.xml file.  If done against service tests, then
> >> you'll get false positives.
> >>
> >> I'm fine with not having this <test-group>, so long as we don't
> allow
> >> for running separate tests, that TestRunContainer currently
> allows.
> >>
> >> Currently, we can run all tests suites inside a component, or one
> >> separate test in a component, in whatever test suite it resides
> in.
> >> We have no way to restrict the lookup code to just a single test
> suite.
> > 
> > What does all of this have to do with it?
> > 
> > Are you just saying that we need a way to run one test suite at a
> time?
> > If we're going to move toward the practice of having dependencies
> only
> > within a single test suite, then yes, I'd agree this is necessary.
> > 
> > At the minute the practice is to allow dependencies within a
> component.
> > 
> >>> Still, this hasn't been discussed. The only thing we've discussed
> and
> >>> somewhat decided on so far (including the ApacheCon discussions I
> was
> >>> involved in, which was the same as what was discussed before) is
> to only
> >>> allow dependencies between test cases within components.
> >>
> >> Well, in some cases, multiple tests defined inside some
> test-suite.xml
> >> can *not* run in series, or in parallel.  Each defined test case
> needs
> >> a *clean* database.  In other cases, the test cases inside the
> >> test-suite.xml *do* require running each in series.
> > 
> > Oh yeah? Where?
> > 
> > Here's the scary thing that I'm starting to believe to be the case:
> > because you didn't read and research before starting to change
> things
> > you didn't try running the tests for a component all at once. You
> made
> > incorrect assumptions about how they were written and how they were
> > meant to be run.
> 
> But I did do some.
> 
> The service tests require running everything in the test suite.  That
> I am absolutely certain of.
> 
> The other thing we know, is that running some tests together causes
> problems.  I'm not entirely certain if these same bad tests together
> exist in a single test suite, or if different test suites conflict.
> 
> > Your rants and complaints about the tests may be incorrect, and the
> > "fixes" you've been making for them may be unnecessary. They are
> only
> > true if you adopt your assertion that all test cases should be
> > independent. Otherwise, they are not true and you're heading down a
> road
> > that others may not appreciate.
> > 
> > Could we maybe step back and talk about this before you cruise along
> too
> > much more?
> 
> I still stand by the <test-group> addition; it's a new feature, that
> seems usefull.  It allows combining separately defined test cases
> into
> a single whole, without giving them each a separate name.  It allows
> for code reuse, without having to write some java code, or a
> simple-method, to combine them.
> 
> However, what we are heading towards is making the granularity be a
> test suite.  In that case, the service test definition should be
> changed back, as well as changing my recent build.xml and
> TestListContainer policy.
> 
> Just to restate, I will change the policy to be each suite gets run
> separately.

Re: Test Practices and Patterns (was Re: svn commit: r750296 - in /ofbiz/trunk/framework/testtools: dtd/test-suite.xsd src/org/ofbiz/testtools/ModelTestSuite.java src/org/ofbiz/testtools/TestListContainer.java)

[Adam Heath <do...@brainfood.com>]

David E Jones wrote:
> *** First off Adam: it's great to see you working on this stuff, the
> testing stuff in general needs a LOT more work and hasn't seen much
> attention, so yeah... this is great.

Completely automatic tests with no side-effects is what I am striving
for.  I believe we are close.

> On Mar 4, 2009, at 9:25 PM, Adam Heath wrote:
> 
>>>> The service engine tests define several different test cases.  The
>>>> second test case does a data load.  Then each subsequent test modifies
>>>> the data state of the previous test.  They all have to run in
>>>> sequence, for them to pass.
>>>>
>>>> This means that they are *not* separate test cases, but a single test
>>>> case.  A test case is supposed to be self-contained, not depending on
>>>> the state of anything else.  It's supposed to start from empty, do
>>>> whatever mutative changes it needs to, without any external
>>>> dependencies.
>>>
>>> Who says?
>>>
>>> Why?
>>>
>>> The change you made doesn't change this, it just creates something new
>>> called a "test group" that I'm guessing is supposed to group tests
>>> together than depend on eachother. However, it doesn't make the test
>>> cases independent.
>>>
>>> Also, why introduce a test group concept when we already have the
>>> test-suite concept? Why not just use that? Run one suite at a time and
>>> only allow dependencies within that suite.
>>
>> Well, TestRunContainer supports running a singly-defined test case
>> from some testdef.xml file.  If done against service tests, then
>> you'll get false positives.
>>
>> I'm fine with not having this <test-group>, so long as we don't allow
>> for running separate tests, that TestRunContainer currently allows.
>>
>> Currently, we can run all tests suites inside a component, or one
>> separate test in a component, in whatever test suite it resides in.
>> We have no way to restrict the lookup code to just a single test suite.
> 
> What does all of this have to do with it?
> 
> Are you just saying that we need a way to run one test suite at a time?
> If we're going to move toward the practice of having dependencies only
> within a single test suite, then yes, I'd agree this is necessary.
> 
> At the minute the practice is to allow dependencies within a component.
> 
>>> Still, this hasn't been discussed. The only thing we've discussed and
>>> somewhat decided on so far (including the ApacheCon discussions I was
>>> involved in, which was the same as what was discussed before) is to only
>>> allow dependencies between test cases within components.
>>
>> Well, in some cases, multiple tests defined inside some test-suite.xml
>> can *not* run in series, or in parallel.  Each defined test case needs
>> a *clean* database.  In other cases, the test cases inside the
>> test-suite.xml *do* require running each in series.
> 
> Oh yeah? Where?
> 
> Here's the scary thing that I'm starting to believe to be the case:
> because you didn't read and research before starting to change things
> you didn't try running the tests for a component all at once. You made
> incorrect assumptions about how they were written and how they were
> meant to be run.

But I did do some.

The service tests require running everything in the test suite.  That
I am absolutely certain of.

The other thing we know, is that running some tests together causes
problems.  I'm not entirely certain if these same bad tests together
exist in a single test suite, or if different test suites conflict.

> Your rants and complaints about the tests may be incorrect, and the
> "fixes" you've been making for them may be unnecessary. They are only
> true if you adopt your assertion that all test cases should be
> independent. Otherwise, they are not true and you're heading down a road
> that others may not appreciate.
> 
> Could we maybe step back and talk about this before you cruise along too
> much more?

I still stand by the <test-group> addition; it's a new feature, that
seems usefull.  It allows combining separately defined test cases into
a single whole, without giving them each a separate name.  It allows
for code reuse, without having to write some java code, or a
simple-method, to combine them.

However, what we are heading towards is making the granularity be a
test suite.  In that case, the service test definition should be
changed back, as well as changing my recent build.xml and
TestListContainer policy.

Just to restate, I will change the policy to be each suite gets run
separately.

Re: Test Practices and Patterns (was Re: svn commit: r750296 - in /ofbiz/trunk/framework/testtools: dtd/test-suite.xsd src/org/ofbiz/testtools/ModelTestSuite.java src/org/ofbiz/testtools/TestListContainer.java)

[David E Jones <da...@hotwaxmedia.com>]

*** First off Adam: it's great to see you working on this stuff, the  
testing stuff in general needs a LOT more work and hasn't seen much  
attention, so yeah... this is great.


On Mar 4, 2009, at 9:25 PM, Adam Heath wrote:

>>> The service engine tests define several different test cases.  The
>>> second test case does a data load.  Then each subsequent test  
>>> modifies
>>> the data state of the previous test.  They all have to run in
>>> sequence, for them to pass.
>>>
>>> This means that they are *not* separate test cases, but a single  
>>> test
>>> case.  A test case is supposed to be self-contained, not depending  
>>> on
>>> the state of anything else.  It's supposed to start from empty, do
>>> whatever mutative changes it needs to, without any external
>>> dependencies.
>>
>> Who says?
>>
>> Why?
>>
>> The change you made doesn't change this, it just creates something  
>> new
>> called a "test group" that I'm guessing is supposed to group tests
>> together than depend on eachother. However, it doesn't make the test
>> cases independent.
>>
>> Also, why introduce a test group concept when we already have the
>> test-suite concept? Why not just use that? Run one suite at a time  
>> and
>> only allow dependencies within that suite.
>
> Well, TestRunContainer supports running a singly-defined test case
> from some testdef.xml file.  If done against service tests, then
> you'll get false positives.
>
> I'm fine with not having this <test-group>, so long as we don't allow
> for running separate tests, that TestRunContainer currently allows.
>
> Currently, we can run all tests suites inside a component, or one
> separate test in a component, in whatever test suite it resides in.
> We have no way to restrict the lookup code to just a single test  
> suite.

What does all of this have to do with it?

Are you just saying that we need a way to run one test suite at a  
time? If we're going to move toward the practice of having  
dependencies only within a single test suite, then yes, I'd agree this  
is necessary.

At the minute the practice is to allow dependencies within a component.

>> Still, this hasn't been discussed. The only thing we've discussed and
>> somewhat decided on so far (including the ApacheCon discussions I was
>> involved in, which was the same as what was discussed before) is to  
>> only
>> allow dependencies between test cases within components.
>
> Well, in some cases, multiple tests defined inside some test-suite.xml
> can *not* run in series, or in parallel.  Each defined test case needs
> a *clean* database.  In other cases, the test cases inside the
> test-suite.xml *do* require running each in series.

Oh yeah? Where?

Here's the scary thing that I'm starting to believe to be the case:  
because you didn't read and research before starting to change things  
you didn't try running the tests for a component all at once. You made  
incorrect assumptions about how they were written and how they were  
meant to be run.

Your rants and complaints about the tests may be incorrect, and the  
"fixes" you've been making for them may be unnecessary. They are only  
true if you adopt your assertion that all test cases should be  
independent. Otherwise, they are not true and you're heading down a  
road that others may not appreciate.

Could we maybe step back and talk about this before you cruise along  
too much more?

-David

Re: Test Practices and Patterns (was Re: svn commit: r750296 - in /ofbiz/trunk/framework/testtools: dtd/test-suite.xsd src/org/ofbiz/testtools/ModelTestSuite.java src/org/ofbiz/testtools/TestListContainer.java)

[Adam Heath <do...@brainfood.com>]

David E Jones wrote:
> 
> On Mar 4, 2009, at 8:03 PM, Adam Heath wrote:
> 
>> David E Jones wrote:
>>>
>>> Why not just use different test-suite elements for this? Why have an
>>> hierarchy in this?
>>>
>>> Actually... I'm not even sure what "this" is, ie what is the point of a
>>> group? Maybe we should discuss that first...
>>
>> The service engine tests define several different test cases.  The
>> second test case does a data load.  Then each subsequent test modifies
>> the data state of the previous test.  They all have to run in
>> sequence, for them to pass.
>>
>> This means that they are *not* separate test cases, but a single test
>> case.  A test case is supposed to be self-contained, not depending on
>> the state of anything else.  It's supposed to start from empty, do
>> whatever mutative changes it needs to, without any external
>> dependencies.
> 
> Who says?
> 
> Why?
> 
> The change you made doesn't change this, it just creates something new
> called a "test group" that I'm guessing is supposed to group tests
> together than depend on eachother. However, it doesn't make the test
> cases independent.
> 
> Also, why introduce a test group concept when we already have the
> test-suite concept? Why not just use that? Run one suite at a time and
> only allow dependencies within that suite.

Well, TestRunContainer supports running a singly-defined test case
from some testdef.xml file.  If done against service tests, then
you'll get false positives.

I'm fine with not having this <test-group>, so long as we don't allow
for running separate tests, that TestRunContainer currently allows.

Currently, we can run all tests suites inside a component, or one
separate test in a component, in whatever test suite it resides in.
We have no way to restrict the lookup code to just a single test suite.

> Still, this hasn't been discussed. The only thing we've discussed and
> somewhat decided on so far (including the ApacheCon discussions I was
> involved in, which was the same as what was discussed before) is to only
> allow dependencies between test cases within components.

Well, in some cases, multiple tests defined inside some test-suite.xml
can *not* run in series, or in parallel.  Each defined test case needs
a *clean* database.  In other cases, the test cases inside the
test-suite.xml *do* require running each in series.

> We could restrict that more by only allowing dependencies between test
> cases within a suite (which I think is the more normal practice), and
> the terms "test case" and "test suite" are more known as those are used
> in JUnit and other things.
> 
>> The service test cases did not follow this.
> 
> Of course not. It's never been discussed or defined or established even
> as a recommended practice, let alone one we'll adopt... possibly because
> it doesn't make much sense?
> 
> Backing up a little bit, you can make the assertion that each test case
> should be 100% independent, but that is only the beginning of the
> discussion, not the end of it. We do need to decide on what dependencies
> will be allowed, but I'm pretty sure won't end up with an agreed on
> answer of "no dependencies period". That leads to excessive redundancy.
> 
> I guess one thing to keep in mind is that while we use JUnit underneath
> we're actually talking about something higher level than unit tests.
> These are general test cases and test suites, not necessarily just unit
> tests as those are really of limited utility. In fact, we're mostly
> testing processes with many steps and in the case of the applications
> tests these are meant to call a series of services that would normally
> be called through the UI, but we're dropping down to the service layer
> for them to test on that level.

You and I are on the same wave length.

>> I've got a script that runs each defined test case separately; it
>> stops/starts ofbiz for each individual test case, and restores a
>> previous saved copy of the derby data folder between each test.
> 
> Yep, I saw that, looks cool.
> 
> Either way, unless there is a good reason to have these test-groups,
> let's just stick with test-suite. I still haven't seen anything you've
> written about what the test-group will be used for, so at least that
> would be helpful.

I had a later commit that changed the service tests over to it;
however, that was just an automatic-type change, so that my script
wouldn't break, by trying to run each one separately.