Re: Test Practices and Patterns (was Re: svn commit: r750296 - in /ofbiz/trunk/framework/testtools: dtd/test-suite.xsd src/org/ofbiz/testtools/ModelTestSuite.java src/org/ofbiz/testtools/TestListContainer.java)

[guo weizhan <gu...@gmail.com>]

I'm fine with the test-group, I think one test suite can have some groups
and one group can contain some test cases. It's good for cases organization.

As my experience, the test cases can share some precondition and post
condition in group scope, also the groups can share precondition and post
condition too in current suite, that allows for code reuse as Adam say.

And it's the bad habit if one case depend on the other one, we should avoid
this.Test cases can share precondtion and post condition, but don't share
the test case code each other.



2009/3/5 Tim Ruppert <ti...@hotwaxmedia.com>

> Gentlemen, great debate.  I will provide a bit more information a bit later
> as I don't want to answer without fully running thru all of the scenarios
> that everyone lays out.  Adam - thanks for pushing on this - it's needed it
> for a long time.
>
> Cheers,
> Tim
> --
> Tim Ruppert
> HotWax Media
> http://www.hotwaxmedia.com
>
> o:801.649.6594
> f:801.649.6595
>
> ----- "Adam Heath" <do...@brainfood.com> wrote:
>
> > David E Jones wrote:
> > > *** First off Adam: it's great to see you working on this stuff,
> > the
> > > testing stuff in general needs a LOT more work and hasn't seen much
> > > attention, so yeah... this is great.
> >
> > Completely automatic tests with no side-effects is what I am striving
> > for.  I believe we are close.
> >
> > > On Mar 4, 2009, at 9:25 PM, Adam Heath wrote:
> > >
> > >>>> The service engine tests define several different test cases.
> > The
> > >>>> second test case does a data load.  Then each subsequent test
> > modifies
> > >>>> the data state of the previous test.  They all have to run in
> > >>>> sequence, for them to pass.
> > >>>>
> > >>>> This means that they are *not* separate test cases, but a single
> > test
> > >>>> case.  A test case is supposed to be self-contained, not
> > depending on
> > >>>> the state of anything else.  It's supposed to start from empty,
> > do
> > >>>> whatever mutative changes it needs to, without any external
> > >>>> dependencies.
> > >>>
> > >>> Who says?
> > >>>
> > >>> Why?
> > >>>
> > >>> The change you made doesn't change this, it just creates something
> > new
> > >>> called a "test group" that I'm guessing is supposed to group
> > tests
> > >>> together than depend on eachother. However, it doesn't make the
> > test
> > >>> cases independent.
> > >>>
> > >>> Also, why introduce a test group concept when we already have the
> > >>> test-suite concept? Why not just use that? Run one suite at a time
> > and
> > >>> only allow dependencies within that suite.
> > >>
> > >> Well, TestRunContainer supports running a singly-defined test case
> > >> from some testdef.xml file.  If done against service tests, then
> > >> you'll get false positives.
> > >>
> > >> I'm fine with not having this <test-group>, so long as we don't
> > allow
> > >> for running separate tests, that TestRunContainer currently
> > allows.
> > >>
> > >> Currently, we can run all tests suites inside a component, or one
> > >> separate test in a component, in whatever test suite it resides
> > in.
> > >> We have no way to restrict the lookup code to just a single test
> > suite.
> > >
> > > What does all of this have to do with it?
> > >
> > > Are you just saying that we need a way to run one test suite at a
> > time?
> > > If we're going to move toward the practice of having dependencies
> > only
> > > within a single test suite, then yes, I'd agree this is necessary.
> > >
> > > At the minute the practice is to allow dependencies within a
> > component.
> > >
> > >>> Still, this hasn't been discussed. The only thing we've discussed
> > and
> > >>> somewhat decided on so far (including the ApacheCon discussions I
> > was
> > >>> involved in, which was the same as what was discussed before) is
> > to only
> > >>> allow dependencies between test cases within components.
> > >>
> > >> Well, in some cases, multiple tests defined inside some
> > test-suite.xml
> > >> can *not* run in series, or in parallel.  Each defined test case
> > needs
> > >> a *clean* database.  In other cases, the test cases inside the
> > >> test-suite.xml *do* require running each in series.
> > >
> > > Oh yeah? Where?
> > >
> > > Here's the scary thing that I'm starting to believe to be the case:
> > > because you didn't read and research before starting to change
> > things
> > > you didn't try running the tests for a component all at once. You
> > made
> > > incorrect assumptions about how they were written and how they were
> > > meant to be run.
> >
> > But I did do some.
> >
> > The service tests require running everything in the test suite.  That
> > I am absolutely certain of.
> >
> > The other thing we know, is that running some tests together causes
> > problems.  I'm not entirely certain if these same bad tests together
> > exist in a single test suite, or if different test suites conflict.
> >
> > > Your rants and complaints about the tests may be incorrect, and the
> > > "fixes" you've been making for them may be unnecessary. They are
> > only
> > > true if you adopt your assertion that all test cases should be
> > > independent. Otherwise, they are not true and you're heading down a
> > road
> > > that others may not appreciate.
> > >
> > > Could we maybe step back and talk about this before you cruise along
> > too
> > > much more?
> >
> > I still stand by the <test-group> addition; it's a new feature, that
> > seems usefull.  It allows combining separately defined test cases
> > into
> > a single whole, without giving them each a separate name.  It allows
> > for code reuse, without having to write some java code, or a
> > simple-method, to combine them.
> >
> > However, what we are heading towards is making the granularity be a
> > test suite.  In that case, the service test definition should be
> > changed back, as well as changing my recent build.xml and
> > TestListContainer policy.
> >
> > Just to restate, I will change the policy to be each suite gets run
> > separately.
>