test harness issues

[Gary Shea <sh...@gtsdesign.com>]

I have been struggling with the test harness and want to propose some
changes.  The things that are bothering me are:

1) It is currently impossible to run any of the integration tests (and I
believe some of the unit tests) by themselves.  When I have lots of
debuggin turned on, it's a real pain to wade through the output of a
hundred tests in order to look at just one.

2) In the integration tests, I'm often not seeing the name of the test
in stack traces.  I haven't figured this one out yet, but it makes it
really hard to figure out what test caused the problem.

I would like to go to a more 'standard' junit test harness where each
test can be run standalone.  The usual way to do that is of course to
extend TestCase so that it sets up the test fixtures for the test, and
let the test extend the extended TestCase class.  I don't see any
inherent technical problems with doing that.

In the most naive possible implementation the worst thing about this
approach would be allocating a new XmlDbClient for each test, hardly a
big deal when the time scale is being set by disk accesses.

Anyone have opinions on the subject?

If there's no major disagreement, I'd appreciate some help from Vladimir
(aka Dr. Test) on how to make these changes while taking advantage of
his junit addons in order to get guaranteed tearDown.

Regards,

	Gary

Re: test harness issues

["Vladimir R. Bossicard" <vl...@apache.org>]

> I agree that the design of JUnit leaves a lot to be desired.

The design is clever but it has some difficulties when your tests are more
complicated.

> I hope you can get VJUnit working soon, or can think of a better
> alternative than return to JUnit.  I'm sure trying!

I'll meet tomorrow with the author of the Atrima TestSuiteRunner and I'll ask
him if he has a ready-to-go solution.

-Vladimir

-- 
Vladimir R. Bossicard
Apache Xindice - http://xml.apache.org/xindice

Re: test harness issues

[Gary Shea <sh...@gtsdesign.com>]

Hi Vladimir --

On Wed, 12 Mar 2003, at 13:06 [-0800], Vladimir R. Bossicard...:

> > I have been struggling with the test harness and want to propose some
> > changes.  The things that are bothering me are:
> 
> Finally a real-life problem to solve :-)
> 
> > 1) It is currently impossible to run any of the integration tests (and I
> > believe some of the unit tests) by themselves.  When I have lots of
> > debuggin turned on, it's a real pain to wade through the output of a
> > hundred tests in order to look at just one.
> 
> To have fewer connection/disconnection I (too) heavily used SetUp extensions. 
> But this is a JUnit problem more than a Xindice-test problems.  When a suite()
> method returns a TestSetup object, the TestSetup methods are not executed if you
> reexecute a single test contained in the suite() returned object.  And
> reexecuting a test is the same thing as executing a single test from a suite.
> 
> It's an excellent point but changing the tests to accomodate JUnit is not the
> best solution I can think of.  But let me a few days to think of it.  I have a
> TestRunner on the work so maybe it's the solution (finally a real TestRunner!).

I agree that the design of JUnit leaves a lot to be desired.  How they
missed putting the tearDown into a finally block I'll never know.  And
why not do so in a new release?

I hope you can get VJUnit working soon, or can think of a better
alternative than return to JUnit.  I'm sure trying!

However, if you foresee it being a while before you have a solid solution,
and we can't come up with something that allows clean flexible use of
the tests, I'd prefer to go back to dumb old JUnit and do the dumb JUnit
thing: do the clean up before and after.  Tacky, annoying... but
effective.

> > 2) In the integration tests, I'm often not seeing the name of the test
> > in stack traces.  I haven't figured this one out yet, but it makes it
> > really hard to figure out what test caused the problem.
> 
> can you give me the name of a test method and I'll have a look.

I wish I could, and I know how hard it is to debug a problem that you
can't see in action!  I was hoping you'd just know what I was talking
about.  Next time I see it, I'll let you know.

> 
> > big deal when the time scale is being set by disk accesses.
> 
> and by XMLRPC connection time... which may (or may not) be a big deal
> 
> > If there's no major disagreement, I'd appreciate some help from Vladimir
> > (aka Dr. Test) on how to make these changes while taking advantage of
> > his junit addons in order to get guaranteed tearDown.
> 
> That's what I said: if you want to guarantee tearDown, you'll have a hard time
> executing one single test.  I am only 99% sure of what I just said, I have to
> check this out.
> 
> In the meantime, you can always redefine your AllTests.java to only include part
> of the tests to run (it's not optimal, I know).

Urk!  :)

> 
> -Vladimir
> 
> -- 
> Vladimir R. Bossicard
> Apache Xindice - http://xml.apache.org/xindice
> 
> 
> 

Regards,

	Gary

Re: test harness issues

["Vladimir R. Bossicard" <vl...@apache.org>]

> I have been struggling with the test harness and want to propose some
> changes.  The things that are bothering me are:

Finally a real-life problem to solve :-)

> 1) It is currently impossible to run any of the integration tests (and I
> believe some of the unit tests) by themselves.  When I have lots of
> debuggin turned on, it's a real pain to wade through the output of a
> hundred tests in order to look at just one.

To have fewer connection/disconnection I (too) heavily used SetUp extensions. 
But this is a JUnit problem more than a Xindice-test problems.  When a suite()
method returns a TestSetup object, the TestSetup methods are not executed if you
reexecute a single test contained in the suite() returned object.  And
reexecuting a test is the same thing as executing a single test from a suite.

It's an excellent point but changing the tests to accomodate JUnit is not the
best solution I can think of.  But let me a few days to think of it.  I have a
TestRunner on the work so maybe it's the solution (finally a real TestRunner!).

> 2) In the integration tests, I'm often not seeing the name of the test
> in stack traces.  I haven't figured this one out yet, but it makes it
> really hard to figure out what test caused the problem.

can you give me the name of a test method and I'll have a look.

> big deal when the time scale is being set by disk accesses.

and by XMLRPC connection time... which may (or may not) be a big deal

> If there's no major disagreement, I'd appreciate some help from Vladimir
> (aka Dr. Test) on how to make these changes while taking advantage of
> his junit addons in order to get guaranteed tearDown.

That's what I said: if you want to guarantee tearDown, you'll have a hard time
executing one single test.  I am only 99% sure of what I just said, I have to
check this out.

In the meantime, you can always redefine your AllTests.java to only include part
of the tests to run (it's not optimal, I know).

-Vladimir

-- 
Vladimir R. Bossicard
Apache Xindice - http://xml.apache.org/xindice