Test Strategy

[Kieran <ki...@esperi.demon.co.uk>]

On Thu, 1 Aug 2002, Sander Striker wrote:

> All tests succeeded with that patch in place, so I think we need
> a new test which takes this into account.  I kind of was expecting
> failures with my patch in place ;)

Have you noticed that all tests pass in make check?  That means
that subversion is officially perfect, right?

Might it be an idea to split tests into "regression" and "bugs"?
That might encourage people to write or solitcit tests before
bug-hunting, which I think would be a win in the long-run.

Regards

Kieran


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Test Strategy

[Karl Fogel <kf...@newton.ch.collab.net>]

Kieran <ki...@esperi.demon.co.uk> writes:
> I don't believe this is the whole story.  You're trying to say that
> no developer had the time to write even one bug test?

There are tests attached to some issues.

> At the moment, I don't think we have any "bug tests".  I'm new,
> but I haven't seen any tests fail yet that I didn't write myself.

We break regression tests *all the time*, in the course of developing
new code.  Of course, you don't see that code get checked in, because
the regression tests told us not to :-).

> Of course, this doesn't deal with the lack of time at all.  Anyway,
> should I shut up, or does my idea have merit?

So the proposal is basically to include currently failing tests in the
dist?  That one can do right now.  Just put the test in a foo_test.py
file, but don't add it to the test_list[], so it doesn't get run.
People have done that before plenty of times, nothing wrong with it.

If you want a way for people to run all currently failing tests, I'm
not sure what the point is...  If someone's working on a specific bug,
they can run that test manually very easily.  If they want to find out
what bugs need to be fixed, they can look in the issue tracker.

I'm not saying it's a bad idea, I just am not seeing the purpose.
Please enlighten?

Thanks,
-Karl

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Test Strategy

[Kieran <ki...@esperi.demon.co.uk>]

On 1 Aug 2002, Karl Fogel wrote:

> Kieran <ki...@esperi.demon.co.uk> writes:
>
> > Might it be an idea to split tests into "regression" and "bugs"?
> > That might encourage people to write or solitcit tests before
> > bug-hunting, which I think would be a win in the long-run.
>
> What prevents people from writing tests for their bug reports or
> patches is lack of time, not a matter of labeling.  There's no real
> difference between a "bug test" and a "regression test".  Before a fix
> is committed, the test is a reproduction recipe; after the fix is
> applied, the same test is now a regression test.
>
I don't believe this is the whole story.  You're trying to say that
no developer had the time to write even one bug test?

At the moment, I don't think we have any "bug tests".  I'm new,
but I haven't seen any tests fail yet that I didn't write myself.

And I'm not all that surprised, since if I'd seen a failed test in my
build, I'd be a bit worried.  Since there are of the order of 50 open
bugs, 50 failed tests would cause a flood of mail to the list.

Since I'm just shooting at the status quo, here's something concrete
for discussion, which should avoid the flood of mail:

Idea 1:
In tests/clients/cmdline, change the names to add the word
regression, as in stat_tests.py -> stat_tests_regression.py, then
add stat_tests_bugs.py.  Each test in ..._bugs would ideally include
reference to an issue number.

Idea 2:
Create 2 directories tests/client/cmdline/stat_tests_regression and
.../stat_tests_bugs.  In here, add single-test files.  This would
make moving tests from bugs to regression easier.

"make check" would run the regression tests, and "make check-all"
would run all the tests

Of course, this doesn't deal with the lack of time at all.  Anyway,
should I shut up, or does my idea have merit?

Regards

Kieran



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Test Strategy

[Karl Fogel <kf...@newton.ch.collab.net>]

Kieran <ki...@esperi.demon.co.uk> writes:
> Have you noticed that all tests pass in make check?  That means
> that subversion is officially perfect, right?

No, no one believes that :-).

Passing the regression suite is not a claim that there are no bugs.
But *not* passing it is a claim that there *are* bugs, that's all.

> Might it be an idea to split tests into "regression" and "bugs"?
> That might encourage people to write or solitcit tests before
> bug-hunting, which I think would be a win in the long-run.

What prevents people from writing tests for their bug reports or
patches is lack of time, not a matter of labeling.  There's no real
difference between a "bug test" and a "regression test".  Before a fix
is committed, the test is a reproduction recipe; after the fix is
applied, the same test is now a regression test.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org