Cleaning up the test framework

["Malte S. Stretz" <ms...@gmx.net>]

Moin,

I think I found something to kill some time on:  Cleaning up the test 
framework.  Currently each test is created by copy-and-pasting some old one 
as a skeleton and modify it according to the needs.  This introduced some 
kludgy stuff, like the three line preamble each test has and sometimes the 
chdir("t") stuff which shouldn't be necessary because that's already done 
in sa_t_init().

I'd like to replace the current medley with some cleaner framework with 
which its easier to write tests.

The first change would replace the current three-line preamble with one 
single 'use SATest'.  That's possible with the attached patch and some 
additional code.  This will mean that the executable attribute on all those 
tests (which you guys always forget to set anyway :) will disappear.  
Instead there will be a short script called "run" or something which you 
can use to execute single tests.

The next thing is that the tests are currently run in more or less random 
order (yeah, alphabetic but logically randomly).  I'd like to introduce a 
schmema like the one we have for the rule files, ie. names like nn_test.t 
where nn is a number, the first digit roughly indicating some group (basic, 
spamd, bayes, ...), the second the precedence within that group.

Then its time for bug 3818 which actually led me to this plan :)  I'm not 
sure if that should be based on the grouping mentioned above or some 
additional keywords for each test, but I'd like to make the disabling (and 
enabling) of single tests more flexible.  So, say, some Bayes SQL test 
could have the keywords "bayes sql net" -- only if all three keywords are 
enabled, the test is run.

I hope a slimmer and documented (and thus easier to understand) framework 
for the tests will make people write more tests.  On the other hand do I 
not want to increase the number of tests run for the mere mortal who 
installs SA via CPAN and already has to wait ages.  So I'd like to keyword 
some of the tests as "basic" (or whatever keyword) and only those tests are 
run per default.  All other tests would be used by us devs, people who we 
ask to debug one of their bug reports aund the BuildBots.

Oh, and another thing is separating the real temp data from the log data, so 
the latter is better to find.  Also make it easier to use the dirs by 
feeding some vars to the scripts.

Plus maybe some helper routines like ok_if_equal() (yeah, I've been working 
a lot with JUnit lately).

And I almost forgot:  Making the scripts actually compliant with 
Test::Harness -- currently any unknown output on stdout is just ignored but 
the man page has a big fat warning that this might change in future.  And 
we really ought to give reasons why tests are skipped.

Any other things you'd like to see?

Cheers,
Malte

-- 
[SGT] Simon G. Tatham: "How to Report Bugs Effectively"
      <http://www.chiark.greenend.org.uk/~sgtatham/bugs.html>
[ESR] Eric S. Raymond: "How To Ask Questions The Smart Way"
      <http://www.catb.org/~esr/faqs/smart-questions.html>

Re: Cleaning up the test framework

[Matt Sergeant <ms...@messagelabs.com>]

On 1 Dec 2004, at 15:13, Malte S. Stretz wrote:

>>> So I'd like to keyword some of the tests as "basic" (or whatever
>>> keyword) and only those tests are run per default.  All other tests
>>> would be used by us devs, people who we ask to debug one of their bug
>>> reports aund the BuildBots.
>>
>> No more options, please.  And there's no reason to speed it up for
>> users because users only run "make test" once in a while.
>
> My idea was that per default all tests are run (except everything which
> requires further set-up or can fail easily like net tests or SQL).

See "prove" (now shipped with core perl) and Test::Verbose's tv command 
(which is indispensable to any perl developer IMHO). Also consider 
adding some tests to CVS but not adding them to the MANIFEST, which 
will achieve what you require there.


______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
______________________________________________________________________

Re: Cleaning up the test framework

["Malte S. Stretz" <ms...@gmx.net>]

On Sunday 28 November 2004 02:14 CET Daniel Quinlan wrote:
> "Malte S. Stretz" <ms...@gmx.net> writes:
> > I think I found something to kill some time on:  Cleaning up the test
> > framework.
>
> Kill some time?  You realize there are hundreds of open bugs?  :-)

Yeah, and each of them would require more time than I could spend (I already 
have three bugs or so assigned to me since a few months...)  I just need 
something I can hack on freely from time to time to distract me from other 
projects with fixed time tables and stuff.

> > So I'd like to keyword some of the tests as "basic" (or whatever
> > keyword) and only those tests are run per default.  All other tests
> > would be used by us devs, people who we ask to debug one of their bug
> > reports aund the BuildBots.
>
> No more options, please.  And there's no reason to speed it up for
> users because users only run "make test" once in a while.

My idea was that per default all tests are run (except everything which 
requires further set-up or can fail easily like net tests or SQL).  But 
additionally I'd like to add a bunch of Unit tests over time which, if it 
works out, would test each and every routine we have (ok, exaggerated, but 
it should at least test each public interface with at least ten test cases 
or so).  And those would really add up in time and I already heard many 
people complaining about how long the tests take (... took since I disabled 
them in the default Gentoo ebuild).

> I just don't have much of an issue with the test framework and I don't
> want to complicate things.  Cleaner is always a fine goal, but this
> hasn't been a major pain point relatively speaking IMO.

Cheers,
Malte


-- 
[SGT] Simon G. Tatham: "How to Report Bugs Effectively"
      <http://www.chiark.greenend.org.uk/~sgtatham/bugs.html>
[ESR] Eric S. Raymond: "How To Ask Questions The Smart Way"
      <http://www.catb.org/~esr/faqs/smart-questions.html>

Re: Cleaning up the test framework

[Daniel Quinlan <qu...@pathname.com>]

"Malte S. Stretz" <ms...@gmx.net> writes:

> I think I found something to kill some time on:  Cleaning up the test 
> framework.

Kill some time?  You realize there are hundreds of open bugs?  :-)

> So I'd like to keyword some of the tests as "basic" (or whatever
> keyword) and only those tests are run per default.  All other tests
> would be used by us devs, people who we ask to debug one of their bug
> reports aund the BuildBots.

No more options, please.  And there's no reason to speed it up for
users because users only run "make test" once in a while.

I just don't have much of an issue with the test framework and I don't
want to complicate things.  Cleaner is always a fine goal, but this
hasn't been a major pain point relatively speaking IMO.

Daniel

-- 
Daniel Quinlan
http://www.pathname.com/~quinlan/