Re: Cleaning up the test framework

[Daniel Quinlan <qu...@pathname.com>]

I generally agree what with Justin said, I just would like to see the
work broken up a bit more (done incrementally) *and* I don't want to see
new options because I don't think they're needed.  The original proposal
had a bit of the sound of creating a monster patch that is hard to test
out and get into the tree.

It would be fantastic to make tests easier to write.  Faster would be
nice, but only via improving the test code, not worrying about options
IMO, we start and stop spamd more often than needed.

Daniel

-- 
Daniel Quinlan
http://www.pathname.com/~quinlan/

Re: Cleaning up the test framework

["Malte S. Stretz" <ms...@gmx.net>]

On Wednesday 01 December 2004 21:54 CET Daniel Quinlan wrote:
> "Malte S. Stretz" <ms...@gmx.net> writes:
> > 1.  It has to work also if only single tests are run, so can't rely on
> > a 99_stop_spamd.t to clean up.
> >
> > 2.  It would change the test semantics.  Currently each test is run on
> > a clean spamd.  If the spamd was persistent, a test could fail because
> > of some bug hidden somewhere else.  Not that finding ng these bugs was
> > a bad thing but it would make the tests a bit non-deterministic.
>
> Here's an optimization to greatly reduce unnecessary start/stop.
>
> After running each test, leave spamd running and store the flags and
> configuration that were used to run spamd.  In the next test:
>
>   - if the flags were changed, stop and restart spamd
>   - else if the configuration was changed, kill -HUP spamd

With the latter we have to watch out for systems which don't support HUP 
(currently tested by spamd_hup.t).  I guess a better idea would be 
completely restart the deamon in those cases.  If the tests are in an 
optimized order, the reloading of spamd could even kept at a minimum.  

> I think it would be good, not just for performance reasons, to reuse
> spamd processes when possible, because it might turn up somewhat more
> complex issues.

But there's still the first issue.  What about one of this:

1.  The complex solution:  Parallel to spamd an extra test daemon is 
started.  Each running test creates some pid file.  The test daemon checks 
regularly if that process is still runnig, if not it kills the daemon.

2.  The (maybe) easier solution:  I try to find a way to feed the names of 
the tests run to the scripts themselves (with some more Makefile.PL 
hacking) and the last one tears down spamd.

3.  The lazy solution:  spamd just keeps running, a 99_stop_spamd.t tears it 
down.  If somebody wants to run single tests manually, they have to take 
care of spamd themselves.

4.  We start and stop spamd as we already do.


My "votes" in ascending order:  I'm totally opposed (as in < 0) to 3 and 1 
sounds a bit complex to me (~ +-0), I would keep with 4 instead.  If 2 is 
possible, thats IMO the way to go.

Cheers,
Malte

-- 
[SGT] Simon G. Tatham: "How to Report Bugs Effectively"
      <http://www.chiark.greenend.org.uk/~sgtatham/bugs.html>
[ESR] Eric S. Raymond: "How To Ask Questions The Smart Way"
      <http://www.catb.org/~esr/faqs/smart-questions.html>

Re: Cleaning up the test framework

[Michael Parker <pa...@pobox.com>]

On Wed, Dec 01, 2004 at 06:16:20PM +0100, Malte S. Stretz wrote:
> 
> 2.  It would change the test semantics.  Currently each test is run on a 
> clean spamd.  If the spamd was persistent, a test could fail because of 
> some bug hidden somewhere else.  Not that finding ng these bugs was a bad 
> thing but it would make the tests a bit non-deterministic.
> 

Not to mention that most of the tests are testing different startup
options for spamd (socket, ldap, port, allow_user_rules, syslog, etc
etc).

There may be some tests that can be grouped a little better and that
will help.

Michael

Re: Cleaning up the test framework

[Daniel Quinlan <qu...@pathname.com>]

"Malte S. Stretz" <ms...@gmx.net> writes:

> 1.  It has to work also if only single tests are run, so can't rely on a 
> 99_stop_spamd.t to clean up.
> 
> 2.  It would change the test semantics.  Currently each test is run on a 
> clean spamd.  If the spamd was persistent, a test could fail because of 
> some bug hidden somewhere else.  Not that finding ng these bugs was a bad 
> thing but it would make the tests a bit non-deterministic.

Here's an optimization to greatly reduce unnecessary start/stop.

After running each test, leave spamd running and store the flags and
configuration that were used to run spamd.  In the next test:

  - if the flags were changed, stop and restart spamd
  - else if the configuration was changed, kill -HUP spamd

I think it would be good, not just for performance reasons, to reuse
spamd processes when possible, because it might turn up somewhat more
complex issues.

Daniel

-- 
Daniel Quinlan
http://www.pathname.com/~quinlan/

Re: Cleaning up the test framework

["Malte S. Stretz" <ms...@gmx.net>]

On Wednesday 01 December 2004 16:45 CET I wrote:
>[...]
> > It would be fantastic to make tests easier to write.  Faster would be
> > nice, but only via improving the test code, not worrying about options
> > IMO, we start and stop spamd more often than needed.
>
> Most people should run find with the default set of options, the most
> used one will probably be "+all".  The default user will just skip that
> load of in-depth long-running Unit tests I hope we will have one day :)

Oh, forgot my second, more important paragraph:  The spamd-start-stop 
overhead is indeed bad and I tried to find a solution for that.  But:

1.  It has to work also if only single tests are run, so can't rely on a 
99_stop_spamd.t to clean up.

2.  It would change the test semantics.  Currently each test is run on a 
clean spamd.  If the spamd was persistent, a test could fail because of 
some bug hidden somewhere else.  Not that finding ng these bugs was a bad 
thing but it would make the tests a bit non-deterministic.

Cheers,
Malte

-- 
[SGT] Simon G. Tatham: "How to Report Bugs Effectively"
      <http://www.chiark.greenend.org.uk/~sgtatham/bugs.html>
[ESR] Eric S. Raymond: "How To Ask Questions The Smart Way"
      <http://www.catb.org/~esr/faqs/smart-questions.html>

Re: Cleaning up the test framework

["Malte S. Stretz" <ms...@gmx.net>]

On Wednesday 01 December 2004 02:27 CET Daniel Quinlan wrote:
> I generally agree what with Justin said, I just would like to see the
> work broken up a bit more (done incrementally) *and* I don't want to see
> new options because I don't think they're needed.  The original proposal
> had a bit of the sound of creating a monster patch that is hard to test
> out and get into the tree.

No, not at all.  I plan to refactor it bit by bit, committing every small 
(or not so small if I have to do some s/// on each file) chunk which is of 
course tested first.  As I said, I don't have time to work on it really 
regularly and often and I learned that if I keep large chunks of unfinished 
code on my disks those tend to get out-of-date quickly and hard to update 
when I continue working on stuff.

> It would be fantastic to make tests easier to write.  Faster would be
> nice, but only via improving the test code, not worrying about options
> IMO, we start and stop spamd more often than needed.

Most people should run find with the default set of options, the most used 
one will probably be "+all".  The default user will just skip that load of 
in-depth long-running Unit tests I hope we will have one day :)

Cheers,
Malte

-- 
[SGT] Simon G. Tatham: "How to Report Bugs Effectively"
      <http://www.chiark.greenend.org.uk/~sgtatham/bugs.html>
[ESR] Eric S. Raymond: "How To Ask Questions The Smart Way"
      <http://www.catb.org/~esr/faqs/smart-questions.html>