You are viewing a plain text version of this content. The canonical link for it is here.
Posted to test-dev@httpd.apache.org by Geoffrey Young <ge...@modperlcookbook.org> on 2003/09/18 18:29:28 UTC
[RFC] running tests after the server stops
hi all...
I'm thinking about implementing a mechansim that allows you to run certain
tests after the server shuts down. the rationale behind this is some work
I'm doing on Devel::Profiler::Apache - the profiler generates statistics
when the children die off, so I need to either kill the children or shut
down the server to have some data to test.
now, killing off the children is easy in 1.0 with child_terminate(), but
that isn't portable to 2.0 from what I can tell.
I have wanted an API for restarting the server midstrean for a while now,
mainly to test per-server cleanup logic. for that I currently use
skip ($Apache::TestConfig::WIN32, kill HUP => Apache::TestServer->new->pid);
and that would seem to work as well here, but again with (different)
portability.
however, what I really want is only one child, which would generate only one
file when the server is shutdown. then I want to be able to use that file
in additional tests.
after poking around, I think I can hack together a few different options to
support this:
- a new command-line option, poststop=foo.t, which would specify tests to
add after the server is shutdown.
- treat a directory, t/poststop, as special, feeding the tests in there
directly to the test harness.
or perhaps something else that somebody here can come up with :)
I can't help but feel there would be other uses for this, but maybe I'm wrong.
ideas? comments?
--Geoff
Re: [RFC] running tests after the server stops
Posted by Geoffrey Young <ge...@modperlcookbook.org>.
>
> There are other uses for it already. We have t/apr-ext tests which
> require no running server at all. a magic directory sounds good to me.
> I'd even suggest having two of them - one for pre-start and one for
> post-stop. e.g.:
>
> t/pre-start/
> t/post-stop/
yeah, I was thinking about balancing them out. good idea.
>
> The only issue is with Test::Harness, we will need to somehow feed tests
> to it so it'll take them all as one bunch, while we have to run first
> t/pre-start/ test, followed by normal tests, followed by t/post-stop/
> tests.
that doesn't seem too hard. TestRun::run is pretty simple. the problem
isparsing out the test files and making sure that all those custom options
(subtests, all.t, etc) are right.
>
> though I don't see how does it help you with restart-in-the middle.
> again we could have a magic directory which will force a server restart
> when it hits it, though I'm not sure whether we want it to restart for
> each test in that t/restart/ dir, or just once. However I think that
> making the client side doing the restart is the best option, provided
> that we provide a portable implementation for it.
an API is best there, for certain. but one thing at a time.
I'll get working on the pre and post directory stuff and see what I can hack
together.
--Geoff
Re: [RFC] running tests after the server stops
Posted by Geoffrey Young <ge...@modperlcookbook.org>.
> anyway, so that's how it looks on the outside. on the inside is another
> matter...
ok, I see at least one problem already - if the server doesn't start
(missing APXS or whatnot) the tests continue (and obviously fail).
I've been playing all afternoon with 'Bail out!' but it seems that when
TestTrace prints to STDERR it messes up Test::Harness' foo and the bailout
signal is not handled properly.
so, off I go chasing that one down... will report back with a (bigger,
possibly massive) patch soon.
ideas/feedback welcome.
--Geoff
Re: [RFC] running tests after the server stops
Posted by Geoffrey Young <ge...@modperlcookbook.org>.
ok, here we go :)
I have a preliminary implementation. what happens now is that two files are
autogenerated if they don't already exist:
t/server-start.t
t/server-stop.t
they are used to control the starting and stopping of the server. so, the
output of make test looks like
# writing file: server-start.t
# writing file: server-stop.t
pre-start/ping....ok
server-start......*** server localhost.localdomain:8529 started
server-start......ok
ping..............ok
request...........ok
server-stop.......*** server localhost.localdomain:8529 shutdown
server-stop.......ok
post-stop/ping....ok
All tests successful.
Files=6, Tests=20, 3 wallclock secs ( 2.03 cusr + 0.16 csys = 2.19 CPU)
# removing file: server-start.t
# removing file: server-stop.t
another condition of server-control file autogeneration is that there be
real server tests to run. this makes it trivial to run pre-start or
post-stop tests by themselves:
$ t/TEST t/pre-start/
*** setting ulimit to allow core files
ulimit -c unlimited; t/TEST 't/pre-start/'
pre-start/ping....ok
All tests successful.
Files=1, Tests=3, 1 wallclock secs ( 0.28 cusr + 0.03 csys = 0.31 CPU)
anyway, so that's how it looks on the outside. on the inside is another
matter...
as I mentioned, I'm really not too sure about mucking with the core engine
like this (although I did take some comfort in the fact that not only did
the changes work for all of my personal A-T modules, but I didn't need any
tweaks to get the mod_perl 2.0 suite running successfully). I might prefer
to create a subclass of TestRun and TestHarness that works this way and
leave it up to people to choose which they use from TEST.PL - it depends on
how much the people here like the implementation (after any suggested
refinements and refactoring are applied, of course :)
so, give it a whirl and let me know what you think.
--Geoff
Re: [RFC] running tests after the server stops
Posted by Geoffrey Young <ge...@modperlcookbook.org>.
> OK, here is an idea how to solve it. instead of starting and stopping
> the server by t/TEST delegate it to a specially designed tests. So the
> last test in t/pre-start/ will start the server and the first test in
> t/post-stop/ will stop the server. Both can be autogenerated. Of course
> this will mess up with the ability to run a single test with t/TEST, but
> we can instruct t/TEST to start the server if it doesn't see
> t/pre-start/ in the list of tests, and stop it if it doesn't see
> t/post-stop/ in the list of tests.
that's the route that I have started in on (in varying forms). it's sorta
working out, but it doesn't really seem to be a clean route - among other
things I need to freeze all the options and essentially build a new (fake)
TestRun object within the start and stop tests. we'll see how it goes, but
I think it will work out better as a subclass of Apache::TestHarness rather
than altering the underlying code.
anyway, I should have something tomorrow.
--Geoff
Re: [RFC] running tests after the server stops
Posted by Stas Bekman <st...@stason.org>.
Geoffrey Young wrote:
>
>>
>> The only issue is with Test::Harness, we will need to somehow feed
>> tests to it so it'll take them all as one bunch, while we have to run
>> first t/pre-start/ test, followed by normal tests, followed by
>> t/post-stop/ tests.
>
>
> oh, I see the issue now.
>
> if we call TestRun::run_tests multiple times, each set of tests is sent
> to Test::Harness as a batch. so, calling it pre and post run results in
> three sets of reports.
>
> I have the start of an implementation now (working via t/TEST) but am
> hesitant to work on it more unless we think that three reports is ok for
> this feature. there will also have to be some trickery involved - if
> one set returns failure the remaining sets don't run - but I'm not sure
> where to put it.
It's actually good, it'd be bad if it'd continue running and users seeing
success report from the last batch and missing failures from previous reports.
e.g. in modperl-2.0, we already have two separate reports, the core and the
registry, and registry tests won't run if the core test suite fails.
> at any rate, the remaining issues are probably trivial if we can accept
> a change in output. if not, I'm not sure how this would work.
OK, here is an idea how to solve it. instead of starting and stopping the
server by t/TEST delegate it to a specially designed tests. So the last test
in t/pre-start/ will start the server and the first test in t/post-stop/ will
stop the server. Both can be autogenerated. Of course this will mess up with
the ability to run a single test with t/TEST, but we can instruct t/TEST to
start the server if it doesn't see t/pre-start/ in the list of tests, and stop
it if it doesn't see t/post-stop/ in the list of tests.
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
Re: [RFC] running tests after the server stops
Posted by Geoffrey Young <ge...@modperlcookbook.org>.
>
> The only issue is with Test::Harness, we will need to somehow feed tests
> to it so it'll take them all as one bunch, while we have to run first
> t/pre-start/ test, followed by normal tests, followed by t/post-stop/
> tests.
oh, I see the issue now.
if we call TestRun::run_tests multiple times, each set of tests is sent to
Test::Harness as a batch. so, calling it pre and post run results in three
sets of reports.
I have the start of an implementation now (working via t/TEST) but am
hesitant to work on it more unless we think that three reports is ok for
this feature. there will also have to be some trickery involved - if one
set returns failure the remaining sets don't run - but I'm not sure where to
put it.
at any rate, the remaining issues are probably trivial if we can accept a
change in output. if not, I'm not sure how this would work.
--Geoff
Re: [RFC] running tests after the server stops
Posted by Stas Bekman <st...@stason.org>.
Geoffrey Young wrote:
> hi all...
>
> I'm thinking about implementing a mechansim that allows you to run
> certain tests after the server shuts down. the rationale behind this is
> some work I'm doing on Devel::Profiler::Apache - the profiler generates
> statistics when the children die off, so I need to either kill the
> children or shut down the server to have some data to test.
>
> now, killing off the children is easy in 1.0 with child_terminate(), but
> that isn't portable to 2.0 from what I can tell.
>
> I have wanted an API for restarting the server midstrean for a while
> now, mainly to test per-server cleanup logic. for that I currently use
>
> skip ($Apache::TestConfig::WIN32, kill HUP =>
> Apache::TestServer->new->pid);
>
> and that would seem to work as well here, but again with (different)
> portability.
>
> however, what I really want is only one child, which would generate only
> one file when the server is shutdown. then I want to be able to use
> that file in additional tests.
>
> after poking around, I think I can hack together a few different options
> to support this:
>
> - a new command-line option, poststop=foo.t, which would specify tests
> to add after the server is shutdown.
>
> - treat a directory, t/poststop, as special, feeding the tests in
> there directly to the test harness.
>
> or perhaps something else that somebody here can come up with :)
>
> I can't help but feel there would be other uses for this, but maybe I'm
> wrong.
There are other uses for it already. We have t/apr-ext tests which require no
running server at all. a magic directory sounds good to me. I'd even suggest
having two of them - one for pre-start and one for post-stop. e.g.:
t/pre-start/
t/post-stop/
The only issue is with Test::Harness, we will need to somehow feed tests to it
so it'll take them all as one bunch, while we have to run first t/pre-start/
test, followed by normal tests, followed by t/post-stop/ tests.
though I don't see how does it help you with restart-in-the middle. again we
could have a magic directory which will force a server restart when it hits
it, though I'm not sure whether we want it to restart for each test in that
t/restart/ dir, or just once. However I think that making the client side
doing the restart is the best option, provided that we provide a portable
implementation for it.
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com