You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@perl.apache.org by Stas Bekman <st...@stason.org> on 2004/02/10 00:20:42 UTC
[mp2] splitting the test suite into several smaller ones
I've planned to post it here for discussion first....
Index: tests
===================================================================
######################################
# test suite issues and improvements #
######################################
- As the number of tests grows it takes longer to start the test
suite. Under threaded mpms it can take a very long time on a loaded
machine (10 minutes and more). At the moment we keep on bumping up the
startup timeout, but there is always someone with a slower/more
loaded machine, for whom this timeout is too low, and the tests
suite misleadingly fails to start. So may be we must start thinking
to split the existing test suite into several smaller suites (for
example we already have a separate ModPerl-Registry test suite). If
we do perl/ithreads* and any tests definging +Clone/+Parent tests
should definitely be moved out from the main suite, as they cause
the biggest overhead (due to perl_clone).
--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org
Re: [mp2] splitting the test suite into several smaller ones
Posted by Stas Bekman <st...@stason.org>.
Philippe M. Chiasson wrote:
> On Mon, 2004-02-09 at 15:20 -0800, Stas Bekman wrote:
>
>>I've planned to post it here for discussion first....
>>
>
>
> Well, I think it's an idea that has a lot of value++
>
> At least, provide a way to group tests together in categories and have
> them each configure/start/stop the server.
>
> I am all in favor.
If we do that, we need to merge in a close issue that we discussed earlier,
about providing optional test suites, which won't run on the normal 'make test'.
I can see at least one drawback in the split. We have a good experiance with
the case of the main suite vs. the ModPerl-Registry suite. At the moment, if a
user has problems with the main suite, they never get to see whether the
ModPerl-Registry suite is successful or not. So first they/we get to resolve
the main suite's problems and only then they will run the other test suite and
possibly find problems there too. So instead of reporting them all at once we
get them reported one by one, making the fixing slower. I'm thinking about
changing the 'make test' to run a program which will run all the test suites
no matter what's their outcome and report the total result at the end, i.e. a
failure to run the first test suite doesn't prevent from the second to run.
Another issue is the error_log file. Most users reporting M-R test suite
problems, don't realize that they need to look in
ModPerl-Registry/t/logs/error_log and not in t/logs/error_log. So I thought
that we could again use a driving program which will show the user the right
error log file's contents and perhaps write the bug report on their behalf.
't/SMOKE -bug_report' already does something similar, it can match the error
entry in the error_log for each failing test, by using tell/seek to the right
error_log file. At the moment, the implementation is slow, because we need to
feed each test separately to Test::Harness. I need to check on the status of
Test::Harness::Straps which was planned to take care of running callbacks
before and after each test T::H runs.
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org
Re: [mp2] splitting the test suite into several smaller ones
Posted by "Philippe M. Chiasson" <go...@cpan.org>.
On Mon, 2004-02-09 at 15:20 -0800, Stas Bekman wrote:
> I've planned to post it here for discussion first....
>
Well, I think it's an idea that has a lot of value++
At least, provide a way to group tests together in categories and have
them each configure/start/stop the server.
I am all in favor.
--
--------------------------------------------------------------------------------
Philippe M. Chiasson /gozer\@(cpan|ectoplasm)\.org/ 88C3A5A5 (122FF51B/C634E37B)
http://gozer.ectoplasm.org/ F9BF E0C2 480E 7680 1AE5 3631 CB32 A107 88C3 A5A5
Q: It is impossible to make anything foolproof because fools are so ingenious.
perl -e'$$=\${gozer};{$_=unpack(P7,pack(L,$$));/^JAm_pH\n$/&&print||$$++&&redo}'
Re: [mp2] splitting the test suite into several smaller ones
Posted by Elizabeth Mattijsen <li...@dijkmat.nl>.
At 08:59 -0800 2/10/04, Stas Bekman wrote:
>Vivek Khera wrote:
>>For sure, mod_perl could do with a split like this.
>We have no developer vs. installer separation. All tests are equally
>important and could detect bugs or environment differences. If we
>split the tests, we will still have users run them all (minus the
>special experimental tests that we have discussed earlier).
>
>The problem that we are trying to solve is the growing startup time,
>not the amount of tests being run (which will probably double in
>size with time).
Indeed. The slow startup time of tests is but a symptom of startup
times of production systems in the future. All inherent to the use
of a threaded Perl, I'm afraid. ;-(
Liz
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org
Re: [mp2] splitting the test suite into several smaller ones
Posted by Stas Bekman <st...@stason.org>.
Vivek Khera wrote:
>
> On Feb 9, 2004, at 6:20 PM, Stas Bekman wrote:
>
>> - As the number of tests grows it takes longer to start the test
>> suite. Under threaded mpms it can take a very long time on a loaded
>> machine (10 minutes and more). At the moment we keep on bumping up
>> the
>>
>
> I've noticed this on lots of CPAN modules other than mod_perl. It seems
> to me that there should be two test suites for most complex modules:
> developer tests and installer tests. The developer tests would check
> every corner case of every complex function -- like a full regression
> test might do. The installer tests would check for the basic
> functionality to see if the thing is installed properly, and perhaps
> test some things that may be system-dependent, and be a subset of the
> developer's tests.
>
> For sure, mod_perl could do with a split like this.
We have no developer vs. installer separation. All tests are equally important
and could detect bugs or environment differences. If we split the tests, we
will still have users run them all (minus the special experimental tests that
we have discussed earlier).
The problem that we are trying to solve is the growing startup time, not the
amount of tests being run (which will probably double in size with time).
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org
Re: [mp2] splitting the test suite into several smaller ones
Posted by Vivek Khera <kh...@kcilink.com>.
On Feb 9, 2004, at 6:20 PM, Stas Bekman wrote:
> - As the number of tests grows it takes longer to start the test
> suite. Under threaded mpms it can take a very long time on a loaded
> machine (10 minutes and more). At the moment we keep on bumping up
> the
>
I've noticed this on lots of CPAN modules other than mod_perl. It
seems to me that there should be two test suites for most complex
modules: developer tests and installer tests. The developer tests
would check every corner case of every complex function -- like a full
regression test might do. The installer tests would check for the
basic functionality to see if the thing is installed properly, and
perhaps test some things that may be system-dependent, and be a subset
of the developer's tests.
For sure, mod_perl could do with a split like this.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Vivek Khera, Ph.D. Khera Communications, Inc.
Internet: khera@kciLink.com Rockville, MD +1-301-869-4449 x806
AIM: vivekkhera Y!: vivek_khera http://www.khera.org/~vivek/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org