You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@perl.apache.org by brian moseley <bc...@maz.org> on 2001/06/24 13:40:44 UTC
test order
i'm using Apache-Test as reference for my (greatly
simplified) wombat test harness. i can't figure out how you
ensure that tests are run in a specific order, or even if
you are. certainly you're not in the case where you build a
list of tests using finddepth. have you addressed this issue
at all?
oh wait, i guess i could use a series of dependency targets
in the Makefile:
run_tests: test_1 test_2 test_3
test_1:
$(FULLPERL) -I$(INST_ARCHLIB) -I$(INST_LIB) t/TEST dir1
test_2:
$(FULLPERL) -I$(INST_ARCHLIB) -I$(INST_LIB) t/TEST dir2
test_3:
$(FULLPERL) -I$(INST_ARCHLIB) -I$(INST_LIB) t/TEST dir3
can anybody think of a better way to do it, other than using
dir and file name prefixes? brainstorm time!
btw, Apache-Test is really cool! i'm trying to decide if it
would be worth my time to figure out how to generalize it so
that you don't have to use it with httpd/mod_perl at all.
for wombat, i'd ideally be able to:
1) run a set of white box tests that instantiate each class
and test each method
2) run a set of black box tests for each connector that fire
up a server and make client requests. so if the user has
mod_perl installed, run a httpd much like you're doing now;
if he has fastcgi installed, run httpd and fastcgi; if he
has lwp installed, run a standalone wombatd; etc.
obviously each set of tests in #1 and #2 has its own setup
and teardown requirements, so Apache-Test would have to be
generalized quite a bit.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org
Re: test order
Posted by brian moseley <bc...@maz.org>.
On Sun, 24 Jun 2001, Stas Bekman wrote:
> I understand. Another option would be to always sort the
> tests unless you have specified their order on the
> command line.
>
> o if you specify none -- everything is sorted.
> o if you specify only dirs -- evething within dirs is sorted, dirs order
> is preserved
> o if you specify file names they aren't sorted.
> o if you specify random, you get it random.
>
> I like this option even better. You?
looks good to me!
> also you could always do:
>
> $ ./t/TEST `echo t/*/*.t | sort`
>
> even in Makefile
yeah, i just hacked my TEST script to sort. my stuff is a
lot simpler than the Apache-Test stuff, and i am ok with
always sorting. i think this stuff will be helpful for
people using Apache-Test tho.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org
Re: test order
Posted by Stas Bekman <st...@stason.org>.
On Sun, 24 Jun 2001, brian moseley wrote:
> On Sun, 24 Jun 2001, Stas Bekman wrote:
>
> > ok will add 'sort'. which will presort tests before
> > applying any other algorithm.
> >
> > should 'sort' be non-exclusive with 'rotate|repeat'? (it
> > won't apply to 'random' of course)
> >
> > but it gets hairy then: --run-tests[=[N][:][order][:][sort]] ?
> >
> > I'd rather have --run-tests --test_times=N --test_order=ORDER
> > --[no]test_sort? Doug?
>
> no opinion on this. whatever y'all think is best :)
>
> > shouldn't your tests be independent from each other? if
> > they are why not to put them in one test? Or have one
> > test calling a few others. what you are saying doesn't
> > sound well, but you know your requirements better :)
>
> the tests are independent, but it doesn't make sense to test
> class B if class A doesn't work, or even compile. for black
> box tests, i agree you should be able to run them in any
> order, but i definitely need sorting for lower level tests.
>
> thanks stas.
I understand. Another option would be to always sort the tests unless you
have specified their order on the command line.
o if you specify none -- everything is sorted.
o if you specify only dirs -- evething within dirs is sorted, dirs order
is preserved
o if you specify file names they aren't sorted.
o if you specify random, you get it random.
I like this option even better. You?
also you could always do:
$ ./t/TEST `echo t/*/*.t | sort`
even in Makefile
_____________________________________________________________________
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:stas@stason.org http://apachetoday.com http://eXtropia.com/
http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org
Re: test order
Posted by brian moseley <bc...@maz.org>.
On Sun, 24 Jun 2001, Stas Bekman wrote:
> ok will add 'sort'. which will presort tests before
> applying any other algorithm.
>
> should 'sort' be non-exclusive with 'rotate|repeat'? (it
> won't apply to 'random' of course)
>
> but it gets hairy then: --run-tests[=[N][:][order][:][sort]] ?
>
> I'd rather have --run-tests --test_times=N --test_order=ORDER
> --[no]test_sort? Doug?
no opinion on this. whatever y'all think is best :)
> shouldn't your tests be independent from each other? if
> they are why not to put them in one test? Or have one
> test calling a few others. what you are saying doesn't
> sound well, but you know your requirements better :)
the tests are independent, but it doesn't make sense to test
class B if class A doesn't work, or even compile. for black
box tests, i agree you should be able to run them in any
order, but i definitely need sorting for lower level tests.
thanks stas.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org
Re: test order
Posted by Doug MacEachern <do...@covalent.net>.
On Tue, 26 Jun 2001, Stas Bekman wrote:
> what do you think about this para about default sorting?
>
> > I understand. Another option would be to always sort the
> > tests unless you have specified their order on the
> > command line.
> >
> > o if you specify none -- everything is sorted.
> > o if you specify only dirs -- evething within dirs is sorted, dirs order
> > is preserved
> > o if you specify file names they aren't sorted.
> > o if you specify random, you get it random.
> >
> > I like this option even better. You?
sounds good to me.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org
Re: test order
Posted by Stas Bekman <st...@stason.org>.
On Mon, 25 Jun 2001, Doug MacEachern wrote:
> On Sun, 24 Jun 2001, Stas Bekman wrote:
>
> > should 'sort' be non-exclusive with 'rotate|repeat'? (it won't apply to
> > 'random' of course)
> >
> > but it gets hairy then: --run-tests[=[N][:][order][:][sort]] ?
> >
> > I'd rather have --run-tests --test_times=N --test_order=ORDER
> > --[no]test_sort? Doug?
>
> how about -times=N -order=ORDER
let it be:
--run-tests -times=N -order=ORDER
what do you think about this para about default sorting?
> I understand. Another option would be to always sort the
> tests unless you have specified their order on the
> command line.
>
> o if you specify none -- everything is sorted.
> o if you specify only dirs -- evething within dirs is sorted, dirs order
> is preserved
> o if you specify file names they aren't sorted.
> o if you specify random, you get it random.
>
> I like this option even better. You?
_____________________________________________________________________
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:stas@stason.org http://apachetoday.com http://eXtropia.com/
http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org
Re: test order
Posted by Doug MacEachern <do...@covalent.net>.
On Sun, 24 Jun 2001, Stas Bekman wrote:
> should 'sort' be non-exclusive with 'rotate|repeat'? (it won't apply to
> 'random' of course)
>
> but it gets hairy then: --run-tests[=[N][:][order][:][sort]] ?
>
> I'd rather have --run-tests --test_times=N --test_order=ORDER
> --[no]test_sort? Doug?
how about -times=N -order=ORDER
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org
Re: test order
Posted by Stas Bekman <st...@stason.org>.
On Sun, 24 Jun 2001, brian moseley wrote:
> On Sun, 24 Jun 2001, Stas Bekman wrote:
>
> > the only way to ensure identical order from run to run,
> > across different systems (and changes in file inodes
> > ordering) is to explicitly list the tests in the wanted
> > order. otherwise it's up to the inodes order under ./t.
> >
> > Since I mess with tests anyway now (see my other post
> > from a few minutes ago), do you want to add yet another
> > mode 'sort'?
>
> makes sense. that would allow me to at least use naming
> conventions to ensure run order.
ok will add 'sort'. which will presort tests before applying any other
algorithm.
should 'sort' be non-exclusive with 'rotate|repeat'? (it won't apply to
'random' of course)
but it gets hairy then: --run-tests[=[N][:][order][:][sort]] ?
I'd rather have --run-tests --test_times=N --test_order=ORDER
--[no]test_sort? Doug?
> > But, hey, look at the random mode I've just added. Why
> > would you want to ensure the same order in tests? That's
> > not the way QA works :)
>
> dependencies. if class B extends or uses class A, i want to
> run the tests for class A before those for class B.
shouldn't your tests be independent from each other? if they are why not
to put them in one test? Or have one test calling a few others. what you
are saying doesn't sound well, but you know your requirements better :)
_____________________________________________________________________
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:stas@stason.org http://apachetoday.com http://eXtropia.com/
http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org
Re: test order
Posted by brian moseley <bc...@maz.org>.
On Sun, 24 Jun 2001, Stas Bekman wrote:
> the only way to ensure identical order from run to run,
> across different systems (and changes in file inodes
> ordering) is to explicitly list the tests in the wanted
> order. otherwise it's up to the inodes order under ./t.
>
> Since I mess with tests anyway now (see my other post
> from a few minutes ago), do you want to add yet another
> mode 'sort'?
makes sense. that would allow me to at least use naming
conventions to ensure run order.
> But, hey, look at the random mode I've just added. Why
> would you want to ensure the same order in tests? That's
> not the way QA works :)
dependencies. if class B extends or uses class A, i want to
run the tests for class A before those for class B.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org
Re: test order
Posted by Stas Bekman <st...@stason.org>.
On Sun, 24 Jun 2001, brian moseley wrote:
>
> i'm using Apache-Test as reference for my (greatly
> simplified) wombat test harness. i can't figure out how you
> ensure that tests are run in a specific order, or even if
> you are. certainly you're not in the case where you build a
> list of tests using finddepth. have you addressed this issue
> at all?
> oh wait, i guess i could use a series of dependency targets
> in the Makefile:
>
> run_tests: test_1 test_2 test_3
>
> test_1:
> $(FULLPERL) -I$(INST_ARCHLIB) -I$(INST_LIB) t/TEST dir1
>
> test_2:
> $(FULLPERL) -I$(INST_ARCHLIB) -I$(INST_LIB) t/TEST dir2
>
> test_3:
> $(FULLPERL) -I$(INST_ARCHLIB) -I$(INST_LIB) t/TEST dir3
>
> can anybody think of a better way to do it, other than using
> dir and file name prefixes? brainstorm time!
the only way to ensure identical order from run to run, across different
systems (and changes in file inodes ordering) is to explicitly list the
tests in the wanted order. otherwise it's up to the inodes order under
./t.
Since I mess with tests anyway now (see my other post from a few minutes
ago), do you want to add yet another mode 'sort'?
But, hey, look at the random mode I've just added. Why would you want to
ensure the same order in tests? That's not the way QA works :)
_____________________________________________________________________
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:stas@stason.org http://apachetoday.com http://eXtropia.com/
http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org