You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@perl.apache.org by Fred Moyer <fr...@redhotpenguin.com> on 2011/01/23 19:46:59 UTC
Re: test framework and latest Fedora
On Fri, Jan 21, 2011 at 1:29 AM, Joe Orton <jo...@redhat.com> wrote:
> CC'ing test-dev@.
>
> On Thu, Jan 20, 2011 at 12:00:41PM -0500, Jim Jagielski wrote:
>> On the latest Fedora, -times=X no longer works:
>>
>> t/modules/rewrite.t .. ok
>> t/modules/rewrite.t .. ok
>> You already have a parser for (t/modules/rewrite.t). Perhaps you have run the same test twice. at /usr/share/perl5/TAP/Harness.pm line 521
>>
>> CPAN itself is up to date... or is that the problem?
That looks like the problem. When TAP/Parser/Aggregator.pm was added
in March 2007, it croaked if you tried to reuse the test description.
As the primary Apache-Test maintainer right now, I am tempted to
deprecate this feature in favor of making the codebase simple and
easier to maintain in the long run. Updating this feature to work
with TAP-Harness is an unknown amount of work. And running tests
multiple times is more fitting of something like t/SMOKE
(Apache::TestSmoke) rather than t/TEST. I can get in there and muck
with the internals of Apache::TestRun (and A-TR-PHP, Perl, etc) to
make it behave properly, but my tuits are better spent on insuring
quality in other parts of this module, and also releasing mod_perl
2.0.5.
I welcome feedback here on desired approaches. What is 'times=' being
used for? Can you tryout t/SMOKE instead?
> Seeing the same here with Test::Harness 3.17, not that I've used that
> feature before.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org
Re: [PATCH] New default sort order in Apache-Test-1.36 (WAS: test
framework and latest Fedora)
Posted by Rainer Jung <ra...@kippdata.de>.
Hi Fred,
On 26.05.2011 02:15, Fred Moyer wrote:
> +1
>
> Do you have commit privileges on Apache-Test?
Surprise, yes :)
Committed as r1127812.
Regards,
Rainer
> On Fri, May 20, 2011 at 1:51 PM, Rainer Jung <ra...@kippdata.de> wrote:
>> I noticed, that the SMOKE change in 1.36 also introduced a new sorting
>> behaviour. In the change the test suite was running by default using the
>> repeat order, that means the tests were ordered alpabetically.
>>
>> Now the default order is random.
>>
>> Furthermore the -order=rotate (A B C A B C) is gone, and -order=repeat
>> now behaves like previously -order=rotate (it seems it previously was A
>> A B B C C).
>>
>> I suggest:
>>
>> - keep behaviour and name of repeat as it is now
>> - don't try to reimplement old repeat, because it might be tricky in
>> combination with the new way iterations are done (SMOKE)
>> - switch back to the new *repeat* (=old rotate) as the default for
>> non-SMOKE tests, i.e. run the tests by default in sorted order and only
>> do random if -order=random is explicitely given.
>> - stay with random as default for SMOKE
>>
>> The following one line change should do that:
>>
>> Index: Apache-Test/lib/Apache/TestSort.pm
>> ===================================================================
>> --- Apache-Test/lib/Apache/TestSort.pm (revision 1125303)
>> +++ Apache-Test/lib/Apache/TestSort.pm (working copy)
>> @@ -54,7 +54,7 @@
>> sub run {
>> my($self, $list, $args) = @_;
>>
>> - my $order = $args->{order} || 'random';
>> + my $order = $args->{order} || 'repeat';
>> if ($order =~ /^\d+$/) {
>> #dont want an explicit -seed option but env var can be a pain
>> #so if -order is number assume it is the random seed
>>
>>
>> Furthermore the READMEs and Changelog should get updated:
>>
>> ´Index: README
>> ===================================================================
>> --- README (revision 1125303)
>> +++ README (working copy)
>> @@ -77,18 +77,20 @@
>>
>> It's also possible that a test will pass when it's run after a
>> particular test, but if moved to run after a different state it may
>> -fail. For this reason it's important to ran the randomization smoking
>> -testing with -order=random option.
>> +fail. For this reason by default the tests run in random order.
>>
>> Since it's important to be able to reproduce the problem with the
>> random testing, whenever -order=random is used, the used seed is
>> printed to STDERR. Which can be then fed into the future tests with:
>> via APACHE_TEST_SEED environment variable.
>>
>> +By adding the option -order=repeat, the tests will be run in
>> +alphabetical order.
>> +
>> Combining these two important smoke testing techiques, one can run
>> tests with:
>>
>> - t/SMOKE -times=N -order=(repeat|rotate|random)
>> + t/SMOKE -times=N -order=(repeat|random)
>>
>> For example, to run the mod_rewrite tests 5 times, one would:
>>
>> @@ -98,8 +100,7 @@
>> modes:
>>
>> - randomize all tests
>> -- repeat the same test N times in sequence
>> -- rotate the whole tests suite N times
>> +- repeat the whole tests suite N times
>>
>> For configuration options and default settings run:
>>
>> Index: Apache-Test/README
>> ===================================================================
>> --- Apache-Test/README (revision 1125303)
>> +++ Apache-Test/README (working copy)
>> @@ -203,18 +203,15 @@
>>
>> run all tests 10 times in a random order (the seed is autogenerated
>> and reported)
>> -% t/TEST -times=10 -order=random
>> +% t/SMOKE -times=10 -order=random
>>
>> run all tests 10 times in a random order using the seed obtained from
>> the previous random run (e.g. 2352211):
>> -% t/TEST -times=10 -order=2352211
>> +% t/SMOKE -times=10 -order=2352211
>>
>> -rotate all tests 10 times (a, b, c, a, b, c)
>> -% t/TEST -times=10 -order=rotate
>> +repeat all tests 10 times (a, b, c, a, b, c)
>> +% t/SMOKE -times=10 -order=repeat
>>
>> -repeat all tests 10 times (a, a, b, b, c, c)
>> -% t/TEST -times=10 -order=repeat
>> -
>> When certain tests fail when running with -times option, you want to
>> find out the minimal sequence of tests that lead to the
>> failure. Apache::TestSmoke helps to ease this task, simply run:
>> Index: Apache-Test/Changes
>> ===================================================================
>> --- Apache-Test/Changes (revision 1125303)
>> +++ Apache-Test/Changes (working copy)
>> @@ -8,6 +8,9 @@
>>
>> =item 1.37-dev
>>
>> +Run t/TEST tests by default in alphabetical order and
>> +only t/SMOKE tests by default in random order.
>> +
>> Add t_file_watch_for to Apache::TestUtil [Torsten Foertsch]
>>
>> Add $boolean parameter to Apache::TestHandler::ok and
>> Apache::TestHandler::ok1
>>
>> Regards,
>>
>> Rainer
Re: [PATCH] New default sort order in Apache-Test-1.36 (WAS: test
framework and latest Fedora)
Posted by Fred Moyer <fr...@redhotpenguin.com>.
+1
Do you have commit privileges on Apache-Test?
On Fri, May 20, 2011 at 1:51 PM, Rainer Jung <ra...@kippdata.de> wrote:
> I noticed, that the SMOKE change in 1.36 also introduced a new sorting
> behaviour. In the change the test suite was running by default using the
> repeat order, that means the tests were ordered alpabetically.
>
> Now the default order is random.
>
> Furthermore the -order=rotate (A B C A B C) is gone, and -order=repeat
> now behaves like previously -order=rotate (it seems it previously was A
> A B B C C).
>
> I suggest:
>
> - keep behaviour and name of repeat as it is now
> - don't try to reimplement old repeat, because it might be tricky in
> combination with the new way iterations are done (SMOKE)
> - switch back to the new *repeat* (=old rotate) as the default for
> non-SMOKE tests, i.e. run the tests by default in sorted order and only
> do random if -order=random is explicitely given.
> - stay with random as default for SMOKE
>
> The following one line change should do that:
>
> Index: Apache-Test/lib/Apache/TestSort.pm
> ===================================================================
> --- Apache-Test/lib/Apache/TestSort.pm (revision 1125303)
> +++ Apache-Test/lib/Apache/TestSort.pm (working copy)
> @@ -54,7 +54,7 @@
> sub run {
> my($self, $list, $args) = @_;
>
> - my $order = $args->{order} || 'random';
> + my $order = $args->{order} || 'repeat';
> if ($order =~ /^\d+$/) {
> #dont want an explicit -seed option but env var can be a pain
> #so if -order is number assume it is the random seed
>
>
> Furthermore the READMEs and Changelog should get updated:
>
> ´Index: README
> ===================================================================
> --- README (revision 1125303)
> +++ README (working copy)
> @@ -77,18 +77,20 @@
>
> It's also possible that a test will pass when it's run after a
> particular test, but if moved to run after a different state it may
> -fail. For this reason it's important to ran the randomization smoking
> -testing with -order=random option.
> +fail. For this reason by default the tests run in random order.
>
> Since it's important to be able to reproduce the problem with the
> random testing, whenever -order=random is used, the used seed is
> printed to STDERR. Which can be then fed into the future tests with:
> via APACHE_TEST_SEED environment variable.
>
> +By adding the option -order=repeat, the tests will be run in
> +alphabetical order.
> +
> Combining these two important smoke testing techiques, one can run
> tests with:
>
> - t/SMOKE -times=N -order=(repeat|rotate|random)
> + t/SMOKE -times=N -order=(repeat|random)
>
> For example, to run the mod_rewrite tests 5 times, one would:
>
> @@ -98,8 +100,7 @@
> modes:
>
> - randomize all tests
> -- repeat the same test N times in sequence
> -- rotate the whole tests suite N times
> +- repeat the whole tests suite N times
>
> For configuration options and default settings run:
>
> Index: Apache-Test/README
> ===================================================================
> --- Apache-Test/README (revision 1125303)
> +++ Apache-Test/README (working copy)
> @@ -203,18 +203,15 @@
>
> run all tests 10 times in a random order (the seed is autogenerated
> and reported)
> -% t/TEST -times=10 -order=random
> +% t/SMOKE -times=10 -order=random
>
> run all tests 10 times in a random order using the seed obtained from
> the previous random run (e.g. 2352211):
> -% t/TEST -times=10 -order=2352211
> +% t/SMOKE -times=10 -order=2352211
>
> -rotate all tests 10 times (a, b, c, a, b, c)
> -% t/TEST -times=10 -order=rotate
> +repeat all tests 10 times (a, b, c, a, b, c)
> +% t/SMOKE -times=10 -order=repeat
>
> -repeat all tests 10 times (a, a, b, b, c, c)
> -% t/TEST -times=10 -order=repeat
> -
> When certain tests fail when running with -times option, you want to
> find out the minimal sequence of tests that lead to the
> failure. Apache::TestSmoke helps to ease this task, simply run:
> Index: Apache-Test/Changes
> ===================================================================
> --- Apache-Test/Changes (revision 1125303)
> +++ Apache-Test/Changes (working copy)
> @@ -8,6 +8,9 @@
>
> =item 1.37-dev
>
> +Run t/TEST tests by default in alphabetical order and
> +only t/SMOKE tests by default in random order.
> +
> Add t_file_watch_for to Apache::TestUtil [Torsten Foertsch]
>
> Add $boolean parameter to Apache::TestHandler::ok and
> Apache::TestHandler::ok1
>
> Regards,
>
> Rainer
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org
Re: [PATCH] New default sort order in Apache-Test-1.36 (WAS: test
framework and latest Fedora)
Posted by Fred Moyer <fr...@redhotpenguin.com>.
+1
Do you have commit privileges on Apache-Test?
On Fri, May 20, 2011 at 1:51 PM, Rainer Jung <ra...@kippdata.de> wrote:
> I noticed, that the SMOKE change in 1.36 also introduced a new sorting
> behaviour. In the change the test suite was running by default using the
> repeat order, that means the tests were ordered alpabetically.
>
> Now the default order is random.
>
> Furthermore the -order=rotate (A B C A B C) is gone, and -order=repeat
> now behaves like previously -order=rotate (it seems it previously was A
> A B B C C).
>
> I suggest:
>
> - keep behaviour and name of repeat as it is now
> - don't try to reimplement old repeat, because it might be tricky in
> combination with the new way iterations are done (SMOKE)
> - switch back to the new *repeat* (=old rotate) as the default for
> non-SMOKE tests, i.e. run the tests by default in sorted order and only
> do random if -order=random is explicitely given.
> - stay with random as default for SMOKE
>
> The following one line change should do that:
>
> Index: Apache-Test/lib/Apache/TestSort.pm
> ===================================================================
> --- Apache-Test/lib/Apache/TestSort.pm (revision 1125303)
> +++ Apache-Test/lib/Apache/TestSort.pm (working copy)
> @@ -54,7 +54,7 @@
> sub run {
> my($self, $list, $args) = @_;
>
> - my $order = $args->{order} || 'random';
> + my $order = $args->{order} || 'repeat';
> if ($order =~ /^\d+$/) {
> #dont want an explicit -seed option but env var can be a pain
> #so if -order is number assume it is the random seed
>
>
> Furthermore the READMEs and Changelog should get updated:
>
> ´Index: README
> ===================================================================
> --- README (revision 1125303)
> +++ README (working copy)
> @@ -77,18 +77,20 @@
>
> It's also possible that a test will pass when it's run after a
> particular test, but if moved to run after a different state it may
> -fail. For this reason it's important to ran the randomization smoking
> -testing with -order=random option.
> +fail. For this reason by default the tests run in random order.
>
> Since it's important to be able to reproduce the problem with the
> random testing, whenever -order=random is used, the used seed is
> printed to STDERR. Which can be then fed into the future tests with:
> via APACHE_TEST_SEED environment variable.
>
> +By adding the option -order=repeat, the tests will be run in
> +alphabetical order.
> +
> Combining these two important smoke testing techiques, one can run
> tests with:
>
> - t/SMOKE -times=N -order=(repeat|rotate|random)
> + t/SMOKE -times=N -order=(repeat|random)
>
> For example, to run the mod_rewrite tests 5 times, one would:
>
> @@ -98,8 +100,7 @@
> modes:
>
> - randomize all tests
> -- repeat the same test N times in sequence
> -- rotate the whole tests suite N times
> +- repeat the whole tests suite N times
>
> For configuration options and default settings run:
>
> Index: Apache-Test/README
> ===================================================================
> --- Apache-Test/README (revision 1125303)
> +++ Apache-Test/README (working copy)
> @@ -203,18 +203,15 @@
>
> run all tests 10 times in a random order (the seed is autogenerated
> and reported)
> -% t/TEST -times=10 -order=random
> +% t/SMOKE -times=10 -order=random
>
> run all tests 10 times in a random order using the seed obtained from
> the previous random run (e.g. 2352211):
> -% t/TEST -times=10 -order=2352211
> +% t/SMOKE -times=10 -order=2352211
>
> -rotate all tests 10 times (a, b, c, a, b, c)
> -% t/TEST -times=10 -order=rotate
> +repeat all tests 10 times (a, b, c, a, b, c)
> +% t/SMOKE -times=10 -order=repeat
>
> -repeat all tests 10 times (a, a, b, b, c, c)
> -% t/TEST -times=10 -order=repeat
> -
> When certain tests fail when running with -times option, you want to
> find out the minimal sequence of tests that lead to the
> failure. Apache::TestSmoke helps to ease this task, simply run:
> Index: Apache-Test/Changes
> ===================================================================
> --- Apache-Test/Changes (revision 1125303)
> +++ Apache-Test/Changes (working copy)
> @@ -8,6 +8,9 @@
>
> =item 1.37-dev
>
> +Run t/TEST tests by default in alphabetical order and
> +only t/SMOKE tests by default in random order.
> +
> Add t_file_watch_for to Apache::TestUtil [Torsten Foertsch]
>
> Add $boolean parameter to Apache::TestHandler::ok and
> Apache::TestHandler::ok1
>
> Regards,
>
> Rainer
>
[PATCH] New default sort order in Apache-Test-1.36 (WAS: test framework
and latest Fedora)
Posted by Rainer Jung <ra...@kippdata.de>.
I noticed, that the SMOKE change in 1.36 also introduced a new sorting
behaviour. In the change the test suite was running by default using the
repeat order, that means the tests were ordered alpabetically.
Now the default order is random.
Furthermore the -order=rotate (A B C A B C) is gone, and -order=repeat
now behaves like previously -order=rotate (it seems it previously was A
A B B C C).
I suggest:
- keep behaviour and name of repeat as it is now
- don't try to reimplement old repeat, because it might be tricky in
combination with the new way iterations are done (SMOKE)
- switch back to the new *repeat* (=old rotate) as the default for
non-SMOKE tests, i.e. run the tests by default in sorted order and only
do random if -order=random is explicitely given.
- stay with random as default for SMOKE
The following one line change should do that:
Index: Apache-Test/lib/Apache/TestSort.pm
===================================================================
--- Apache-Test/lib/Apache/TestSort.pm (revision 1125303)
+++ Apache-Test/lib/Apache/TestSort.pm (working copy)
@@ -54,7 +54,7 @@
sub run {
my($self, $list, $args) = @_;
- my $order = $args->{order} || 'random';
+ my $order = $args->{order} || 'repeat';
if ($order =~ /^\d+$/) {
#dont want an explicit -seed option but env var can be a pain
#so if -order is number assume it is the random seed
Furthermore the READMEs and Changelog should get updated:
´Index: README
===================================================================
--- README (revision 1125303)
+++ README (working copy)
@@ -77,18 +77,20 @@
It's also possible that a test will pass when it's run after a
particular test, but if moved to run after a different state it may
-fail. For this reason it's important to ran the randomization smoking
-testing with -order=random option.
+fail. For this reason by default the tests run in random order.
Since it's important to be able to reproduce the problem with the
random testing, whenever -order=random is used, the used seed is
printed to STDERR. Which can be then fed into the future tests with:
via APACHE_TEST_SEED environment variable.
+By adding the option -order=repeat, the tests will be run in
+alphabetical order.
+
Combining these two important smoke testing techiques, one can run
tests with:
- t/SMOKE -times=N -order=(repeat|rotate|random)
+ t/SMOKE -times=N -order=(repeat|random)
For example, to run the mod_rewrite tests 5 times, one would:
@@ -98,8 +100,7 @@
modes:
- randomize all tests
-- repeat the same test N times in sequence
-- rotate the whole tests suite N times
+- repeat the whole tests suite N times
For configuration options and default settings run:
Index: Apache-Test/README
===================================================================
--- Apache-Test/README (revision 1125303)
+++ Apache-Test/README (working copy)
@@ -203,18 +203,15 @@
run all tests 10 times in a random order (the seed is autogenerated
and reported)
-% t/TEST -times=10 -order=random
+% t/SMOKE -times=10 -order=random
run all tests 10 times in a random order using the seed obtained from
the previous random run (e.g. 2352211):
-% t/TEST -times=10 -order=2352211
+% t/SMOKE -times=10 -order=2352211
-rotate all tests 10 times (a, b, c, a, b, c)
-% t/TEST -times=10 -order=rotate
+repeat all tests 10 times (a, b, c, a, b, c)
+% t/SMOKE -times=10 -order=repeat
-repeat all tests 10 times (a, a, b, b, c, c)
-% t/TEST -times=10 -order=repeat
-
When certain tests fail when running with -times option, you want to
find out the minimal sequence of tests that lead to the
failure. Apache::TestSmoke helps to ease this task, simply run:
Index: Apache-Test/Changes
===================================================================
--- Apache-Test/Changes (revision 1125303)
+++ Apache-Test/Changes (working copy)
@@ -8,6 +8,9 @@
=item 1.37-dev
+Run t/TEST tests by default in alphabetical order and
+only t/SMOKE tests by default in random order.
+
Add t_file_watch_for to Apache::TestUtil [Torsten Foertsch]
Add $boolean parameter to Apache::TestHandler::ok and
Apache::TestHandler::ok1
Regards,
Rainer
Re: test framework and latest Fedora
Posted by Jim Jagielski <ji...@jagunet.com>.
Looks perfect! ;)
On Jan 23, 2011, at 4:27 PM, Fred Moyer wrote:
> On Sun, Jan 23, 2011 at 10:52 AM, Jim Jagielski <ji...@jagunet.com> wrote:
>>
>> I'll try SMOKE... I basically use times simply to run the same
>> test over and over. For example, when people said that the proxy
>> changes caused the rewrite tests to occasionally fail, a
>>
>> t/TEST -times=100 t/modules/rewrite
>>
>> was a good way to try to recreate that...
>
> I've made a number of changes which will hopefully fill the needs
> here. This solution may not be able to go through tests as rapid fire
> as previously, but should be able to do 100 times in less than 2
> hours. See the output below fo the syntax I used. Feedback welcome.
>
> New Revision: 1062529
> URL: http://svn.apache.org/viewvc?rev=1062529&view=rev
>
> So -times=X in ./t/TEST has been removed, but is now in t/SMOKE.
> t/SMOKE also does away with the concept of iterations, and for random
> ordering it creates a new seed before each 'time'.
>
> phred@pooky ~/dev/svn/httpd/test/framework/trunk $ time t/SMOKE
> -times=10 -verbose t/modules/rewrite
> Using random number seed: 3285025243 (autogenerated)
> Report file: /Users/phred/dev/svn/httpd/test/framework/trunk/smoke-report-Sun_Jan_23_13-23-07_2011.txt
>
> ------------------------------------------------------------
> [001-00-10] running all tests
> Using random number seed: 3285025243 (autogenerated)
> t/modules/rewrite..ok
>
> ------------------------------------------------------------
> [002-00-10] running all tests
> Using random number seed: 3285025188 (autogenerated)
> t/modules/rewrite..ok
>
> real 1m26.050s
>
> ...
>
> ========================= Summary ==========================
> Completion : Completed
> Status : +++ OK +++
> Tests run : 10
> Iterations (random) made : 10
> ------------------------------------------------------------
> --- Started at: Sun Jan 23 13:23:07 2011 ---
> --- Ended at: Sun Jan 23 13:24:32 2011 ---
> ------------------------------------------------------------
>
[PATCH] New default sort order in Apache-Test-1.36 (WAS: test framework
and latest Fedora)
Posted by Rainer Jung <ra...@kippdata.de>.
I noticed, that the SMOKE change in 1.36 also introduced a new sorting
behaviour. In the change the test suite was running by default using the
repeat order, that means the tests were ordered alpabetically.
Now the default order is random.
Furthermore the -order=rotate (A B C A B C) is gone, and -order=repeat
now behaves like previously -order=rotate (it seems it previously was A
A B B C C).
I suggest:
- keep behaviour and name of repeat as it is now
- don't try to reimplement old repeat, because it might be tricky in
combination with the new way iterations are done (SMOKE)
- switch back to the new *repeat* (=old rotate) as the default for
non-SMOKE tests, i.e. run the tests by default in sorted order and only
do random if -order=random is explicitely given.
- stay with random as default for SMOKE
The following one line change should do that:
Index: Apache-Test/lib/Apache/TestSort.pm
===================================================================
--- Apache-Test/lib/Apache/TestSort.pm (revision 1125303)
+++ Apache-Test/lib/Apache/TestSort.pm (working copy)
@@ -54,7 +54,7 @@
sub run {
my($self, $list, $args) = @_;
- my $order = $args->{order} || 'random';
+ my $order = $args->{order} || 'repeat';
if ($order =~ /^\d+$/) {
#dont want an explicit -seed option but env var can be a pain
#so if -order is number assume it is the random seed
Furthermore the READMEs and Changelog should get updated:
´Index: README
===================================================================
--- README (revision 1125303)
+++ README (working copy)
@@ -77,18 +77,20 @@
It's also possible that a test will pass when it's run after a
particular test, but if moved to run after a different state it may
-fail. For this reason it's important to ran the randomization smoking
-testing with -order=random option.
+fail. For this reason by default the tests run in random order.
Since it's important to be able to reproduce the problem with the
random testing, whenever -order=random is used, the used seed is
printed to STDERR. Which can be then fed into the future tests with:
via APACHE_TEST_SEED environment variable.
+By adding the option -order=repeat, the tests will be run in
+alphabetical order.
+
Combining these two important smoke testing techiques, one can run
tests with:
- t/SMOKE -times=N -order=(repeat|rotate|random)
+ t/SMOKE -times=N -order=(repeat|random)
For example, to run the mod_rewrite tests 5 times, one would:
@@ -98,8 +100,7 @@
modes:
- randomize all tests
-- repeat the same test N times in sequence
-- rotate the whole tests suite N times
+- repeat the whole tests suite N times
For configuration options and default settings run:
Index: Apache-Test/README
===================================================================
--- Apache-Test/README (revision 1125303)
+++ Apache-Test/README (working copy)
@@ -203,18 +203,15 @@
run all tests 10 times in a random order (the seed is autogenerated
and reported)
-% t/TEST -times=10 -order=random
+% t/SMOKE -times=10 -order=random
run all tests 10 times in a random order using the seed obtained from
the previous random run (e.g. 2352211):
-% t/TEST -times=10 -order=2352211
+% t/SMOKE -times=10 -order=2352211
-rotate all tests 10 times (a, b, c, a, b, c)
-% t/TEST -times=10 -order=rotate
+repeat all tests 10 times (a, b, c, a, b, c)
+% t/SMOKE -times=10 -order=repeat
-repeat all tests 10 times (a, a, b, b, c, c)
-% t/TEST -times=10 -order=repeat
-
When certain tests fail when running with -times option, you want to
find out the minimal sequence of tests that lead to the
failure. Apache::TestSmoke helps to ease this task, simply run:
Index: Apache-Test/Changes
===================================================================
--- Apache-Test/Changes (revision 1125303)
+++ Apache-Test/Changes (working copy)
@@ -8,6 +8,9 @@
=item 1.37-dev
+Run t/TEST tests by default in alphabetical order and
+only t/SMOKE tests by default in random order.
+
Add t_file_watch_for to Apache::TestUtil [Torsten Foertsch]
Add $boolean parameter to Apache::TestHandler::ok and
Apache::TestHandler::ok1
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org
Re: test framework and latest Fedora
Posted by Jim Jagielski <ji...@jagunet.com>.
Looks perfect! ;)
On Jan 23, 2011, at 4:27 PM, Fred Moyer wrote:
> On Sun, Jan 23, 2011 at 10:52 AM, Jim Jagielski <ji...@jagunet.com> wrote:
>>
>> I'll try SMOKE... I basically use times simply to run the same
>> test over and over. For example, when people said that the proxy
>> changes caused the rewrite tests to occasionally fail, a
>>
>> t/TEST -times=100 t/modules/rewrite
>>
>> was a good way to try to recreate that...
>
> I've made a number of changes which will hopefully fill the needs
> here. This solution may not be able to go through tests as rapid fire
> as previously, but should be able to do 100 times in less than 2
> hours. See the output below fo the syntax I used. Feedback welcome.
>
> New Revision: 1062529
> URL: http://svn.apache.org/viewvc?rev=1062529&view=rev
>
> So -times=X in ./t/TEST has been removed, but is now in t/SMOKE.
> t/SMOKE also does away with the concept of iterations, and for random
> ordering it creates a new seed before each 'time'.
>
> phred@pooky ~/dev/svn/httpd/test/framework/trunk $ time t/SMOKE
> -times=10 -verbose t/modules/rewrite
> Using random number seed: 3285025243 (autogenerated)
> Report file: /Users/phred/dev/svn/httpd/test/framework/trunk/smoke-report-Sun_Jan_23_13-23-07_2011.txt
>
> ------------------------------------------------------------
> [001-00-10] running all tests
> Using random number seed: 3285025243 (autogenerated)
> t/modules/rewrite..ok
>
> ------------------------------------------------------------
> [002-00-10] running all tests
> Using random number seed: 3285025188 (autogenerated)
> t/modules/rewrite..ok
>
> real 1m26.050s
>
> ...
>
> ========================= Summary ==========================
> Completion : Completed
> Status : +++ OK +++
> Tests run : 10
> Iterations (random) made : 10
> ------------------------------------------------------------
> --- Started at: Sun Jan 23 13:23:07 2011 ---
> --- Ended at: Sun Jan 23 13:24:32 2011 ---
> ------------------------------------------------------------
>
[PATCH] New default sort order in Apache-Test-1.36 (WAS: test framework
and latest Fedora)
Posted by Rainer Jung <ra...@kippdata.de>.
I noticed, that the SMOKE change in 1.36 also introduced a new sorting
behaviour. In the change the test suite was running by default using the
repeat order, that means the tests were ordered alpabetically.
Now the default order is random.
Furthermore the -order=rotate (A B C A B C) is gone, and -order=repeat
now behaves like previously -order=rotate (it seems it previously was A
A B B C C).
I suggest:
- keep behaviour and name of repeat as it is now
- don't try to reimplement old repeat, because it might be tricky in
combination with the new way iterations are done (SMOKE)
- switch back to the new *repeat* (=old rotate) as the default for
non-SMOKE tests, i.e. run the tests by default in sorted order and only
do random if -order=random is explicitely given.
- stay with random as default for SMOKE
The following one line change should do that:
Index: Apache-Test/lib/Apache/TestSort.pm
===================================================================
--- Apache-Test/lib/Apache/TestSort.pm (revision 1125303)
+++ Apache-Test/lib/Apache/TestSort.pm (working copy)
@@ -54,7 +54,7 @@
sub run {
my($self, $list, $args) = @_;
- my $order = $args->{order} || 'random';
+ my $order = $args->{order} || 'repeat';
if ($order =~ /^\d+$/) {
#dont want an explicit -seed option but env var can be a pain
#so if -order is number assume it is the random seed
Furthermore the READMEs and Changelog should get updated:
´Index: README
===================================================================
--- README (revision 1125303)
+++ README (working copy)
@@ -77,18 +77,20 @@
It's also possible that a test will pass when it's run after a
particular test, but if moved to run after a different state it may
-fail. For this reason it's important to ran the randomization smoking
-testing with -order=random option.
+fail. For this reason by default the tests run in random order.
Since it's important to be able to reproduce the problem with the
random testing, whenever -order=random is used, the used seed is
printed to STDERR. Which can be then fed into the future tests with:
via APACHE_TEST_SEED environment variable.
+By adding the option -order=repeat, the tests will be run in
+alphabetical order.
+
Combining these two important smoke testing techiques, one can run
tests with:
- t/SMOKE -times=N -order=(repeat|rotate|random)
+ t/SMOKE -times=N -order=(repeat|random)
For example, to run the mod_rewrite tests 5 times, one would:
@@ -98,8 +100,7 @@
modes:
- randomize all tests
-- repeat the same test N times in sequence
-- rotate the whole tests suite N times
+- repeat the whole tests suite N times
For configuration options and default settings run:
Index: Apache-Test/README
===================================================================
--- Apache-Test/README (revision 1125303)
+++ Apache-Test/README (working copy)
@@ -203,18 +203,15 @@
run all tests 10 times in a random order (the seed is autogenerated
and reported)
-% t/TEST -times=10 -order=random
+% t/SMOKE -times=10 -order=random
run all tests 10 times in a random order using the seed obtained from
the previous random run (e.g. 2352211):
-% t/TEST -times=10 -order=2352211
+% t/SMOKE -times=10 -order=2352211
-rotate all tests 10 times (a, b, c, a, b, c)
-% t/TEST -times=10 -order=rotate
+repeat all tests 10 times (a, b, c, a, b, c)
+% t/SMOKE -times=10 -order=repeat
-repeat all tests 10 times (a, a, b, b, c, c)
-% t/TEST -times=10 -order=repeat
-
When certain tests fail when running with -times option, you want to
find out the minimal sequence of tests that lead to the
failure. Apache::TestSmoke helps to ease this task, simply run:
Index: Apache-Test/Changes
===================================================================
--- Apache-Test/Changes (revision 1125303)
+++ Apache-Test/Changes (working copy)
@@ -8,6 +8,9 @@
=item 1.37-dev
+Run t/TEST tests by default in alphabetical order and
+only t/SMOKE tests by default in random order.
+
Add t_file_watch_for to Apache::TestUtil [Torsten Foertsch]
Add $boolean parameter to Apache::TestHandler::ok and
Apache::TestHandler::ok1
Regards,
Rainer
Re: test framework and latest Fedora
Posted by Fred Moyer <fr...@redhotpenguin.com>.
On Sun, Jan 23, 2011 at 10:52 AM, Jim Jagielski <ji...@jagunet.com> wrote:
>
> I'll try SMOKE... I basically use times simply to run the same
> test over and over. For example, when people said that the proxy
> changes caused the rewrite tests to occasionally fail, a
>
> t/TEST -times=100 t/modules/rewrite
>
> was a good way to try to recreate that...
I've made a number of changes which will hopefully fill the needs
here. This solution may not be able to go through tests as rapid fire
as previously, but should be able to do 100 times in less than 2
hours. See the output below fo the syntax I used. Feedback welcome.
New Revision: 1062529
URL: http://svn.apache.org/viewvc?rev=1062529&view=rev
So -times=X in ./t/TEST has been removed, but is now in t/SMOKE.
t/SMOKE also does away with the concept of iterations, and for random
ordering it creates a new seed before each 'time'.
phred@pooky ~/dev/svn/httpd/test/framework/trunk $ time t/SMOKE
-times=10 -verbose t/modules/rewrite
Using random number seed: 3285025243 (autogenerated)
Report file: /Users/phred/dev/svn/httpd/test/framework/trunk/smoke-report-Sun_Jan_23_13-23-07_2011.txt
------------------------------------------------------------
[001-00-10] running all tests
Using random number seed: 3285025243 (autogenerated)
t/modules/rewrite..ok
------------------------------------------------------------
[002-00-10] running all tests
Using random number seed: 3285025188 (autogenerated)
t/modules/rewrite..ok
real 1m26.050s
...
========================= Summary ==========================
Completion : Completed
Status : +++ OK +++
Tests run : 10
Iterations (random) made : 10
------------------------------------------------------------
--- Started at: Sun Jan 23 13:23:07 2011 ---
--- Ended at: Sun Jan 23 13:24:32 2011 ---
------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org
Re: test framework and latest Fedora
Posted by Fred Moyer <fr...@redhotpenguin.com>.
On Sun, Jan 23, 2011 at 10:52 AM, Jim Jagielski <ji...@jagunet.com> wrote:
>
> On Jan 23, 2011, at 1:46 PM, Fred Moyer wrote:
>> I welcome feedback here on desired approaches. What is 'times=' being
>> used for? Can you tryout t/SMOKE instead?
>>
>
> I'll try SMOKE... I basically use times simply to run the same
> test over and over. For example, when people said that the proxy
> changes caused the rewrite tests to occasionally fail, a
>
> t/TEST -times=100 t/modules/rewrite
>
> was a good way to try to recreate that...
Ah good point... I'll see if I can tweak SMOKE to do the same thing,
and have it show up like TEST does when you run Makefile.PL.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org
Re: test framework and latest Fedora
Posted by Fred Moyer <fr...@redhotpenguin.com>.
On Sun, Jan 23, 2011 at 10:52 AM, Jim Jagielski <ji...@jagunet.com> wrote:
>
> On Jan 23, 2011, at 1:46 PM, Fred Moyer wrote:
>> I welcome feedback here on desired approaches. What is 'times=' being
>> used for? Can you tryout t/SMOKE instead?
>>
>
> I'll try SMOKE... I basically use times simply to run the same
> test over and over. For example, when people said that the proxy
> changes caused the rewrite tests to occasionally fail, a
>
> t/TEST -times=100 t/modules/rewrite
>
> was a good way to try to recreate that...
Ah good point... I'll see if I can tweak SMOKE to do the same thing,
and have it show up like TEST does when you run Makefile.PL.
Re: test framework and latest Fedora
Posted by Fred Moyer <fr...@redhotpenguin.com>.
On Sun, Jan 23, 2011 at 10:52 AM, Jim Jagielski <ji...@jagunet.com> wrote:
>
> I'll try SMOKE... I basically use times simply to run the same
> test over and over. For example, when people said that the proxy
> changes caused the rewrite tests to occasionally fail, a
>
> t/TEST -times=100 t/modules/rewrite
>
> was a good way to try to recreate that...
I've made a number of changes which will hopefully fill the needs
here. This solution may not be able to go through tests as rapid fire
as previously, but should be able to do 100 times in less than 2
hours. See the output below fo the syntax I used. Feedback welcome.
New Revision: 1062529
URL: http://svn.apache.org/viewvc?rev=1062529&view=rev
So -times=X in ./t/TEST has been removed, but is now in t/SMOKE.
t/SMOKE also does away with the concept of iterations, and for random
ordering it creates a new seed before each 'time'.
phred@pooky ~/dev/svn/httpd/test/framework/trunk $ time t/SMOKE
-times=10 -verbose t/modules/rewrite
Using random number seed: 3285025243 (autogenerated)
Report file: /Users/phred/dev/svn/httpd/test/framework/trunk/smoke-report-Sun_Jan_23_13-23-07_2011.txt
------------------------------------------------------------
[001-00-10] running all tests
Using random number seed: 3285025243 (autogenerated)
t/modules/rewrite..ok
------------------------------------------------------------
[002-00-10] running all tests
Using random number seed: 3285025188 (autogenerated)
t/modules/rewrite..ok
real 1m26.050s
...
========================= Summary ==========================
Completion : Completed
Status : +++ OK +++
Tests run : 10
Iterations (random) made : 10
------------------------------------------------------------
--- Started at: Sun Jan 23 13:23:07 2011 ---
--- Ended at: Sun Jan 23 13:24:32 2011 ---
------------------------------------------------------------
Re: test framework and latest Fedora
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Jan 23, 2011, at 1:46 PM, Fred Moyer wrote:
> On Fri, Jan 21, 2011 at 1:29 AM, Joe Orton <jo...@redhat.com> wrote:
>> CC'ing test-dev@.
>>
>> On Thu, Jan 20, 2011 at 12:00:41PM -0500, Jim Jagielski wrote:
>>> On the latest Fedora, -times=X no longer works:
>>>
>>> t/modules/rewrite.t .. ok
>>> t/modules/rewrite.t .. ok
>>> You already have a parser for (t/modules/rewrite.t). Perhaps you have run the same test twice. at /usr/share/perl5/TAP/Harness.pm line 521
>>>
>>> CPAN itself is up to date... or is that the problem?
>
> That looks like the problem. When TAP/Parser/Aggregator.pm was added
> in March 2007, it croaked if you tried to reuse the test description.
>
> As the primary Apache-Test maintainer right now, I am tempted to
> deprecate this feature in favor of making the codebase simple and
> easier to maintain in the long run. Updating this feature to work
> with TAP-Harness is an unknown amount of work. And running tests
> multiple times is more fitting of something like t/SMOKE
> (Apache::TestSmoke) rather than t/TEST. I can get in there and muck
> with the internals of Apache::TestRun (and A-TR-PHP, Perl, etc) to
> make it behave properly, but my tuits are better spent on insuring
> quality in other parts of this module, and also releasing mod_perl
> 2.0.5.
>
> I welcome feedback here on desired approaches. What is 'times=' being
> used for? Can you tryout t/SMOKE instead?
>
I'll try SMOKE... I basically use times simply to run the same
test over and over. For example, when people said that the proxy
changes caused the rewrite tests to occasionally fail, a
t/TEST -times=100 t/modules/rewrite
was a good way to try to recreate that...
Re: test framework and latest Fedora
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Jan 23, 2011, at 1:46 PM, Fred Moyer wrote:
> On Fri, Jan 21, 2011 at 1:29 AM, Joe Orton <jo...@redhat.com> wrote:
>> CC'ing test-dev@.
>>
>> On Thu, Jan 20, 2011 at 12:00:41PM -0500, Jim Jagielski wrote:
>>> On the latest Fedora, -times=X no longer works:
>>>
>>> t/modules/rewrite.t .. ok
>>> t/modules/rewrite.t .. ok
>>> You already have a parser for (t/modules/rewrite.t). Perhaps you have run the same test twice. at /usr/share/perl5/TAP/Harness.pm line 521
>>>
>>> CPAN itself is up to date... or is that the problem?
>
> That looks like the problem. When TAP/Parser/Aggregator.pm was added
> in March 2007, it croaked if you tried to reuse the test description.
>
> As the primary Apache-Test maintainer right now, I am tempted to
> deprecate this feature in favor of making the codebase simple and
> easier to maintain in the long run. Updating this feature to work
> with TAP-Harness is an unknown amount of work. And running tests
> multiple times is more fitting of something like t/SMOKE
> (Apache::TestSmoke) rather than t/TEST. I can get in there and muck
> with the internals of Apache::TestRun (and A-TR-PHP, Perl, etc) to
> make it behave properly, but my tuits are better spent on insuring
> quality in other parts of this module, and also releasing mod_perl
> 2.0.5.
>
> I welcome feedback here on desired approaches. What is 'times=' being
> used for? Can you tryout t/SMOKE instead?
>
I'll try SMOKE... I basically use times simply to run the same
test over and over. For example, when people said that the proxy
changes caused the rewrite tests to occasionally fail, a
t/TEST -times=100 t/modules/rewrite
was a good way to try to recreate that...