You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Rainer Jung <ra...@kippdata.de> on 2011/05/20 22:51:20 UTC

[PATCH] New default sort order in Apache-Test-1.36 (WAS: test framework and latest Fedora)

I noticed, that the SMOKE change in 1.36 also introduced a new sorting
behaviour. In the change the test suite was running by default using the
repeat order, that means the tests were ordered alpabetically.

Now the default order is random.

Furthermore the -order=rotate (A B C A B C) is gone, and -order=repeat
now behaves like previously -order=rotate (it seems it previously was A
A B B C C).

I suggest:

- keep behaviour and name of repeat as it is now
- don't try to reimplement old repeat, because it might be tricky in
combination with the new way iterations are done (SMOKE)
- switch back to the new *repeat* (=old rotate) as the default for
non-SMOKE tests, i.e. run the tests by default in sorted order and only
do random if -order=random is explicitely given.
- stay with random as default for SMOKE

The following one line change should do that:

Index: Apache-Test/lib/Apache/TestSort.pm
===================================================================
--- Apache-Test/lib/Apache/TestSort.pm  (revision 1125303)
+++ Apache-Test/lib/Apache/TestSort.pm  (working copy)
@@ -54,7 +54,7 @@
 sub run {
     my($self, $list, $args) = @_;

-    my $order = $args->{order} || 'random';
+    my $order = $args->{order} || 'repeat';
     if ($order =~ /^\d+$/) {
         #dont want an explicit -seed option but env var can be a pain
         #so if -order is number assume it is the random seed


Furthermore the READMEs and Changelog should get updated:

´Index: README
===================================================================
--- README      (revision 1125303)
+++ README      (working copy)
@@ -77,18 +77,20 @@

 It's also possible that a test will pass when it's run after a
 particular test, but if moved to run after a different state it may
-fail. For this reason it's important to ran the randomization smoking
-testing with -order=random option.
+fail. For this reason by default the tests run in random order.

 Since it's important to be able to reproduce the problem with the
 random testing, whenever -order=random is used, the used seed is
 printed to STDERR. Which can be then fed into the future tests with:
 via APACHE_TEST_SEED environment variable.

+By adding the option -order=repeat, the tests will be run in
+alphabetical order.
+
 Combining these two important smoke testing techiques, one can run
 tests with:

-    t/SMOKE -times=N -order=(repeat|rotate|random)
+    t/SMOKE -times=N -order=(repeat|random)

 For example, to run the mod_rewrite tests 5 times, one would:

@@ -98,8 +100,7 @@
 modes:

 - randomize all tests
-- repeat the same test N times in sequence
-- rotate the whole tests suite N times
+- repeat the whole tests suite N times

 For configuration options and default settings run:

Index: Apache-Test/README
===================================================================
--- Apache-Test/README  (revision 1125303)
+++ Apache-Test/README  (working copy)
@@ -203,18 +203,15 @@

 run all tests 10 times in a random order (the seed is autogenerated
 and reported)
-% t/TEST -times=10 -order=random
+% t/SMOKE -times=10 -order=random

 run all tests 10 times in a random order using the seed obtained from
 the previous random run (e.g. 2352211):
-% t/TEST -times=10 -order=2352211
+% t/SMOKE -times=10 -order=2352211

-rotate all tests 10 times (a, b, c, a, b, c)
-% t/TEST -times=10 -order=rotate
+repeat all tests 10 times (a, b, c, a, b, c)
+% t/SMOKE -times=10 -order=repeat

-repeat all tests 10 times (a, a, b, b, c, c)
-% t/TEST -times=10 -order=repeat
-
 When certain tests fail when running with -times option, you want to
 find out the minimal sequence of tests that lead to the
 failure. Apache::TestSmoke helps to ease this task, simply run:
Index: Apache-Test/Changes
===================================================================
--- Apache-Test/Changes (revision 1125303)
+++ Apache-Test/Changes (working copy)
@@ -8,6 +8,9 @@

 =item 1.37-dev

+Run t/TEST tests by default in alphabetical order and
+only t/SMOKE tests by default in random order.
+
 Add t_file_watch_for to Apache::TestUtil [Torsten Foertsch]

 Add $boolean parameter to Apache::TestHandler::ok and
Apache::TestHandler::ok1

Regards,

Rainer

Re: [PATCH] New default sort order in Apache-Test-1.36 (WAS: test framework and latest Fedora)

Posted by Rainer Jung <ra...@kippdata.de>.
Hi Fred,

On 26.05.2011 02:15, Fred Moyer wrote:
> +1
> 
> Do you have commit privileges on Apache-Test?

Surprise, yes :)

Committed as r1127812.

Regards,

Rainer

> On Fri, May 20, 2011 at 1:51 PM, Rainer Jung <ra...@kippdata.de> wrote:
>> I noticed, that the SMOKE change in 1.36 also introduced a new sorting
>> behaviour. In the change the test suite was running by default using the
>> repeat order, that means the tests were ordered alpabetically.
>>
>> Now the default order is random.
>>
>> Furthermore the -order=rotate (A B C A B C) is gone, and -order=repeat
>> now behaves like previously -order=rotate (it seems it previously was A
>> A B B C C).
>>
>> I suggest:
>>
>> - keep behaviour and name of repeat as it is now
>> - don't try to reimplement old repeat, because it might be tricky in
>> combination with the new way iterations are done (SMOKE)
>> - switch back to the new *repeat* (=old rotate) as the default for
>> non-SMOKE tests, i.e. run the tests by default in sorted order and only
>> do random if -order=random is explicitely given.
>> - stay with random as default for SMOKE
>>
>> The following one line change should do that:
>>
>> Index: Apache-Test/lib/Apache/TestSort.pm
>> ===================================================================
>> --- Apache-Test/lib/Apache/TestSort.pm  (revision 1125303)
>> +++ Apache-Test/lib/Apache/TestSort.pm  (working copy)
>> @@ -54,7 +54,7 @@
>>  sub run {
>>     my($self, $list, $args) = @_;
>>
>> -    my $order = $args->{order} || 'random';
>> +    my $order = $args->{order} || 'repeat';
>>     if ($order =~ /^\d+$/) {
>>         #dont want an explicit -seed option but env var can be a pain
>>         #so if -order is number assume it is the random seed
>>
>>
>> Furthermore the READMEs and Changelog should get updated:
>>
>> ´Index: README
>> ===================================================================
>> --- README      (revision 1125303)
>> +++ README      (working copy)
>> @@ -77,18 +77,20 @@
>>
>>  It's also possible that a test will pass when it's run after a
>>  particular test, but if moved to run after a different state it may
>> -fail. For this reason it's important to ran the randomization smoking
>> -testing with -order=random option.
>> +fail. For this reason by default the tests run in random order.
>>
>>  Since it's important to be able to reproduce the problem with the
>>  random testing, whenever -order=random is used, the used seed is
>>  printed to STDERR. Which can be then fed into the future tests with:
>>  via APACHE_TEST_SEED environment variable.
>>
>> +By adding the option -order=repeat, the tests will be run in
>> +alphabetical order.
>> +
>>  Combining these two important smoke testing techiques, one can run
>>  tests with:
>>
>> -    t/SMOKE -times=N -order=(repeat|rotate|random)
>> +    t/SMOKE -times=N -order=(repeat|random)
>>
>>  For example, to run the mod_rewrite tests 5 times, one would:
>>
>> @@ -98,8 +100,7 @@
>>  modes:
>>
>>  - randomize all tests
>> -- repeat the same test N times in sequence
>> -- rotate the whole tests suite N times
>> +- repeat the whole tests suite N times
>>
>>  For configuration options and default settings run:
>>
>> Index: Apache-Test/README
>> ===================================================================
>> --- Apache-Test/README  (revision 1125303)
>> +++ Apache-Test/README  (working copy)
>> @@ -203,18 +203,15 @@
>>
>>  run all tests 10 times in a random order (the seed is autogenerated
>>  and reported)
>> -% t/TEST -times=10 -order=random
>> +% t/SMOKE -times=10 -order=random
>>
>>  run all tests 10 times in a random order using the seed obtained from
>>  the previous random run (e.g. 2352211):
>> -% t/TEST -times=10 -order=2352211
>> +% t/SMOKE -times=10 -order=2352211
>>
>> -rotate all tests 10 times (a, b, c, a, b, c)
>> -% t/TEST -times=10 -order=rotate
>> +repeat all tests 10 times (a, b, c, a, b, c)
>> +% t/SMOKE -times=10 -order=repeat
>>
>> -repeat all tests 10 times (a, a, b, b, c, c)
>> -% t/TEST -times=10 -order=repeat
>> -
>>  When certain tests fail when running with -times option, you want to
>>  find out the minimal sequence of tests that lead to the
>>  failure. Apache::TestSmoke helps to ease this task, simply run:
>> Index: Apache-Test/Changes
>> ===================================================================
>> --- Apache-Test/Changes (revision 1125303)
>> +++ Apache-Test/Changes (working copy)
>> @@ -8,6 +8,9 @@
>>
>>  =item 1.37-dev
>>
>> +Run t/TEST tests by default in alphabetical order and
>> +only t/SMOKE tests by default in random order.
>> +
>>  Add t_file_watch_for to Apache::TestUtil [Torsten Foertsch]
>>
>>  Add $boolean parameter to Apache::TestHandler::ok and
>> Apache::TestHandler::ok1
>>
>> Regards,
>>
>> Rainer

Re: [PATCH] New default sort order in Apache-Test-1.36 (WAS: test framework and latest Fedora)

Posted by Fred Moyer <fr...@redhotpenguin.com>.
+1

Do you have commit privileges on Apache-Test?

On Fri, May 20, 2011 at 1:51 PM, Rainer Jung <ra...@kippdata.de> wrote:
> I noticed, that the SMOKE change in 1.36 also introduced a new sorting
> behaviour. In the change the test suite was running by default using the
> repeat order, that means the tests were ordered alpabetically.
>
> Now the default order is random.
>
> Furthermore the -order=rotate (A B C A B C) is gone, and -order=repeat
> now behaves like previously -order=rotate (it seems it previously was A
> A B B C C).
>
> I suggest:
>
> - keep behaviour and name of repeat as it is now
> - don't try to reimplement old repeat, because it might be tricky in
> combination with the new way iterations are done (SMOKE)
> - switch back to the new *repeat* (=old rotate) as the default for
> non-SMOKE tests, i.e. run the tests by default in sorted order and only
> do random if -order=random is explicitely given.
> - stay with random as default for SMOKE
>
> The following one line change should do that:
>
> Index: Apache-Test/lib/Apache/TestSort.pm
> ===================================================================
> --- Apache-Test/lib/Apache/TestSort.pm  (revision 1125303)
> +++ Apache-Test/lib/Apache/TestSort.pm  (working copy)
> @@ -54,7 +54,7 @@
>  sub run {
>     my($self, $list, $args) = @_;
>
> -    my $order = $args->{order} || 'random';
> +    my $order = $args->{order} || 'repeat';
>     if ($order =~ /^\d+$/) {
>         #dont want an explicit -seed option but env var can be a pain
>         #so if -order is number assume it is the random seed
>
>
> Furthermore the READMEs and Changelog should get updated:
>
> ´Index: README
> ===================================================================
> --- README      (revision 1125303)
> +++ README      (working copy)
> @@ -77,18 +77,20 @@
>
>  It's also possible that a test will pass when it's run after a
>  particular test, but if moved to run after a different state it may
> -fail. For this reason it's important to ran the randomization smoking
> -testing with -order=random option.
> +fail. For this reason by default the tests run in random order.
>
>  Since it's important to be able to reproduce the problem with the
>  random testing, whenever -order=random is used, the used seed is
>  printed to STDERR. Which can be then fed into the future tests with:
>  via APACHE_TEST_SEED environment variable.
>
> +By adding the option -order=repeat, the tests will be run in
> +alphabetical order.
> +
>  Combining these two important smoke testing techiques, one can run
>  tests with:
>
> -    t/SMOKE -times=N -order=(repeat|rotate|random)
> +    t/SMOKE -times=N -order=(repeat|random)
>
>  For example, to run the mod_rewrite tests 5 times, one would:
>
> @@ -98,8 +100,7 @@
>  modes:
>
>  - randomize all tests
> -- repeat the same test N times in sequence
> -- rotate the whole tests suite N times
> +- repeat the whole tests suite N times
>
>  For configuration options and default settings run:
>
> Index: Apache-Test/README
> ===================================================================
> --- Apache-Test/README  (revision 1125303)
> +++ Apache-Test/README  (working copy)
> @@ -203,18 +203,15 @@
>
>  run all tests 10 times in a random order (the seed is autogenerated
>  and reported)
> -% t/TEST -times=10 -order=random
> +% t/SMOKE -times=10 -order=random
>
>  run all tests 10 times in a random order using the seed obtained from
>  the previous random run (e.g. 2352211):
> -% t/TEST -times=10 -order=2352211
> +% t/SMOKE -times=10 -order=2352211
>
> -rotate all tests 10 times (a, b, c, a, b, c)
> -% t/TEST -times=10 -order=rotate
> +repeat all tests 10 times (a, b, c, a, b, c)
> +% t/SMOKE -times=10 -order=repeat
>
> -repeat all tests 10 times (a, a, b, b, c, c)
> -% t/TEST -times=10 -order=repeat
> -
>  When certain tests fail when running with -times option, you want to
>  find out the minimal sequence of tests that lead to the
>  failure. Apache::TestSmoke helps to ease this task, simply run:
> Index: Apache-Test/Changes
> ===================================================================
> --- Apache-Test/Changes (revision 1125303)
> +++ Apache-Test/Changes (working copy)
> @@ -8,6 +8,9 @@
>
>  =item 1.37-dev
>
> +Run t/TEST tests by default in alphabetical order and
> +only t/SMOKE tests by default in random order.
> +
>  Add t_file_watch_for to Apache::TestUtil [Torsten Foertsch]
>
>  Add $boolean parameter to Apache::TestHandler::ok and
> Apache::TestHandler::ok1
>
> Regards,
>
> Rainer
>

Re: [PATCH] New default sort order in Apache-Test-1.36 (WAS: test framework and latest Fedora)

Posted by Fred Moyer <fr...@redhotpenguin.com>.
+1

Do you have commit privileges on Apache-Test?

On Fri, May 20, 2011 at 1:51 PM, Rainer Jung <ra...@kippdata.de> wrote:
> I noticed, that the SMOKE change in 1.36 also introduced a new sorting
> behaviour. In the change the test suite was running by default using the
> repeat order, that means the tests were ordered alpabetically.
>
> Now the default order is random.
>
> Furthermore the -order=rotate (A B C A B C) is gone, and -order=repeat
> now behaves like previously -order=rotate (it seems it previously was A
> A B B C C).
>
> I suggest:
>
> - keep behaviour and name of repeat as it is now
> - don't try to reimplement old repeat, because it might be tricky in
> combination with the new way iterations are done (SMOKE)
> - switch back to the new *repeat* (=old rotate) as the default for
> non-SMOKE tests, i.e. run the tests by default in sorted order and only
> do random if -order=random is explicitely given.
> - stay with random as default for SMOKE
>
> The following one line change should do that:
>
> Index: Apache-Test/lib/Apache/TestSort.pm
> ===================================================================
> --- Apache-Test/lib/Apache/TestSort.pm  (revision 1125303)
> +++ Apache-Test/lib/Apache/TestSort.pm  (working copy)
> @@ -54,7 +54,7 @@
>  sub run {
>     my($self, $list, $args) = @_;
>
> -    my $order = $args->{order} || 'random';
> +    my $order = $args->{order} || 'repeat';
>     if ($order =~ /^\d+$/) {
>         #dont want an explicit -seed option but env var can be a pain
>         #so if -order is number assume it is the random seed
>
>
> Furthermore the READMEs and Changelog should get updated:
>
> ´Index: README
> ===================================================================
> --- README      (revision 1125303)
> +++ README      (working copy)
> @@ -77,18 +77,20 @@
>
>  It's also possible that a test will pass when it's run after a
>  particular test, but if moved to run after a different state it may
> -fail. For this reason it's important to ran the randomization smoking
> -testing with -order=random option.
> +fail. For this reason by default the tests run in random order.
>
>  Since it's important to be able to reproduce the problem with the
>  random testing, whenever -order=random is used, the used seed is
>  printed to STDERR. Which can be then fed into the future tests with:
>  via APACHE_TEST_SEED environment variable.
>
> +By adding the option -order=repeat, the tests will be run in
> +alphabetical order.
> +
>  Combining these two important smoke testing techiques, one can run
>  tests with:
>
> -    t/SMOKE -times=N -order=(repeat|rotate|random)
> +    t/SMOKE -times=N -order=(repeat|random)
>
>  For example, to run the mod_rewrite tests 5 times, one would:
>
> @@ -98,8 +100,7 @@
>  modes:
>
>  - randomize all tests
> -- repeat the same test N times in sequence
> -- rotate the whole tests suite N times
> +- repeat the whole tests suite N times
>
>  For configuration options and default settings run:
>
> Index: Apache-Test/README
> ===================================================================
> --- Apache-Test/README  (revision 1125303)
> +++ Apache-Test/README  (working copy)
> @@ -203,18 +203,15 @@
>
>  run all tests 10 times in a random order (the seed is autogenerated
>  and reported)
> -% t/TEST -times=10 -order=random
> +% t/SMOKE -times=10 -order=random
>
>  run all tests 10 times in a random order using the seed obtained from
>  the previous random run (e.g. 2352211):
> -% t/TEST -times=10 -order=2352211
> +% t/SMOKE -times=10 -order=2352211
>
> -rotate all tests 10 times (a, b, c, a, b, c)
> -% t/TEST -times=10 -order=rotate
> +repeat all tests 10 times (a, b, c, a, b, c)
> +% t/SMOKE -times=10 -order=repeat
>
> -repeat all tests 10 times (a, a, b, b, c, c)
> -% t/TEST -times=10 -order=repeat
> -
>  When certain tests fail when running with -times option, you want to
>  find out the minimal sequence of tests that lead to the
>  failure. Apache::TestSmoke helps to ease this task, simply run:
> Index: Apache-Test/Changes
> ===================================================================
> --- Apache-Test/Changes (revision 1125303)
> +++ Apache-Test/Changes (working copy)
> @@ -8,6 +8,9 @@
>
>  =item 1.37-dev
>
> +Run t/TEST tests by default in alphabetical order and
> +only t/SMOKE tests by default in random order.
> +
>  Add t_file_watch_for to Apache::TestUtil [Torsten Foertsch]
>
>  Add $boolean parameter to Apache::TestHandler::ok and
> Apache::TestHandler::ok1
>
> Regards,
>
> Rainer
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org