You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by di...@covalent.net on 2002/05/01 17:05:27 UTC

Can AB be compared ?

Just to have some fun - Below is the result of running a build of AB
against the same apache 1.3.0 (stock)

It is a simple loop - checkout against a tag; cd apache-1.3/src && cp
Configuration.tmp && Configuration && ./Configure && make && cd support &&
make) and then run 20 times ./ab -c 30 -n 10000 and take the last 25 runs.

By the way - not a -single- tag failed to build on FreeBSD 5-Current and
not a single tag needed a differnet config ! A record hard to beat by
commercial software !

For each is displayed to total avaerage time/connection; the 98%
confidence interval (all in mSec) and the averegate for that version
of AB.

Note that typically (with eceptiosn early on) each version of AB produces
run within the confidence interval - and that successive versions their
confidence intertvals do generally not overlap (except early in the
development) in the more recent series.

So though this shows that comparing versions of AB from different versions
is generally waranted (same mean, same SD) - for recent versions of
1.3 - and that comparisions between version no's is not a good idea.

BUT at the same time - that has not always been the case - and that is is
kind of a miracle (or a tribute to release management/fact that AB hardly
ever changed) that comparison is possible. Also if you include 1.3c and
1.3a then you can conclude that these ranges match - thus from these
measurements you would conclude that the code did not change substantially
(it may of course do for keep alive or something).

Dw.

        Tag checout     #define in ab   mean/SD aver   CI
Tag:    APACHE_1_3_24   AB:     1.3d    66 +-3  66.5 +-3
Tag:    APACHE_1_3_23   AB:     1.3d    67 +-3
Tag:    APACHE_1_3_22   AB:     1.3d    67 +-3
Tag:    APACHE_1_3_21   AB:     1.3d    65 +-3

Tag:    APACHE_1_3_20   AB:     1.3c    63 +-2  61.8 +-2
Tag:    APACHE_1_3_19   AB:     1.3c    60 +-2
Tag:    APACHE_1_3_18   AB:     1.3c    61 +-2
Tag:    APACHE_1_3_17   AB:     1.3c    61 +-2
Tag:    APACHE_1_3_16   AB:     1.3c    63 +-2
Tag:    APACHE_1_3_15   AB:     1.3c    62 +-2
Tag:    APACHE_1_3_14   AB:     1.3c    63 +-2
Tag:    APACHE_1_3_13   AB:     1.3c    62 +-2
Tag:    APACHE_1_3_12   AB:     1.3c    62 +-2
Tag:    APACHE_1_3_11   AB:     1.3c    60 +-2
Tag:    APACHE_1_3_10   AB:     1.3c    63 +-2

Tag:    APACHE_1_3_9    AB:     1.3a    60 +-1  60.4 +-1
Tag:    APACHE_1_3_8    AB:     1.3a    61 +-1
Tag:    APACHE_1_3_7    AB:     1.3a    61 +-1

Tag:    APACHE_1_3_6    AB:     1.3     64 +-2  65.2 +-2
Tag:    APACHE_1_3_5    AB:     1.3     66 +-3

Tag:    APACHE_1_3_4    AB:     1.2     61 +-3  62.9 +-3
Tag:    APACHE_1_3_3    AB:     1.2     61 +-4
Tag:    APACHE_1_3_2    AB:     1.2     64 +-3

Tag:    APACHE_1_3_1    AB:     1.1     60 +-9  65.8 +-10
Tag:    APACHE_1_3_0    AB:     1.1     67 +-6
Tag:    APACHE_1_3b7    AB:     1.1     64 +-9
Tag:    APACHE_1_3b6    AB:     1.1     79 +-8

Dw
-- 
Dirk-Willem van Gulik



Re: Can AB be compared ?

Posted by di...@covalent.net.
What platform does not have writev() at the moment ?

Dw.
-- 
Dirk-Willem van Gulik


Re: Can AB be compared ?

Posted by David Reid <dr...@jetnet.co.uk>.
FWIW, is you don't have writev and are not using ssl, ab in the 1.3 tree is
broken and won't even compile at present :(

We really should fix it - but maybe the person who broke it could do that as
I'm sure they know what they were intending with the changes.

david

----- Original Message -----
From: <di...@covalent.net>
To: <de...@httpd.apache.org>
Sent: Wednesday, May 01, 2002 4:05 PM
Subject: Can AB be compared ?


>
> Just to have some fun - Below is the result of running a build of AB
> against the same apache 1.3.0 (stock)
>
> It is a simple loop - checkout against a tag; cd apache-1.3/src && cp
> Configuration.tmp && Configuration && ./Configure && make && cd support &&
> make) and then run 20 times ./ab -c 30 -n 10000 and take the last 25 runs.
>
> By the way - not a -single- tag failed to build on FreeBSD 5-Current and
> not a single tag needed a differnet config ! A record hard to beat by
> commercial software !
>
> For each is displayed to total avaerage time/connection; the 98%
> confidence interval (all in mSec) and the averegate for that version
> of AB.
>
> Note that typically (with eceptiosn early on) each version of AB produces
> run within the confidence interval - and that successive versions their
> confidence intertvals do generally not overlap (except early in the
> development) in the more recent series.
>
> So though this shows that comparing versions of AB from different versions
> is generally waranted (same mean, same SD) - for recent versions of
> 1.3 - and that comparisions between version no's is not a good idea.
>
> BUT at the same time - that has not always been the case - and that is is
> kind of a miracle (or a tribute to release management/fact that AB hardly
> ever changed) that comparison is possible. Also if you include 1.3c and
> 1.3a then you can conclude that these ranges match - thus from these
> measurements you would conclude that the code did not change substantially
> (it may of course do for keep alive or something).
>
> Dw.
>
>         Tag checout     #define in ab   mean/SD aver   CI
> Tag:    APACHE_1_3_24   AB:     1.3d    66 +-3  66.5 +-3
> Tag:    APACHE_1_3_23   AB:     1.3d    67 +-3
> Tag:    APACHE_1_3_22   AB:     1.3d    67 +-3
> Tag:    APACHE_1_3_21   AB:     1.3d    65 +-3
>
> Tag:    APACHE_1_3_20   AB:     1.3c    63 +-2  61.8 +-2
> Tag:    APACHE_1_3_19   AB:     1.3c    60 +-2
> Tag:    APACHE_1_3_18   AB:     1.3c    61 +-2
> Tag:    APACHE_1_3_17   AB:     1.3c    61 +-2
> Tag:    APACHE_1_3_16   AB:     1.3c    63 +-2
> Tag:    APACHE_1_3_15   AB:     1.3c    62 +-2
> Tag:    APACHE_1_3_14   AB:     1.3c    63 +-2
> Tag:    APACHE_1_3_13   AB:     1.3c    62 +-2
> Tag:    APACHE_1_3_12   AB:     1.3c    62 +-2
> Tag:    APACHE_1_3_11   AB:     1.3c    60 +-2
> Tag:    APACHE_1_3_10   AB:     1.3c    63 +-2
>
> Tag:    APACHE_1_3_9    AB:     1.3a    60 +-1  60.4 +-1
> Tag:    APACHE_1_3_8    AB:     1.3a    61 +-1
> Tag:    APACHE_1_3_7    AB:     1.3a    61 +-1
>
> Tag:    APACHE_1_3_6    AB:     1.3     64 +-2  65.2 +-2
> Tag:    APACHE_1_3_5    AB:     1.3     66 +-3
>
> Tag:    APACHE_1_3_4    AB:     1.2     61 +-3  62.9 +-3
> Tag:    APACHE_1_3_3    AB:     1.2     61 +-4
> Tag:    APACHE_1_3_2    AB:     1.2     64 +-3
>
> Tag:    APACHE_1_3_1    AB:     1.1     60 +-9  65.8 +-10
> Tag:    APACHE_1_3_0    AB:     1.1     67 +-6
> Tag:    APACHE_1_3b7    AB:     1.1     64 +-9
> Tag:    APACHE_1_3b6    AB:     1.1     79 +-8
>
> Dw
> --
> Dirk-Willem van Gulik
>
>
>