You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by di...@covalent.net on 2002/05/01 17:05:27 UTC
Can AB be compared ?
Just to have some fun - Below is the result of running a build of AB
against the same apache 1.3.0 (stock)
It is a simple loop - checkout against a tag; cd apache-1.3/src && cp
Configuration.tmp && Configuration && ./Configure && make && cd support &&
make) and then run 20 times ./ab -c 30 -n 10000 and take the last 25 runs.
By the way - not a -single- tag failed to build on FreeBSD 5-Current and
not a single tag needed a differnet config ! A record hard to beat by
commercial software !
For each is displayed to total avaerage time/connection; the 98%
confidence interval (all in mSec) and the averegate for that version
of AB.
Note that typically (with eceptiosn early on) each version of AB produces
run within the confidence interval - and that successive versions their
confidence intertvals do generally not overlap (except early in the
development) in the more recent series.
So though this shows that comparing versions of AB from different versions
is generally waranted (same mean, same SD) - for recent versions of
1.3 - and that comparisions between version no's is not a good idea.
BUT at the same time - that has not always been the case - and that is is
kind of a miracle (or a tribute to release management/fact that AB hardly
ever changed) that comparison is possible. Also if you include 1.3c and
1.3a then you can conclude that these ranges match - thus from these
measurements you would conclude that the code did not change substantially
(it may of course do for keep alive or something).
Dw.
Tag checout #define in ab mean/SD aver CI
Tag: APACHE_1_3_24 AB: 1.3d 66 +-3 66.5 +-3
Tag: APACHE_1_3_23 AB: 1.3d 67 +-3
Tag: APACHE_1_3_22 AB: 1.3d 67 +-3
Tag: APACHE_1_3_21 AB: 1.3d 65 +-3
Tag: APACHE_1_3_20 AB: 1.3c 63 +-2 61.8 +-2
Tag: APACHE_1_3_19 AB: 1.3c 60 +-2
Tag: APACHE_1_3_18 AB: 1.3c 61 +-2
Tag: APACHE_1_3_17 AB: 1.3c 61 +-2
Tag: APACHE_1_3_16 AB: 1.3c 63 +-2
Tag: APACHE_1_3_15 AB: 1.3c 62 +-2
Tag: APACHE_1_3_14 AB: 1.3c 63 +-2
Tag: APACHE_1_3_13 AB: 1.3c 62 +-2
Tag: APACHE_1_3_12 AB: 1.3c 62 +-2
Tag: APACHE_1_3_11 AB: 1.3c 60 +-2
Tag: APACHE_1_3_10 AB: 1.3c 63 +-2
Tag: APACHE_1_3_9 AB: 1.3a 60 +-1 60.4 +-1
Tag: APACHE_1_3_8 AB: 1.3a 61 +-1
Tag: APACHE_1_3_7 AB: 1.3a 61 +-1
Tag: APACHE_1_3_6 AB: 1.3 64 +-2 65.2 +-2
Tag: APACHE_1_3_5 AB: 1.3 66 +-3
Tag: APACHE_1_3_4 AB: 1.2 61 +-3 62.9 +-3
Tag: APACHE_1_3_3 AB: 1.2 61 +-4
Tag: APACHE_1_3_2 AB: 1.2 64 +-3
Tag: APACHE_1_3_1 AB: 1.1 60 +-9 65.8 +-10
Tag: APACHE_1_3_0 AB: 1.1 67 +-6
Tag: APACHE_1_3b7 AB: 1.1 64 +-9
Tag: APACHE_1_3b6 AB: 1.1 79 +-8
Dw
--
Dirk-Willem van Gulik
Re: Can AB be compared ?
Posted by di...@covalent.net.
What platform does not have writev() at the moment ?
Dw.
--
Dirk-Willem van Gulik
Re: Can AB be compared ?
Posted by David Reid <dr...@jetnet.co.uk>.
FWIW, is you don't have writev and are not using ssl, ab in the 1.3 tree is
broken and won't even compile at present :(
We really should fix it - but maybe the person who broke it could do that as
I'm sure they know what they were intending with the changes.
david
----- Original Message -----
From: <di...@covalent.net>
To: <de...@httpd.apache.org>
Sent: Wednesday, May 01, 2002 4:05 PM
Subject: Can AB be compared ?
>
> Just to have some fun - Below is the result of running a build of AB
> against the same apache 1.3.0 (stock)
>
> It is a simple loop - checkout against a tag; cd apache-1.3/src && cp
> Configuration.tmp && Configuration && ./Configure && make && cd support &&
> make) and then run 20 times ./ab -c 30 -n 10000 and take the last 25 runs.
>
> By the way - not a -single- tag failed to build on FreeBSD 5-Current and
> not a single tag needed a differnet config ! A record hard to beat by
> commercial software !
>
> For each is displayed to total avaerage time/connection; the 98%
> confidence interval (all in mSec) and the averegate for that version
> of AB.
>
> Note that typically (with eceptiosn early on) each version of AB produces
> run within the confidence interval - and that successive versions their
> confidence intertvals do generally not overlap (except early in the
> development) in the more recent series.
>
> So though this shows that comparing versions of AB from different versions
> is generally waranted (same mean, same SD) - for recent versions of
> 1.3 - and that comparisions between version no's is not a good idea.
>
> BUT at the same time - that has not always been the case - and that is is
> kind of a miracle (or a tribute to release management/fact that AB hardly
> ever changed) that comparison is possible. Also if you include 1.3c and
> 1.3a then you can conclude that these ranges match - thus from these
> measurements you would conclude that the code did not change substantially
> (it may of course do for keep alive or something).
>
> Dw.
>
> Tag checout #define in ab mean/SD aver CI
> Tag: APACHE_1_3_24 AB: 1.3d 66 +-3 66.5 +-3
> Tag: APACHE_1_3_23 AB: 1.3d 67 +-3
> Tag: APACHE_1_3_22 AB: 1.3d 67 +-3
> Tag: APACHE_1_3_21 AB: 1.3d 65 +-3
>
> Tag: APACHE_1_3_20 AB: 1.3c 63 +-2 61.8 +-2
> Tag: APACHE_1_3_19 AB: 1.3c 60 +-2
> Tag: APACHE_1_3_18 AB: 1.3c 61 +-2
> Tag: APACHE_1_3_17 AB: 1.3c 61 +-2
> Tag: APACHE_1_3_16 AB: 1.3c 63 +-2
> Tag: APACHE_1_3_15 AB: 1.3c 62 +-2
> Tag: APACHE_1_3_14 AB: 1.3c 63 +-2
> Tag: APACHE_1_3_13 AB: 1.3c 62 +-2
> Tag: APACHE_1_3_12 AB: 1.3c 62 +-2
> Tag: APACHE_1_3_11 AB: 1.3c 60 +-2
> Tag: APACHE_1_3_10 AB: 1.3c 63 +-2
>
> Tag: APACHE_1_3_9 AB: 1.3a 60 +-1 60.4 +-1
> Tag: APACHE_1_3_8 AB: 1.3a 61 +-1
> Tag: APACHE_1_3_7 AB: 1.3a 61 +-1
>
> Tag: APACHE_1_3_6 AB: 1.3 64 +-2 65.2 +-2
> Tag: APACHE_1_3_5 AB: 1.3 66 +-3
>
> Tag: APACHE_1_3_4 AB: 1.2 61 +-3 62.9 +-3
> Tag: APACHE_1_3_3 AB: 1.2 61 +-4
> Tag: APACHE_1_3_2 AB: 1.2 64 +-3
>
> Tag: APACHE_1_3_1 AB: 1.1 60 +-9 65.8 +-10
> Tag: APACHE_1_3_0 AB: 1.1 67 +-6
> Tag: APACHE_1_3b7 AB: 1.1 64 +-9
> Tag: APACHE_1_3b6 AB: 1.1 79 +-8
>
> Dw
> --
> Dirk-Willem van Gulik
>
>
>