You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by Roy Wilson <de...@bellatlantic.net> on 2000/08/22 17:56:13 UTC
Performance req's and using ab.c to see if they are, or can be, met
Picking up the nearly dead thread ...
PF: I believe you didn't really understand what we mean by performance...
I think you are/were partly correct.
<snip>
PF: What you did is not even a benchmark :) What you measured is the
latency
time between codeflux.com and jakarta.apache.org, over the INTERNET...
You are right. After conferring with codeflux.com and examining the ab.c
code, it seems that codeflux estimates server processing time by
subtracting time to establish a connection from total connection time. As
you so trenchantly observed :-), this includes network latency and there
is no way to separate actual server processing time and network latency
on the basis of the data collected by ab.c alone.
Of course, it's important to be able to compare what is actually measured
and what is claimed to be measured, since the two are sometimes
different. In response to an earlier post inquiring about the kinds of
measurement tools that might be used, ApacheBench was mentioned without
any qualification about how it should be used. The fact that codeflux.com
use of ab.c seems to confuse what is actually measured and what is
claimed to be measured suggests the need for explicit, up-front,
consideration.
PF: Well, as far as I can see, our benchmarks are more accurate than
yours.
It was not claimed that a benchmark was carried out: it was an example
intended to illustrate the perhaps obvious fact that - for a given
hardware configuration - "speed" is a function of both software and the
arrival rate and mix of requests that it must process. If all requests
are "small", then response time is "good": if some are "small" and some
are "huge", the "small" may suffer from having to wait. Yes?
<snip>
PF: Of course we cannot say which server or servlet engine is
faster, but we can make real-world examples and give a test methodology
that in your environment can be followed and used by you to determine
what's best for your needs.
Precisely my point. So, which "real-world" examples were used or are
planned to be used? I ask because what may seem like a real world example
to one person may not to another, as just illustrated by how you used
ab.c and how I/codeflux.com used it.
> QUESTION 2: Using the criteria given above [namely, a server response time
of 100 > ms or less], does this mean that the
> Apache web server is not a "good" web server? (I understand that, per
> the goals of the ASF, functionality should dominate performance).
PF: Given that the test metodology is wrong this question is not
pertinent.
Why not? Your methodology makes it possible to estimate the
webserver/container average response time W(server) under a given load
using ab.c. Webserver/container response time(s) is/are a lower bound on
the user response time.
> QUESTION 3: Does Catalina have any performance REQUIREMENTS? On page
> 12 of "Java Platform Performance", Steve Wilson and Jeff Kesselman of
> Sun suggest the need to "reach a consensus about the performance
> requirements for your system. If you don't, how will you know when
> your product is fast enough". If such advice doesn't apply here, why
> not?
PF: I don't agree with Wilson and Kesselman on that. Our requirements
<snip>
are to be "faster".
Ok, but (again) faster under what kind, and intensity, of load?
PF: And the codebase is clean enough to see where our
performance weak points are. We know how current VMs are working, what
is an overhead and what is not. I believe we can achieve very good
results only looking at our code...
I agree that looking at code can provide very good results: my only
question is, what number constitutes a very good result? If you believe
the information put out by Mercury Interactive (which claims to support
Apache as a webserver) in promoting their LoadTest software (endorsed by
LoudCloud), web users will only tolerate a response time of less than 8
seconds, whereas (based on some sort of survey of websites) the average
response time is 10 seconds.
Let me offer the following very crude example of the kind of information
I think is important to have sooner rather than later. Assume for the
sake of argument that the average network response time (including wait
time) is W(net) seconds. If you buy 8 seconds as a reasonable user
average response time goal (or, if you don't supply your own) this
leaves 8-W(net) seconds for Apache(Tomcat-4.0) [an important ambiguity
here, yes?]. So, for what mix of requests and at what arrival rates will
Apache(Tomcat-4.0) average response time W(server) <= 8-W(net) seconds?
Your example shows how to use ab.c to calculate X and W given N (the
number of requests) for ONE kind of request (ignoring request concurrency
for the time being). By specifying other kinds of requests and seeing how
the software performs as measured by ab.c (under the PF method :-)). we
can track how close we are to meeting the requirement and can decide
whether to investigate or not. If the measurements are bogus or the
inferences are bogus (as in the case of how codeflux interprets ab.c
output), that will become evident. I am proposing nothing more than an
early warning system to insure that Apache(Tomcat-4.0) meets a meaningful
set of "real-world" performance requirements.
<snip>
> Sorry for the preachiness.: please educate me.
You did that. Thanks.
--
Roy Wilson
E-mail: designrw@bellatlantic.net
Re: Measurement, etc.
Posted by "Craig R. McClanahan" <Cr...@eng.sun.com>.
Shawn McMurdo wrote:
> Thanks for the info.
> Can you give a reference on where to get ab?
> Thanks.
> Shawn
>
"ab" is in the Apache web server source tree, in directory "src/support". It ends up
under $APACHE_HOME/bin for you (at least on Unix systems).
Craig McClanahan
Re: Measurement, etc.
Posted by Shawn McMurdo <sh...@lutris.com>.
Thanks for the info.
Can you give a reference on where to get ab?
Thanks.
Shawn
Costin Manolache wrote:
> Roy Wilson wrote:
>
> > Hi,
> > I'm interested in some of the kinds of questions that you are interested
> > in. As time permits, I'll do some "processing" of my own in connection
> > with them and ship back some questions, speculations and maybe some
> > tentative results. Others might find the process interesting or be
> > willing to take a friendly poke to move it along.
>
> FYI:
>
> I'm doing mostly comparative tests ( "before" and "after"
> each change that may impact the performance).
>
> I use "ab" with a "warm-up" of 1000 requests and then I do few
> 10.000 sets. I do the tests with 1, 40, 80 concurent clients ( and
> sometimes with 200 ).
>
> That means: "ab -n 10000 -c 80 http://... "
>
> I also do the tests with JDK1.2.2, JDK1.3 (sun) and JDK1.3( ibm ).
> Sometimes I use JDK1.1.8.
>
> The machine is a Linux with celeron 366MHz ( my laptop),
> ( 362 BogoMIPS ). I also have a Ultra 1 and a Pentium II
> ( I plan to have Linux, FreeBSD, NT - as soon as I'll be able to install it).
>
> I also use OptimizeIt with "ab -n 500 -c 1 " to evaluate
> the number of objects allocated / request and to find
> hotspots.
>
> Sometimes I use verbose GC to gather data about the time
> spent in GC and the number of requests/GC ( it's less important
> with tomcat 3.2, and not usefull with tomcat 3.1, but it used
> to be a big factor ).
>
> The numbers I look at are the average and ( more important
> for me) the maximum response time ( if it is > 5 secs means
> trubles - no normal person waits this long for a web page - if
> you add the graphics in it's 10..15 sec).
>
> I use 2 servlets ( HelloWorld using OutputStream and Writer - but
> not the one in examples because it does a lot of extra stuff with
> the message file), and one simple JSP page. Since I'm interested
> in servlet overhead the servlet doesn't do any allocation or
> anything complex.
> For API performance I use simple servlets doing a particular call
> in a loop. So far I did very little work on this, probably more for ...
> 3.4.
>
> I also do the tests with various amount of output ( few bytes, 100,
> 1K, 10K ).
>
> If you read so far you'll notice it is a lot of work for evenings and
> weekends, so I don't run all of them everytime - I used to when
> I had more time.
>
> Costin
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: tomcat-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: tomcat-dev-help@jakarta.apache.org
--
Shawn McMurdo mailto:shawn@lutris.com
Lutris Technologies http://www.lutris.com
Enhydra.Org http://www.enhydra.org
Re: Measurement, etc.
Posted by Costin Manolache <cm...@yahoo.com>.
Roy Wilson wrote:
> Hi,
> I'm interested in some of the kinds of questions that you are interested
> in. As time permits, I'll do some "processing" of my own in connection
> with them and ship back some questions, speculations and maybe some
> tentative results. Others might find the process interesting or be
> willing to take a friendly poke to move it along.
FYI:
I'm doing mostly comparative tests ( "before" and "after"
each change that may impact the performance).
I use "ab" with a "warm-up" of 1000 requests and then I do few
10.000 sets. I do the tests with 1, 40, 80 concurent clients ( and
sometimes with 200 ).
That means: "ab -n 10000 -c 80 http://... "
I also do the tests with JDK1.2.2, JDK1.3 (sun) and JDK1.3( ibm ).
Sometimes I use JDK1.1.8.
The machine is a Linux with celeron 366MHz ( my laptop),
( 362 BogoMIPS ). I also have a Ultra 1 and a Pentium II
( I plan to have Linux, FreeBSD, NT - as soon as I'll be able to install it).
I also use OptimizeIt with "ab -n 500 -c 1 " to evaluate
the number of objects allocated / request and to find
hotspots.
Sometimes I use verbose GC to gather data about the time
spent in GC and the number of requests/GC ( it's less important
with tomcat 3.2, and not usefull with tomcat 3.1, but it used
to be a big factor ).
The numbers I look at are the average and ( more important
for me) the maximum response time ( if it is > 5 secs means
trubles - no normal person waits this long for a web page - if
you add the graphics in it's 10..15 sec).
I use 2 servlets ( HelloWorld using OutputStream and Writer - but
not the one in examples because it does a lot of extra stuff with
the message file), and one simple JSP page. Since I'm interested
in servlet overhead the servlet doesn't do any allocation or
anything complex.
For API performance I use simple servlets doing a particular call
in a loop. So far I did very little work on this, probably more for ...
3.4.
I also do the tests with various amount of output ( few bytes, 100,
1K, 10K ).
If you read so far you'll notice it is a lot of work for evenings and
weekends, so I don't run all of them everytime - I used to when
I had more time.
Costin
Measurement, etc.
Posted by Roy Wilson <de...@bellatlantic.net>.
Hi,
I'm interested in some of the kinds of questions that you are interested
in. As time permits, I'll do some "processing" of my own in connection
with them and ship back some questions, speculations and maybe some
tentative results. Others might find the process interesting or be
willing to take a friendly poke to move it along.
Roy
--
Roy Wilson
E-mail: designrw@bellatlantic.net
>>>>>>>>>>>>>>>>>> Original Message <<<<<<<<<<<<<<<<<<
On 8/22/00, 2:06:56 PM, "yhs@mimic.onesourcecorp.com"
<yh...@mimic.onesourcecorp.com> wrote regarding Re: Whoops - I don't want
20MB yet!:
> On Tue, 22 Aug 2000, Roy Wilson wrote:
> > Ys,
> > Is it the benchmarking software that's 10-20MB, the data generated,
or
> > the sum of the two?
> > Is your test designed to stress the components (a synthetic
benchmark)
> > or does it generate the kinds of requests users make against your site?
> > Do you collect a ("small") HTTP log of activity against the server?
Some
> > people claim these can be helpful in describing a natural workload model.
> >
> > Roy
> > --
> > Roy Wilson
> > E-mail: designrw@bellatlantic.net
> >
> its the data generated. i use apache's jbench (not great
> but it works) for the benchmarking. i generate typical user requests
(with
> 20% extra workload factored in for randomness). the synthetic benchmarks
> dont work too well..whats the point of stressing I/O when you know that
> hdparm -Tt can generate better numbers on disk throughput than your
> benchmark ever would ? (typically i get 20 megabytes/sec off a quantum
> 30gig ATA66 with half a meg of disk cache).
> HTTP/application logs in general are too big..the graphical plots
> generated by jbench are usually ok along with a dump of system
> activity. note that most of my tests are focused on jitter and mean
> throughput rather than the usually "my server is bigger than your
> server" crap that comes out of winbench or whatever. in practice,
> jitter is more useful identifying the problems with the system under load
> and seeing if the system will scale under load than beating a system to
> death and stating - look my XXXXMHz server blows away your YYYYMHz
server.
> In short, im not interested in what happens when 600 users hammer
> a 386SX-16 to death over a 9600bps modem (or something equally silly)
..im
> more interested in what happens when 10 or a 100 concurrent users hit a
> dual P-III 650 with an average of 20 megabyte/sec of disk throughput, T3
> link and a gig of memory...in other words, whats the worst case you
expect
> on average with this setup ? and why ? and whats causing the worst case
> ? whats the jitter like ? whats the best case ? why ? what can i do to
> improve the worst case ? whats the effect of cache size ? what does load
> balancing get me ? how many JVMs are optimal ? how much cpu/memory/io is
> used up by which components ?
> Most benchmarks are artifical - theres no right or wrong way of
> doing it...ask yourself what works for you ? then do it with a setup you
> actually *use* as opposed to some setup which is never around when you
> need it. do i care what tomcats throughput is on a Cray SV-1 ? no. Do i
> care what its throughput is on a dual piii ? sure.
> -Ys-
> yhs@mimic.onesourcecorp.com
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: tomcat-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: tomcat-dev-help@jakarta.apache.org
Re: Whoops - I don't want 20MB yet!
Posted by "yhs@mimic.onesourcecorp.com" <yh...@mimic.onesourcecorp.com>.
On Tue, 22 Aug 2000, Roy Wilson wrote:
> Ys,
> Is it the benchmarking software that's 10-20MB, the data generated, or
> the sum of the two?
> Is your test designed to stress the components (a synthetic benchmark)
> or does it generate the kinds of requests users make against your site?
> Do you collect a ("small") HTTP log of activity against the server? Some
> people claim these can be helpful in describing a natural workload model.
>
> Roy
> --
> Roy Wilson
> E-mail: designrw@bellatlantic.net
>
its the data generated. i use apache's jbench (not great
but it works) for the benchmarking. i generate typical user requests (with
20% extra workload factored in for randomness). the synthetic benchmarks
dont work too well..whats the point of stressing I/O when you know that
hdparm -Tt can generate better numbers on disk throughput than your
benchmark ever would ? (typically i get 20 megabytes/sec off a quantum
30gig ATA66 with half a meg of disk cache).
HTTP/application logs in general are too big..the graphical plots
generated by jbench are usually ok along with a dump of system
activity. note that most of my tests are focused on jitter and mean
throughput rather than the usually "my server is bigger than your
server" crap that comes out of winbench or whatever. in practice,
jitter is more useful identifying the problems with the system under load
and seeing if the system will scale under load than beating a system to
death and stating - look my XXXXMHz server blows away your YYYYMHz server.
In short, im not interested in what happens when 600 users hammer
a 386SX-16 to death over a 9600bps modem (or something equally silly) ..im
more interested in what happens when 10 or a 100 concurrent users hit a
dual P-III 650 with an average of 20 megabyte/sec of disk throughput, T3
link and a gig of memory...in other words, whats the worst case you expect
on average with this setup ? and why ? and whats causing the worst case
? whats the jitter like ? whats the best case ? why ? what can i do to
improve the worst case ? whats the effect of cache size ? what does load
balancing get me ? how many JVMs are optimal ? how much cpu/memory/io is
used up by which components ?
Most benchmarks are artifical - theres no right or wrong way of
doing it...ask yourself what works for you ? then do it with a setup you
actually *use* as opposed to some setup which is never around when you
need it. do i care what tomcats throughput is on a Cray SV-1 ? no. Do i
care what its throughput is on a dual piii ? sure.
-Ys-
yhs@mimic.onesourcecorp.com
Whoops - I don't want 20MB yet!
Posted by Roy Wilson <de...@bellatlantic.net>.
Ys,
Is it the benchmarking software that's 10-20MB, the data generated, or
the sum of the two?
Is your test designed to stress the components (a synthetic benchmark)
or does it generate the kinds of requests users make against your site?
Do you collect a ("small") HTTP log of activity against the server? Some
people claim these can be helpful in describing a natural workload model.
Roy
--
Roy Wilson
E-mail: designrw@bellatlantic.net
>>>>>>>>>>>>>>>>>> Original Message <<<<<<<<<<<<<<<<<<
On 8/22/00, 12:44:38 PM, Roy Wilson <de...@bellatlantic.net> wrote
regarding Re: Performance req's and using ab.c to see if they are, or can
be, met:
> Ys,
> I'm certainly interested. Maybe others have data too.
> Roy
> --
> Roy Wilson
> E-mail: designrw@bellatlantic.net
> >>>>>>>>>>>>>>>>>> Original Message <<<<<<<<<<<<<<<<<<
> On 8/22/00, 12:29:19 PM, "yhs@mimic.onesourcecorp.com"
> <yh...@mimic.onesourcecorp.com> wrote regarding Re: Performance req's and
> using ab.c to see if they are, or can be, met:
> > On Tue, 22 Aug 2000, Roy Wilson wrote:
> > > Picking up the nearly dead thread ...
> > >
> > > Precisely my point. So, which "real-world" examples were used or are
> > > planned to be used? I ask because what may seem like a real world example
> > > to one person may not to another, as just illustrated by how you used
> > > ab.c and how I/codeflux.com used it.
> > ><SNIP>
> > > Roy Wilson
> > > E-mail: designrw@bellatlantic.net
> > i dont know if this is useful but i benchmark using the following
> > configuration :
> > Cluster/System gig ether link gig ether link
> > under test -------------- [ gig switch ] ---------------
> System
> > running
> > test.
> > this knocks out issues of internet latency and stuff right away since its
> > a dedicated switch and bandwidth problems are eliminated over gig ether
> > anyway. i've got a whole stack of benchmark results here to show tomcat
> > scaling (its pretty good at scaling too) nicely under load. i test it out
> > with a servlet that does heavy database read/writes as well as heavy
> > number crunching (load average on the test system is usually around 40 --
> > if its a single system with around 150+ processess). On average ive
> > observed mysql taking up 30% of the CPU, 68% taken up by the JVMs/tomcat
> > and 2% by the httpd processes...of course thats just my app. i can send
> > you the benchmarks if anyone is interested..they are pretty big (around
> > 10-20 megs i think) so give me an ftp site to dump em on...or just a ssh
> > link so i can scp em over as a tarball.
> > -Ys-
> > yhs@mimic.onesourcecorp.com
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: tomcat-dev-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: tomcat-dev-help@jakarta.apache.org
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: tomcat-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: tomcat-dev-help@jakarta.apache.org
Re: Performance req's and using ab.c to see if they are, or can be,
met
Posted by Roy Wilson <de...@bellatlantic.net>.
Ys,
I'm certainly interested. Maybe others have data too.
Roy
--
Roy Wilson
E-mail: designrw@bellatlantic.net
>>>>>>>>>>>>>>>>>> Original Message <<<<<<<<<<<<<<<<<<
On 8/22/00, 12:29:19 PM, "yhs@mimic.onesourcecorp.com"
<yh...@mimic.onesourcecorp.com> wrote regarding Re: Performance req's and
using ab.c to see if they are, or can be, met:
> On Tue, 22 Aug 2000, Roy Wilson wrote:
> > Picking up the nearly dead thread ...
> >
> > Precisely my point. So, which "real-world" examples were used or are
> > planned to be used? I ask because what may seem like a real world example
> > to one person may not to another, as just illustrated by how you used
> > ab.c and how I/codeflux.com used it.
> ><SNIP>
> > Roy Wilson
> > E-mail: designrw@bellatlantic.net
> i dont know if this is useful but i benchmark using the following
> configuration :
> Cluster/System gig ether link gig ether link
> under test -------------- [ gig switch ] ---------------
System
> running
> test.
> this knocks out issues of internet latency and stuff right away since its
> a dedicated switch and bandwidth problems are eliminated over gig ether
> anyway. i've got a whole stack of benchmark results here to show tomcat
> scaling (its pretty good at scaling too) nicely under load. i test it out
> with a servlet that does heavy database read/writes as well as heavy
> number crunching (load average on the test system is usually around 40 --
> if its a single system with around 150+ processess). On average ive
> observed mysql taking up 30% of the CPU, 68% taken up by the JVMs/tomcat
> and 2% by the httpd processes...of course thats just my app. i can send
> you the benchmarks if anyone is interested..they are pretty big (around
> 10-20 megs i think) so give me an ftp site to dump em on...or just a ssh
> link so i can scp em over as a tarball.
> -Ys-
> yhs@mimic.onesourcecorp.com
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: tomcat-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: tomcat-dev-help@jakarta.apache.org
Re: Performance req's and using ab.c to see if they are, or can be,
met
Posted by "yhs@mimic.onesourcecorp.com" <yh...@mimic.onesourcecorp.com>.
On Tue, 22 Aug 2000, Roy Wilson wrote:
> Picking up the nearly dead thread ...
>
> Precisely my point. So, which "real-world" examples were used or are
> planned to be used? I ask because what may seem like a real world example
> to one person may not to another, as just illustrated by how you used
> ab.c and how I/codeflux.com used it.
><SNIP>
> Roy Wilson
> E-mail: designrw@bellatlantic.net
i dont know if this is useful but i benchmark using the following
configuration :
Cluster/System gig ether link gig ether link
under test -------------- [ gig switch ] --------------- System
running
test.
this knocks out issues of internet latency and stuff right away since its
a dedicated switch and bandwidth problems are eliminated over gig ether
anyway. i've got a whole stack of benchmark results here to show tomcat
scaling (its pretty good at scaling too) nicely under load. i test it out
with a servlet that does heavy database read/writes as well as heavy
number crunching (load average on the test system is usually around 40 --
if its a single system with around 150+ processess). On average ive
observed mysql taking up 30% of the CPU, 68% taken up by the JVMs/tomcat
and 2% by the httpd processes...of course thats just my app. i can send
you the benchmarks if anyone is interested..they are pretty big (around
10-20 megs i think) so give me an ftp site to dump em on...or just a ssh
link so i can scp em over as a tarball.
-Ys-
yhs@mimic.onesourcecorp.com