You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by Roy Wilson <de...@bellatlantic.net> on 2000/11/08 13:45:55 UTC
Msg formatting problem: trying again
I apologize for the critical tone (but also perhaps ignorance)of my
[now next to the] last post. I've got the flu or something. Yecch.
Here's the data I collected after running new, slightly modified,
scripts (see attached) for which RequestInfoExample is requested by ab
100,000 times on a Celeron 400Mz, 256MB SD100 memory, WD Caviar disk,
Redhat6.1, with almost nothing else going on:
C
T secs
X requests/sec
Connect avg (ms)
Processing avg (ms)
Cpu busy %
Cpu Ovhead (mins)
Cpu Ovhead (secs)
T - Ovhead (secs)
Cpu per request (ms)
1
2912.78
34.33
0
28
100
0.87
52.2
2860.58
28.6
10
1495.12
66.88
0
149
100
0.92
55.2
1439.92
14.4
20
1488.7
67.17
0
297
100
0.98
58.8
1429.9
14.3
30
1503.03
66.53
0
450
100
1.12
67.2
1435.83
14.4
40
1591.7
62.83
0
636
100
1.31
78.6
1513.1
15.1
The earlier results involving the HelloWorldExample servlet were
affected by variability which I think/hope is averaged out by the large
number of requests made. The disk, as expected shows only about 1.8
requests per second, and is not the bottleneck anyway (as expected).
My earlier results concerning HelloWorldExample were based on a smaller
N and were not adjusted to account for CPU overhead. I'll re-run them
ASAP.
In the earlier results, thread queue locking was observed. Craig noted
that the number of background threads is set to 75 in the config file.
With that in mind, I made sure that the ab script ran with 70 or fewer
concurrent requests. Here's what happened:
catalina_log*.txt shows "HttpProcessor starting background
thread[8080][i]" where i runs from 0/1 up to 50.
catalina.out contains a full thread dump (50 on down), one of which
involves the message "Waiting to be notified HttpProcessor[8080][43]".
This appears to be the result of a segmentation violation involving
HttpProcessor[8080][50] that is also reported. Of course, ab barfs from
C =3D 50 onward. As I recall, SEGV was also the cause of the thread dump=
when I ran HelloWorldExample. So, the thread dump seems to be a
repeatable phenomena (on my machine), apparently a function of N & C.
QUESTION 1: Does segmentation violation occur on other machines? If so,
is it a problem? If not, why not?
QUESTION 2: What do the thruput and average processing times mean? Are
they "good", "bad", or what?
QUESTION 3: Since the scripts can be modified to run any servlet, how
about a "realistic" one as Costin suggested in his ApacheCon
presentation (unless the examples are "representative").
(Hack) Roy
Roy Wilson
E-mail: designrw@bellatlantic.net
Re: Test hardware != User hardware =>applicability problems?
Posted by cm...@yahoo.com.
> criteria on a limited set of requests. So, one more time: What workload
> is Tomcat expected to run, and what will TC thruput and response be
> running that workload on a particular architecture? Or, since TC is free,
> this is up to the user to simply try it and see?
I don't think you can ( or should ) predict what the user will run, but
how much will tomcat add.
It is important to know that at x req/s tomcat is adding
y ms for each request, z ms for an internal dispatch, etc.
( of course, on average )
Costin
Test hardware != User hardware =>applicability problems?
Posted by Roy Wilson <de...@bellatlantic.net>.
See below.
Original Message dated 11/8/00, 12:55:34 PM
Author: "Craig R. McClanahan" <Cr...@eng.sun.com>
Re: Re: Msg formatting problem: trying again:
Craig: That's right ... once a servlet is loaded it stays in memory, so
there would be little or no I/O in this kind of test scenario.
Roy: In the old days we called it cacheing, yes? :-)
Craig: NOTE: Of course, that is one of the issues that leads to caution
in interpreting benchmark results as *absolute* indicators of performance
time and see what happens.
Roy: There is a school of thought that varying multiple factors and then
analyzing their combined effects on performance indicators is a good way
to go unless it is known in advance that interactions are not important
It was used at IBM for Operating System 360 (ugh) parameter tuning.
Craig: ... you might want to try at least the 1.2.2 and 1.3.0 releases
from Sun (on the same hardware) to compare. That would also give some
interesting information on how the new HotSpot in 1.3 (and in particular
the "server version") affects things.
Roy: Will do ASAIC
Craig: One thing that will help in this regard is that Sun is making a
machine with some serious hardware (six processors, 3GB of memory, scads
of disk) available for performance tuning. That should help us shake out
thread-related bugs and performance problems pretty quickly :-).
Roy: But, isn't this optimizing for the SERIOUS machine architecture? Is
there an implicit target architecture for TC? In the past, it would not
have been the SERIOUS machine, would it? (Is the SERIOUS machine part of
the LinuxDevelopmentLab?) How would lab performance be translated to
outside-the-lab performance? (BTW, I think it can, via modeling along the
lines suggested in my presentation that I posted a couple of weeks back).
Roy: And, as all commercial benchmarkers know, it is possible to tweak
almost any system to get performance that satisfies some benchmark
criteria on a limited set of requests. So, one more time: What workload
is Tomcat expected to run, and what will TC thruput and response be
running that workload on a particular architecture? Or, since TC is free,
this is up to the user to simply try it and see?
Roy
Re: Msg formatting problem: trying again
Posted by "Craig R. McClanahan" <Cr...@eng.sun.com>.
Roy Wilson wrote:
>
>
> I apologize for the critical tone (but also perhaps ignorance)of my
> [now next to the] last post. I've got the flu or something. Yecch.
>
Not a problem.
> Here's the data I collected after running new, slightly modified,
> scripts (see attached) for which RequestInfoExample is requested by ab
> 100,000 times on a Celeron 400Mz, 256MB SD100 memory, WD Caviar disk,
> Redhat6.1, with almost nothing else going on:
>
>
C T secs X Connect Processing Cpu Cpu Cpu T - Cpu per
requests/sec avg avg (ms) busy Ovhead Ovhead Ovhead request
(ms) % (mins) (secs) (secs) (ms)
1 2912.78 34.33 0 28 100 0.87 52.2 2860.58 28.6
10 1495.12 66.88 0 149 100 0.92 55.2 1439.92 14.4
20 1488.7 67.17 0 297 100 0.98 58.8 1429.9 14.3
30 1503.03 66.53 0 450 100 1.12 67.2 1435.83 14.4
40 1591.7 62.83 0 636 100 1.31 78.6 1513.1 15.1
>
> The earlier results involving the HelloWorldExample servlet were
> affected by variability which I think/hope is averaged out by the large
> number of requests made. The disk, as expected shows only about 1.8
> requests per second, and is not the bottleneck anyway (as expected).
>
That's right ... once a servlet is loaded it stays in memory, so there
would be little or no I/O in this kind of test scenario.
NOTE: Of course, that is one of the issues that leads to caution in
interpreting benchmark results as *absolute* indicators of performance
-- this is clearly not going to be representative of the performance of
a servlet that heavily accesses a database, for example. However, it is
very useful in *relative* performance measurement. The standard
practices of scientific investigation apply ... tweak one variable at a
time and see what happens.
> My earlier results concerning HelloWorldExample were based on a smaller
> N and were not adjusted to account for CPU overhead. I'll re-run them
> ASAP.
>
> In the earlier results, thread queue locking was observed. Craig noted
> that the number of background threads is set to 75 in the config file.
> With that in mind, I made sure that the ab script ran with 70 or fewer
> concurrent requests. Here's what happened:
>
> catalina_log*.txt shows "HttpProcessor starting background
> thread[8080][i]" where i runs from 0/1 up to 50.
>
> catalina.out contains a full thread dump (50 on down), one of which
> involves the message "Waiting to be notified HttpProcessor[8080][43]".
> This appears to be the result of a segmentation violation involving
> HttpProcessor[8080][50] that is also reported. Of course, ab barfs from
> C =3D 50 onward. As I recall, SEGV was also the cause of the thread
> dump=
>
> when I ran HelloWorldExample. So, the thread dump seems to be a
> repeatable phenomena (on my machine), apparently a function of N & C.
>
> QUESTION 1: Does segmentation violation occur on other machines? If so,
> is it a problem? If not, why not?
>
In a pure Java program like Tomcat 4.0-m4, any SEGV violation is always,
100% of the time, a bug in the JVM (or the underlying OS -- and Linux
thread libraries have been known to have issues). I don't recall which
JVM you are running, but you might want to try at least the 1.2.2 and
1.3.0 releases from Sun (on the same hardware) to compare. That would
also give some interesting information on how the new HotSpot in 1.3
(and in particular the "server version") affects things.
> QUESTION 2: What do the thruput and average processing times mean? Are
> they "good", "bad", or what?
>
As discussed above, I would be cautious about treating these numbers as
absolute indicators of expected performance. What we are after is
identifying places where we can change the code and improve the simple
cases, with the expectation that such improvements will also help in
more realistic cases (even though they will have less impact on the
overall performance, because disk times for database access will tend to
dominate).
When designing a server platform, most people aim for goals like this:
* Maximum processing times should not have huge
spikes versus average
* Graceful degredation (ideally linear) as the load
is increased
Deviations from these goals usually indicate a bottleneck that is being
bumped into, and provides opportunities for tuning.
One thing that will help in this regard is that Sun is making a machine
with some serious hardware (six processors, 3GB of memory, scads of
disk) available for performance tuning. That should help us shake out
thread-related bugs and performance problems pretty quickly :-).
> QUESTION 3: Since the scripts can be modified to run any servlet, how
> about a "realistic" one as Costin suggested in his ApacheCon
> presentation (unless the examples are "representative").
>
I agree with this, and would word it slightly differently: the only
thing a "Hello, World" benchmark predicts is how fast your application
can say "Hello, World". :-)
If you're after a performance predictor for a real application, you need
a test suite that simulates the actual behavior of the system as closely
as possible -- particularly where the stresses are bumping up against
bottlenecks. For example, in the test suite you've quoted the system
runs at 100% CPU, so switching to a faster processor (all things else
being equal) would probably make everything run faster -- but adding a
faster disk (or even more memory, if you're not swapping) would not
matter at all.
Benchmarks for performance prediction are an interesting art.
> (Hack) Roy
>
> Roy Wilson
> E-mail: designrw@bellatlantic.net
>
> ----------------------------------------------------------------
> # Draft script by Roy Wilson used on Redhat6.1 system (bash)
> # Assume ONLY HelloWorld servlet being used: better to have a set, but
> what?
> #
> # To promote comparability, run script immediately after a cold boot
> echo Run script only after a cold boot to make results comparable
> echo When done please and post the summary file to Tomcat dev list
> echo Usage:
> echo sh tcscript servlet-name max-concurrency increment
> echo Example:
> echo sh tcscript HelloWorldExample 100 10
> rm -f summary
> sh tcscript 1 $1
> load="$3"
> while [ "$load" -le "$2" ]
> do
> echo "Starting ab with load = " $load
> sh tcscript $load $1
> load=$(($load+$3))
> done
>
>
>
> ----------------------------------------------------------------
> # Roy Wilson 11/2/00
>
> echo "Starting ab load test with C = " $1 "and servlet = "$2
> rm -f $1$2.sardata
> # Next 2 lines may need adjustment on account of OS differences
> # Start with a fresh temporary system accounting log file
> date >> summary
> cp /var/log/savacct /var/log/tcacct
> /usr/sbin/accton /var/log/tcacct
> # sample CPU and disk usage with 1 second resolution
> (sar -u -b -o $1$2.sardata 1 > /dev/null )&
> sarpid=$!
> # modify path in next line for use of servlet
> /usr/sbin/ab -n 100000 -c $1 http://ALFRED:8080/examples/servlet/$2 >>
> summary
> kill -s TERM ${sarpid}
> # Next line may need modification: Turns off system accounting
> /usr/sbin/accton
> # peel off raw data, keep sar summary line
> sar -f $1$2.sardata 1 | grep Average >> summary
> # collect CPU per minute data from system accounting package
> /usr/sbin/sa -a /var/log/tcacct >> summary
> rm -f $1$2.sardata
> echo "Finished ab load test with C = "$1 "and servlet = "$2
>
> ----------------------------------------------------------------
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: tomcat-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: tomcat-dev-help@jakarta.apache.org
>