You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by Roy Wilson <de...@bellatlantic.net> on 2000/08/15 20:42:05 UTC
Catalina/Tomcat performance/functionality
Since I am not (yet) a Tomcat/Catalina developer, there may not be much
reason to take these performance related questions too seriously, if at
all. I hope, however, they might prove useful.
In the Catalina proposal, the following comments were made concerning
performance.
"No formal benchmarks have been published by the Tomcat development team
describing the performance of Tomcat 3.x or Catalina."
QUESTION 1: What is meant by "formal" benchmark? See below on data
collected (but not on Tomcat) using ApacheBench: is this a formal
benchmark, an informal benchmark, or neither?
"(In addition, I would take any benchmark that purports to predict how
*your* application will perform on a particular server with a very large
grain of salt :-)."
I agree: this is a big can of worms (to be opened later, if ever).
"However, simple tests with tools like ApacheBench and Microsoft's stress
test indicate that Catalina is faster than Tomcat on serving static
resources (substantially faster if caching is enabled, basically on par
with Apache itself), and within a few percentage points on serving
"Hello, World" type servlets and JSP pages."
It would be nice to see the results of at least the ApacheBench test
since ApacheBench is potentially available to all developers via the
ApacheBench Gateway at codeflux.com.
I ran ab gateway tool using jakarta.apache.org as the server hostname and
Apache/1.3.13-dev as the server software. According to the ab gateway
documentation, "a good web server can process a static HTML page in under
10 ms, ..., with overall turnaround time consistently well under 100 ms".
As a benchmarking exercise, I used the following two http addresses. I
believe these are static pages, but I didn't bother to try to determine
this one way or the other.
(1)/tomcat/index.html
(2)/catalina-proposal.html?rev1.1 (URL is no longer current)
The first URL produced an average per request response time of between
250 and 260 ms, even with a (request) concurrency level of 5.
QUESTION 2: Using the criteria given above, does this mean that the
Apache web server is not a "good" web server? (I understand that, per the
goals of the ASF, functionality should dominate performance).
QUESTION 3: Does Catalina have any performance REQUIREMENTS? On page 12
of "Java Platform Performance", Steve Wilson and Jeff Kesselman of Sun
suggest the need to "reach a consensus about the performance requirements
for your system. If you don't, how will you know when your product is
fast enough". If such advice doesn't apply here, why not?
The second URL produced an average response time between 923 and 3440 ms.
Increased request concurrency leads, in the first case, to better
throughput, presumably because of caching of system resources. In the
second case, increased request concurrency actually leads to a decrease
in throughput. One or more underlying hardware/software/data resources
are being saturated by this request. I've attached a StarOffice5.2 Calc
(plus an Excel 95) spreadsheet with the details.
QUESTION 4: will Catalina traffic look more like URL1 or URL2 or some mix
of the two or some other mix.
QUESTION 5: what will be the consequences of answering question 4 for
user-level performance?
"Only modest efforts to tune Catalina have taken place to date (compared
to the substantial -- and successful -- tuning efforts expended between
Tomcat 3.1 and 3.2), so it can be assumed that there is still significant
room for improvement through the usual techniques of isolating hot spots
and optimizing them."
While I agree that this is the usual technique, there are those who
suggest it may not be enough. Some suggest that performance is
functionality: if it takes 10 minutes to login to a system, it isn't
functional in an meaningful sense (I am not predicting that such an event
would occur, only noting that it did occur elsewhere). In the horror
story I mentioned in an earlier post, there were no REQUIREMENTS up
front. It was assumed that intelligent design, coding, tuning. and a
bigger pipe/hammer/etc. would be enough. They weren't. Sorry for the
preachiness.: please educate me.
If there is a version of Tomcat or Catalina that is up that can be
accessed using the codeflux.com ApacheBench Gateway, I am happy to
volunteer for some additional monkey-test work :-) like that done in
producing the data shown in the attachment.
Roy
--
Roy Wilson
E-mail: designrw@bellatlantic.net
Re: Catalina/Tomcat performance/functionality
Posted by "Pier P. Fumagalli" <pi...@apache.org>.
Roy Wilson wrote:
>
> Since I am not (yet) a Tomcat/Catalina developer, there may not be
> much reason to take these performance related questions too seriously,
> if at all. I hope, however, they might prove useful.
I believe you didn't really understand what we mean by performance...
> In the Catalina proposal, the following comments were made concerning
> performance.
>
> "No formal benchmarks have been published by the Tomcat development
> team describing the performance of Tomcat 3.x or Catalina."
Yes, and that's true. I know that some of us ran some performance tests
stressing the different web servers and different http stacks, but we
don't yet have a formal performance report.
> QUESTION 1: What is meant by "formal" benchmark? See below on data
> collected (but not on Tomcat) using ApacheBench: is this a formal
> benchmark, an informal benchmark, or neither?
What you did is not even a benchmark :) What you measured is the latency
time between codeflux.com and jakarta.apache.org, over the INTERNET...
Just a simple example: if I try to run ApacheBench on my machine using
as URL my host name (http://www.betaversion.org/) I get these results:
Server Software:
Apache/1.3.12
Server Hostname: www.betaversion.org
Server Port: 80
Document Path: /
Document Length: 1358 bytes
Concurrency Level: 1
Time taken for tests: 0.384 seconds
Complete requests: 10
Failed requests: 0
Total transferred: 17810 bytes
HTML transferred: 13580 bytes
Requests per second: 26.04
Transfer rate: 46.38 kb/s received
Running the _same_ test from the codeflux gateway I get:
Server Software: Apache/1.3.12
Server Hostname: www.betaversion.org
Server Port: 80
Document Path: /
Document Length: 1358 bytes
Concurrency Level: 1
Time taken for tests: 6.470 seconds
Complete requests: 10
Failed requests: 0
Total transferred: 17810 bytes
HTML transferred: 13580 bytes
Requests per second: 1.55
Transfer rate: 2.75 kb/s received
WHHHATTTTT??? Am I nuts? from 26 requests per seconds to 1.55? What's
wrong with my site? Nothing it's my test methodology. In the second case
I am measuring network latency.
> "(In addition, I would take any benchmark that purports to predict how
> *your* application will perform on a particular server with a very
> large grain of salt :-)."
Well, as far as I can see, our benchmarks are more accurate than yours.
> I agree: this is a big can of worms (to be opened later, if ever).
No, it's not... Performance testin (as Mike Pogue from IBM docet) is a
very intricate matter, but certainly, in some situations, can be
resolved. Of course we cannot say which server or servlet engine is
faster, but we can make real-world examples and give a test methodology
that in your environment can be followed and used by you to determine
what's best for your needs.
> "However, simple tests with tools like ApacheBench and Microsoft's
> stress test indicate that Catalina is faster than Tomcat on serving
> static resources (substantially faster if caching is enabled,
> basically on par with Apache itself), and within a few percentage
> points on serving "Hello, World" type servlets and JSP pages."
>
> It would be nice to see the results of at least the ApacheBench test
> since ApacheBench is potentially available to all developers via the
> ApacheBench Gateway at codeflux.com.
CodeFlux is (as the name says) a gateway that merely measures the
network latency times between their host and whatever URL you enter...
It's not a web-server benchmark, it's a network benchmark.
> [... skipped digression over codeflux tests...]
> QUESTION 2: Using the criteria given above, does this mean that the
> Apache web server is not a "good" web server? (I understand that, per
> the goals of the ASF, functionality should dominate performance).
Given that the test metodology is wrong this question is not pertinent.
Try to re run YOUR test with this URL:
<http://codeflux.com/images/tell.gif>
Apache 1.3.4 processes approximately 36 requests per seconds... Whoha!
> QUESTION 3: Does Catalina have any performance REQUIREMENTS? On page
> 12 of "Java Platform Performance", Steve Wilson and Jeff Kesselman of
> Sun suggest the need to "reach a consensus about the performance
> requirements for your system. If you don't, how will you know when
> your product is fast enough". If such advice doesn't apply here, why
> not?
I don't agree with Wilson and Kesselman on that. Our requirements (after
of course our requirements of being compliant to the spec and bug-free)
are to be "faster". And the codebase is clean enough to see where our
performance weak points are. We know how current VMs are working, what
is an overhead and what is not. I believe we can achieve very good
results only looking at our code...
> [... skipped descriptions of what's in the files...]
>
> QUESTION 4: will Catalina traffic look more like URL1 or URL2 or some
> mix of the two or some other mix.
None of the above. You're measuring the network.
> QUESTION 5: what will be the consequences of answering question 4 for
> user-level performance?
With your test-case, just buy more bandwidth, and sit your host on the
AXXS network (http://www.axxs.net/)
> "Only modest efforts to tune Catalina have taken place to date
> (compared to the substantial -- and successful -- tuning efforts
> expended between Tomcat 3.1 and 3.2), so it can be assumed that there
> is still significant room for improvement through the usual techniques
> of isolating hot spots and optimizing them."
>
> While I agree that this is the usual technique, there are those who
> suggest it may not be enough. Some suggest that performance is
> functionality: if it takes 10 minutes to login to a system, it isn't
> functional in an meaningful sense (I am not predicting that such an
> event would occur, only noting that it did occur elsewhere). In the
> horror story I mentioned in an earlier post, there were no
> REQUIREMENTS up front. It was assumed that intelligent design, coding,
> tuning. and a bigger pipe/hammer/etc. would be enough. They weren't.
> Sorry for the preachiness.: please educate me.
As I told before, maybe we don't have _requirements_ bu we know what's
going on and how to improve our performances. We just need programmers
time to go in and optimize.
> If there is a version of Tomcat or Catalina that is up that can be
> accessed using the codeflux.com ApacheBench Gateway, I am happy to
> volunteer for some additional monkey-test work :-) like that done in
> producing the data shown in the attachment.
Don't worry. Using codeflux is completely useless in terms of
benchmarking the internals of any web server.
Pier