You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by rhull <rh...@pobox.com> on 2007/08/07 13:46:30 UTC
Load testing, benchmarking, and tuning
Hi,
I'm new to the forum, and I'm a little new to Tomcat load
testing/benchmarking. I've done some searching on the 'net, but I haven't
been able to turn up anything useful that addresses my issues. I'm hoping
somebody could provide some helpful thoughts.
I've got Tomcat (5.0) running on Linux with JVM 1.5.0_12, with a pretty thin
servlet. I'm fairly new to benchmarking/load testing with Tomcat, and some
of my numbers seem odd.
Load testing with WAPT from a Windows box across the internet to Tomcat
running on the Linux machine, I'm only seeing something around 15
transactions per second. This seems like an unlikely low number to me.
When I run load testing local to the Linux box using Siege, I'm seeing
around 600 transactions per second. This seems like an unrealistically
large number to me, and I assume the TCP/IP loopback is taking a lot of
network-related effort out of the equation.
I've tried fiddling with the server.xml thread settings, but it didn't seem
to have much affect on my transactions per second.
Running "top" on the box while the Siege load testing was occurring gave me
this information:
------------------------------------------------------
top - 08:49:01 up 60 days, 23:32, 2 users, load average: 3.34, 0.77, 0.33
Tasks: 148 total, 2 running, 146 sleeping, 0 stopped, 0 zombie
Cpu(s): 19.2% us, 71.0% sy, 0.0% ni, 9.8% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 8310224k total, 1227552k used, 7082672k free, 191868k buffers
Swap: 2096472k total, 0k used, 2096472k free, 612832k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
29503 tomcat 25 0 1356m 200m 13m S 711 2.5 41:05.61 java
15500 rhull 19 0 1030m 9124 892 S 10 0.1 0:02.23 siege
13772 rhull 15 0 10424 2060 1380 S 0 0.0 0:05.66 sshd
15603 rhull 16 0 3660 992 764 R 0 0.0 0:00.02 top
1 root 16 0 2020 548 468 S 0 0.0 0:03.50 init
2 root RT 0 0 0 0 S 0 0.0 0:00.25 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:05.19 ksoftirqd/0
4 root RT 0 0 0 0 S 0 0.0 0:00.47 migration/1
5 root 34 19 0 0 0 S 0 0.0 0:00.27 ksoftirqd/1
6 root RT 0 0 0 0 S 0 0.0 0:00.18 migration/2
7 root 34 19 0 0 0 S 0 0.0 0:11.89 ksoftirqd/2
8 root RT 0 0 0 0 S 0 0.0 0:00.23 migration/3
9 root 34 19 0 0 0 S 0 0.0 0:00.02 ksoftirqd/3
10 root RT 0 0 0 0 S 0 0.0 0:00.27 migration/4
11 root 34 19 0 0 0 S 0 0.0 0:18.63 ksoftirqd/4
12 root RT 0 0 0 0 S 0 0.0 0:00.19 migration/5
13 root 34 19 0 0 0 S 0 0.0 0:00.05 ksoftirqd/5
------------------------------------------------------
There's obviously some load on the box (though not necessarily CPU), but it
certainly doesn't seem to be maxed out close to 100% of any resources.
So, I'm kind of stuck:
A) How can I determine what kind of ballpark, realistic transaction
throughput I *should* be expecting with my particular combination of
hardware and software?
B) How can I determine what the bottlenecking factor(s) is(are) on the
throughput?
--
View this message in context: http://www.nabble.com/Load-testing%2C-benchmarking%2C-and-tuning-tf4229615.html#a12032715
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Load testing, benchmarking, and tuning
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
rhull,
rhull wrote:
> I've got Tomcat (5.0) running on Linux with JVM 1.5.0_12, with a pretty thin
> servlet. I'm fairly new to benchmarking/load testing with Tomcat, and some
> of my numbers seem odd.
They always do when you get started ;)
> Load testing with WAPT from a Windows box across the internet to Tomcat
> running on the Linux machine, I'm only seeing something around 15
> transactions per second. This seems like an unlikely low number to me.
How many hops away? What type of Internet connection? If you're testing
from home, then your local connection probably sucks compared to the
server's connection to the Internet. I usually try to load test over a
LAN or short hops over a WAN. That way, I can see what the app can
handle if my server is being swamped by machines that are topologically
diverse, yet all coming through my server's Internet connection.
> When I run load testing local to the Linux box using Siege, I'm seeing
> around 600 transactions per second. This seems like an unrealistically
> large number to me, and I assume the TCP/IP loopback is taking a lot of
> network-related effort out of the equation.
Yup. You're basically taking network lag totally out of the equation.
> I've tried fiddling with the server.xml thread settings, but it didn't seem
> to have much affect on my transactions per second.
If you really do have a "pretty thin" servlet, then your transaction
time is probably being dominated by network traffic.
> Running "top" on the box while the Siege load testing was occurring gave me
> this information:
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 29503 tomcat 25 0 1356m 200m 13m S 711 2.5 41:05.61 java
Holy crap, 711% of the CPU? That's amazing!
Seriously, though, 'top' isn't a serious load-testing tool by any means.
If anything, use something like 'sar'. Top basically tells you what's
running /right now/ and gives you a myopic view of the timings, etc.
> There's obviously some load on the box (though not necessarily CPU), but it
> certainly doesn't seem to be maxed out close to 100% of any resources.
> Cpu(s): 19.2% us, 71.0% sy, 0.0% ni, 9.8% id, 0.0% wa, 0.0% hi,
0.0% si
71% system call usage. That seems kind of high. Then again, if your
servlet is doing almost nothing, then you're living inside of the TCP/IP
library the whole time, which probably uses many system calls.
> So, I'm kind of stuck:
> A) How can I determine what kind of ballpark, realistic transaction
> throughput I *should* be expecting with my particular combination of
> hardware and software?
Where is your server, and what machines do you control that are
topologically close to it?
> B) How can I determine what the bottlenecking factor(s) is(are) on the
> throughput?
You should get a good measurement, first. Don't try to fix something you
haven't verified is broken, yet.
Also don't forget that a local load test isn't worthless. As long as you
know you are comparing apples-to-apples, you can do local load tests to
tweak your code. Your code doesn't run any differently when accessed
locally or remotely. So, if you suspect a bottleneck in your code, you
can use local tests to try to remedy it; if your tests show that your
code is faster, it's still valid.
Just don't use local load testing numbers to guess the number of remote
users you can support ;)
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFGuIDA9CaO5/Lv0PARAjPUAJ9FKPaigU6Or66nfoHsQnih4K5rawCdGR9S
vV9or8NKqvvLjqfLXxzbuBs=
=R0nH
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Load testing, benchmarking, and tuning
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
rhull,
rhull wrote:
>> Also, are you using apr? IF you are serving static content, apr should
>> help a lot (but maybe not if you're planning on caching them in memory).
>> Not sure if a memory cache is going to buy you that much.
>
> Sorry, I'm not sure what "apr" is (besides April, American Public Radio,
> etc).
APR is the Apache Portable Runtime. If you spend any time on the list,
you'll know that a bunch of people have problems getting it working.
Apparently, tossing the library into the native libraries path is very
difficult for some people. APR allows Tomcat to use some native code to
significantly improve performance for several operations including
serving static files from the disk (perhaps even from memory... I dunno,
you'll have to read more about it) and managing SSL connections (uses
OpenSSL for encryption, instead of the Java stuff). You might want to
read about this library to see if it would help at all. Since you're
load testing, you could also do a test to see if it helps at all.
> Yes, even with plain OS file caching, I'm not sure if the memory cache will
> buy me much initially. I don't think I'm going to know until I get to a
> point where my code can realistically test the difference. It's possible
> that the amount of data could significantly increase such that I can still
> viably keep it in memory, but the OS may not cache that many small files.
That's probably something you can tweak, but you'll have to ask people
who know a lot about your particular OS and other operating environment
settings.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFGuJUn9CaO5/Lv0PARAn48AKCdiCDzdL4kVImbvSqubPfdqV3qCwCePXMb
pUju5KIUCymI8ZenSQSLsUk=
=FFVw
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Load testing, benchmarking, and tuning
Posted by rhull <rh...@pobox.com>.
>>I'm running on a 1.5mb download/256kb upload cable connection.
>
>Is that the advertised rate, or the measured rate?
Yes. :-) It's how it's advertised, and I ran a couple online bandwidth
tests to check it.
>Are you just serving static content? What about the POSTs?
The content will be updated 6-12 times an hour, so it's not truly static.
>Also, are you using apr? IF you are serving static content, apr should
>help a lot (but maybe not if you're planning on caching them in memory).
>Not sure if a memory cache is going to buy you that much.
Sorry, I'm not sure what "apr" is (besides April, American Public Radio,
etc).
Yes, even with plain OS file caching, I'm not sure if the memory cache will
buy me much initially. I don't think I'm going to know until I get to a
point where my code can realistically test the difference. It's possible
that the amount of data could significantly increase such that I can still
viably keep it in memory, but the OS may not cache that many small files.
--
View this message in context: http://www.nabble.com/Load-testing%2C-benchmarking%2C-and-tuning-tf4229615.html#a12036101
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Load testing, benchmarking, and tuning
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
rhull,
rhull wrote:
>> Are you bottlenecked on your internet bandwidth? Are you bottlenecked
>> on latency, if you're not re-using HTTP connections?
>
> I can't find any reason to believe I am. I'm running on a 1.5mb
> download/256mb upload cable connection.
Is that the advertised rate, or the measured rate?
> The Linux box is on similar.
On another network, I'm guessing, right?
You have crappy Internet connections, which is more likely to affect
your capacity than your application, unfortunately.
> Right now, the servlet is serving the contents of ~1500 small xml files (the
> next phase of the project moves the files to memory in a service to be
> served up to the servlets). My suspicion was that a lot of the system time
> was due to file activity/swapping.
Are you just serving static content? What about the POSTs?
Also, are you using apr? IF you are serving static content, apr should
help a lot (but maybe not if you're planning on caching them in memory).
Not sure if a memory cache is going to buy you that much.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFGuIHX9CaO5/Lv0PARAhiZAJ9R6UWS3oP3WgceMvwVbYkqOsGtHwCdG0tD
8Qj4wdXZSvhZSsYj2HTRzPo=
=T0Zb
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Load testing, benchmarking, and tuning
Posted by Peter Crowther <Pe...@melandra.com>.
> From: rhull [mailto:rhull@pobox.com]
> I can't find any reason to believe I am. I'm running on a 1.5mb
> download/256mb upload cable connection. The Linux box is on
> similar. The
> requests are fairly short HTTP POST request (couple hundred
> bytes), and the
> responses from the servlet are under 10K each.
OK, you're bottlenecked there I think (assuming you mean 256 *k*b rather
than 256 *m*b upload). Consider: 10 kbytes is around 100 kbits (include
framing and other overheads). 15 transactions per second is therefore
15 x 100 = 1,500 kbits = pretty close to 1.5 Mbit/s. I suspect that
isn't a coincidence.
> Right now, the servlet is serving the contents of ~1500 small
> xml files (the
> next phase of the project moves the files to memory in a service to be
> served up to the servlets). My suspicion was that a lot of
> the system time was due to file activity/swapping.
Sounds likely. Tracing system calls would tell you.
That's a pretty meaty system, doing some pretty small things. 600tps is
600/8 = 75tps per core; you've plenty of memory to go at, there's no
hardware network to drive, and the file contents will be in the OS disk
cache fairly early on in the test. I can believe the numbers you're
getting with that webapp, and I suspect they'll go *up* if you bring the
files into memory. At the moment, it wouldn't totally surprise me if
most of your app fitted into the L2 cache on the CPUs.
- Peter
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Load testing, benchmarking, and tuning
Posted by rhull <rh...@pobox.com>.
>Are you bottlenecked on your internet bandwidth? Are you bottlenecked
>on latency, if you're not re-using HTTP connections?
I can't find any reason to believe I am. I'm running on a 1.5mb
download/256mb upload cable connection. The Linux box is on similar. The
requests are fairly short HTTP POST request (couple hundred bytes), and the
responses from the servlet are under 10K each.
> Why do you feel this is unrealistically large?
Gut feeling...? It just didn't feel right, but I don't have a lot of
experience in this area to draw upon.
>Can you not set up a second Linux box that shares the same LAN and test
>on that?
I'll have to put in a request, but I can't do it myself, no.
>8 core box? Looks like it.
That's correct.
>At under 10% free CPU, I'd call that "close" to maxing out the CPUs.
That's interesting...I was mostly looking at the 20% cpu and 70% system
settings, not the 10% "id".
>71% system time is certainly interesting - are you being clobbered by
>context switches between Siege and Tomcat? A swift "vmstat 5" will tell
>you. Look in the "cs" column.
Right now, the servlet is serving the contents of ~1500 small xml files (the
next phase of the project moves the files to memory in a service to be
served up to the servlets). My suspicion was that a lot of the system time
was due to file activity/swapping.
Thank you for the feedback! Any other thoughts would be welcome.
--
View this message in context: http://www.nabble.com/Load-testing%2C-benchmarking%2C-and-tuning-tf4229615.html#a12033169
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Load testing, benchmarking, and tuning
Posted by Peter Crowther <Pe...@melandra.com>.
> From: rhull [mailto:rhull@pobox.com]
> Load testing with WAPT from a Windows box across the internet
> to Tomcat
> running on the Linux machine, I'm only seeing something around 15
> transactions per second. This seems like an unlikely low
> number to me.
Are you bottlenecked on your internet bandwidth? Are you bottlenecked
on latency, if you're not re-using HTTP connections?
> When I run load testing local to the Linux box using Siege, I'm seeing
> around 600 transactions per second. This seems like an
> unrealistically
> large number to me, and I assume the TCP/IP loopback is
> taking a lot of
> network-related effort out of the equation.
Why do you feel this is unrealistically large?
Loopback will take a lot of the hardware and driver issues out, yes.
Can you not set up a second Linux box that shares the same LAN and test
on that?
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 29503 tomcat 25 0 1356m 200m 13m S 711 2.5 41:05.61 java
8 core box? Looks like it.
> There's obviously some load on the box (though not
> necessarily CPU), but it
> certainly doesn't seem to be maxed out close to 100% of any resources.
At under 10% free CPU, I'd call that "close" to maxing out the CPUs.
71% system time is certainly interesting - are you being clobbered by
context switches between Siege and Tomcat? A swift "vmstat 5" will tell
you. Look in the "cs" column.
> So, I'm kind of stuck:
> A) How can I determine what kind of ballpark, realistic transaction
> throughput I *should* be expecting with my particular combination of
> hardware and software?
As you are, except I'd separate Siege onto its own box.
> B) How can I determine what the bottlenecking factor(s) is(are) on the
> throughput?
Same as always - Java's no different, with the possible addition of
locking issues in some webapps and having to set Tomcat's heap sizes.
If you're used to load testing other apps, you won't see many surprises
with Tomcat.
- Peter
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org