perf tuning (RE: test harness for Apache)

[dean gaudet <dg...@arctic.org>]

On Thu, 5 Jul 2001 Paula_Stankus@doh.state.fl.us wrote:

> Does anyone know the best way/methods/tools to monitor web applications
> running on Apache written in java, jsp and some CGI?  Is server-status good?
> Any hints on what to look for?  Any books on performance or white papers?  I
> need to isolate performance problems on the webserver - would like to not
> have to spend $$ for additional monitoring tools.

i get a lot of mileage out of the standard tools that come with each unix.

"vmstat 5" is usually the first thing i type on a system with a
performance problem.  i eschew "top" 'cause it doesn't give you the info
in a useful format.

strace or truss are usually the second thing i go for.

your jvm may reduce the usefulness of strace -- it may be difficult to
piece together what syscalls are for which request.  strace/truss work at
the kernel syscall level, which is higher than the thread level in some
unixes (i.e. solaris).

typically what i do is use "strace -r -tt -T -ff -o trace -p pid1 -p pid2
...".  then i start studying those -- what i'm looking for is long periods
spent in a syscall or long periods between syscalls.

i suggest writing a perl script which understands your application's
syscall patterns and groups them into larger chunks, such as "time spent
querying database", "time spent accessing NFS", "time spent allocating
memory".  then start looking at the requests which take an inordinate
amount of time.

there's no single approach which attacks a networked system performance
problem.  you have to look at all the info you can dig up.

you can probably also find a profiler for your jvm which will give you
different and useful information on your java code itself.  i tend to look
at profiles after i've done the system level stuff above, because profiles
require me to understand the code.  (whereas system level stuff is
universal.)

the server-status doesn't really give you much to go on for performance
tuning.  although it does help you with one thing:  tuning the MaxChilds
setting.  in concert with vmstat you want to set MaxChilds such that you
don't swap.

-dean

Re: perf tuning (RE: test harness for Apache)

[Brian Pane <bp...@pacbell.net>]

dean gaudet wrote:

>On Thu, 5 Jul 2001 Paula_Stankus@doh.state.fl.us wrote:
>
>>Does anyone know the best way/methods/tools to monitor web applications
>>running on Apache written in java, jsp and some CGI?  Is server-status good?
>>Any hints on what to look for?  Any books on performance or white papers?  I
>>need to isolate performance problems on the webserver - would like to not
>>have to spend $$ for additional monitoring tools.
>>
>
>i get a lot of mileage out of the standard tools that come with each unix.
>
>"vmstat 5" is usually the first thing i type on a system with a
>performance problem.  i eschew "top" 'cause it doesn't give you the info
>in a useful format.
>
On platforms where it's available, I also highly recommend sar; running 
its collector
program frequently (e.g., every couple of minutes) gives you some nice 
historical data
on memory usage, paging, CPU usage, forks/second, execs/second, and 
more.  Just as
vmstat is useful if you want to know why the web server is slow right 
now, sar is
useful if someone asks why the web server was slow yesterday afternoon.

>strace or truss are usually the second thing i go for.
>
>your jvm may reduce the usefulness of strace -- it may be difficult to
>piece together what syscalls are for which request.  strace/truss work at
>the kernel syscall level, which is higher than the thread level in some
>unixes (i.e. solaris).
>
I've also found it useful to get a stack trace of the apps in question, 
on platforms
and environments where the needed tools exist.  On Solaris, for example, 
/usr/proc/bin/pstack
is great for getting the C stack trace of any running process.  It's 
sometimes useful as
a crude profiling tool; if you have a multiprocess httpd, or a lot of 
instances of a CGI
program running concurrently, or a multithreaded JVM, you can grab a 
stack trace for
every thread; if a large percentage of processes/threads are in a given 
code path, that's
often where the bottleneck is.

For routine monitoring, one of my favorite metrics is the number of 
requests being
handled concurrently by each part of the system (web server, application 
server, database,
etc).  For anything that uses a TCP connection for each request, you can 
run netstat
periodically to count connections in established state on each port of 
interest.  A
sudden, large jump in the count of established connections usually 
indicates trouble--
either a spike in incoming traffic or a bottleck on the server that 
slows down the
handling of an otherwise normal traffic load.  (It's also useful to 
count connections
in close_wait at the same time, of course.)

--Brian