Individual servlet statistics

[Tyler DeWitt <ty...@colorado.edu>]

Hello all,
    I was curious if anybody knew of some software that tracked the memory
and cpu usage of each servlet that was running on a Tomcat instance.  I am
looking to be able to say "Servlet A is using 2%CPU and 10 mb of heap and
Servlet B is using 2% CPU and 200 mb of heap."  I need to be able to do this
because the company I work for uses a Tomcat server running 20 servlets.
Occasionally, a servlet will begin consuming huge amounts of resources
(either CPU or memory).  Our only diagnostic tool tells us that Tomcat is
consuming resources, it does not pinpoint the actual misbehaving servlet.
   One limitation is that recompiling all the current servlets is not really
an option - there are just far too many developers involved.  I would like
the monitoring to be dynamic.  So far I have been playing with Aspect
Oriented Programming (AspectJ) to try and develop my own solution, but I
figured I would ask if anybody knew of or already had something I might find
useful.


Thanks so much!
Tyler

Re: Individual servlet statistics

[Juha Laiho <Ju...@iki.fi>]

Tyler DeWitt wrote:
>     I was curious if anybody knew of some software that tracked the memory
> and cpu usage of each servlet that was running on a Tomcat instance.  I am
> looking to be able to say "Servlet A is using 2%CPU and 10 mb of heap and
> Servlet B is using 2% CPU and 200 mb of heap."  I need to be able to do this
> because the company I work for uses a Tomcat server running 20 servlets.
> Occasionally, a servlet will begin consuming huge amounts of resources
> (either CPU or memory). 

If it is really occasionally, then a continuous metering might not be
that good; more effective would be to be able to gather data when the
abnormal situation arises.

> Our only diagnostic tool tells us that Tomcat is consuming resources, it
 > does not pinpoint the actual misbehaving servlet.

As for CPU (this depends on platform of course, which you did not
identify):
- you can get a thread dump of the JVM, which will (among other
   things) disclose what was running at the time of the thread
   dump
- you may be able to see per-thread CPU statistics from the OS
   level (this is the platform-dependent part!), and correlate
   them with the thread information in the thread dump; this should
   give you enough to identify the CPU hog; most of the time when
   I've seen issues with Tomcat eating all of the CPU it has been
   either a servlet (or something called via a servlet) stuck in
   a loop for some oversight in application logic, or alternatively
   it has been the garbage collector trying to survive from
   an application that leaks memory

Then to memory usage issues; my recommendation is YourKit memory
profiler (http://www.yourkit.com/). YourKit will give you the tools
to take a memory snapshot at the time your application is having
problems with memory usage (or can even take a snapshot by itself
when memory usage exceeds a pre-set threshold). This snapshot can
then be analyzed off-line to find out the object reference graphs
in the running application. A big plus in this way of analyzing
memory issues is the non-intrusiveness - and from that the extremely
low impact (I haven't been able to notice any - something might be
visible if measuring with accurate timing tools) to the runtime
performance (except for the moments when a snapshot is just being
taken; at those point the JVM may freeze completely for a short
while).

No coding, no recompilation required.
-- 
..Juha

---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

RE: Individual servlet statistics

["Caldarale, Charles R" <Ch...@unisys.com>]

> From: masterdewitt@gmail.com [mailto:masterdewitt@gmail.com]
> On Behalf Of Tyler DeWitt
> Subject: Individual servlet statistics
>
> I was curious if anybody knew of some software that
> tracked the memory and cpu usage of each servlet that
> was running on a Tomcat instance.

You can't directly get memory usage on a servlet basis, but any of the heap profilers can capture the call stack at object allocation time; even the simple one that ships with the JDK (hprof) will do that.  There's a noticeable performance impact, of course.

As far as CPU usage, pretty much all of the profilers (including hprof) can tell you how much time is spent in individual methods, and this has less of an impact that heap monitoring.  Also, the MoSKito package can insert code to observe CPU usage on a webapp basis; not sure about individual servlets:
http://moskito.anotheria.net/howto/HowToMonitorExistingServletInTomcat.html

We've used Wily Introscope in the past with good results, but it ain't cheap.

Some recent tool updates from Sun might be of interest:
http://java.sun.com/performance/jvmstat/index.html

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.

---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Individual servlet statistics

[Tyler DeWitt <ty...@colorado.edu>]

Thanks Peter!
    That information is really useful.  Just FYI, I'm working on extending
the GlassBox Inspector (which does just what you say and monitors individual
requests).  I'm using the new Instrumentation package in Java 6 to find
object sizes, just the integration of all these aspects is a little tricky.
True, the information will not be super accurate, but we just need to know
when a servlet consumes vastly more resources than is normal.

Thanks for the quick reply,
Tyler

On Thu, Jun 26, 2008 at 10:11 AM, Peter Crowther <
Peter.Crowther@melandra.com> wrote:

> > From: masterdewitt@gmail.com [mailto:masterdewitt@gmail.com]
> >     I was curious if anybody knew of some software that
> > tracked the memory
> > and cpu usage of each servlet that was running on a Tomcat
> > instance.
>
> Unfortunately, as all the servlets run on a common JVM and using a common
> heap, there is no way of tracking memory use reliably (unless you want to do
> something slow and intrusive like using the debugger to trace references
> from known application roots).  You *might* be able to do something about
> tracking something to do with CPU by thread, but remember that Tomcat shares
> threads between all the servlets so you'd have to do this on a per-request
> basis.  I can't think of a way of persuading Java to hand over the
> information you'd need even here, but better brains on this list might have
> some ideas.
>
> In general, if you want to monitor the servlets independently, you'll have
> to run 'em on independent JVMs.
>
>                - Peter
>

RE: Individual servlet statistics

[Peter Crowther <Pe...@melandra.com>]

> From: masterdewitt@gmail.com [mailto:masterdewitt@gmail.com]
>     I was curious if anybody knew of some software that
> tracked the memory
> and cpu usage of each servlet that was running on a Tomcat
> instance.

Unfortunately, as all the servlets run on a common JVM and using a common heap, there is no way of tracking memory use reliably (unless you want to do something slow and intrusive like using the debugger to trace references from known application roots).  You *might* be able to do something about tracking something to do with CPU by thread, but remember that Tomcat shares threads between all the servlets so you'd have to do this on a per-request basis.  I can't think of a way of persuading Java to hand over the information you'd need even here, but better brains on this list might have some ideas.

In general, if you want to monitor the servlets independently, you'll have to run 'em on independent JVMs.

                - Peter

---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org