High CPU usage in Tomcat 7

[James Lampert <ja...@touchtonecorp.com>]

We just had a report of extremely high CPU usage from the Tomcat job on 
one of our customer installations. A WRKACTJOB screen shot from before 
we forcibly shut Tomcat down and restarted it shows:

Subsystem/Job   Type  CPU %  Function        Status
   CATALINA      BCH      .0  CMD-QSH         TIMW
   QP0ZSPWT      BCI   112.2  JVM-org.apache  TIMW
(QP0ZSPWT being the system-generated job that's doing the actual work 
for the CATALINA job.)

Of particular interest is that, at least at the moment the screen shot 
was taken, the QP0ZSPWT job was taking up what appears to be more than 
an entire processor, even though it's in a "time-wait" state.

Based on a Google search on "tomcat 7 high cpu usage," I'm suspecting a 
previously unknown tightloop in our application (which was what I 
suspected even before I did the Google search). The pages I looked at 
also said something about profiling and thread dumps, to find the 
offending thread, but since the job has been terminated and restarted, 
and is not currently malfunctioning, I wouldn't be able to do so even if 
I knew how (which at present I don't).

I've passed on the log files generated by our application itself to 
someone better equipped to deal with them than I, and I've asked the 
Java-400 List at Midrange.com about AS/400-specific steps to track down 
the offending thread if the problem is observed again, but I would also 
value any insights this list might offer.

--
JHHL

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: High CPU usage in Tomcat 7

[Christopher Schultz <ch...@christopherschultz.net>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

James,

On 6/20/12 12:27 PM, James Lampert wrote:
> We just had a report of extremely high CPU usage from the Tomcat
> job on one of our customer installations. A WRKACTJOB screen shot
> from before we forcibly shut Tomcat down and restarted it shows:
> 
> Subsystem/Job   Type  CPU %  Function        Status CATALINA
> BCH      .0  CMD-QSH         TIMW QP0ZSPWT      BCI   112.2
> JVM-org.apache  TIMW (QP0ZSPWT being the system-generated job
> that's doing the actual work for the CATALINA job.)
> 
> Of particular interest is that, at least at the moment the screen
> shot was taken, the QP0ZSPWT job was taking up what appears to be
> more than an entire processor, even though it's in a "time-wait"
> state.
> 
> Based on a Google search on "tomcat 7 high cpu usage," I'm
> suspecting a previously unknown tightloop in our application (which
> was what I suspected even before I did the Google search). The
> pages I looked at also said something about profiling and thread
> dumps, to find the offending thread, but since the job has been
> terminated and restarted, and is not currently malfunctioning, I
> wouldn't be able to do so even if I knew how (which at present I
> don't).
> 
> I've passed on the log files generated by our application itself
> to someone better equipped to deal with them than I, and I've asked
> the Java-400 List at Midrange.com about AS/400-specific steps to
> track down the offending thread if the problem is observed again,
> but I would also value any insights this list might offer.

The advice you got about thread dumps was spot-on: get yourself a
thread dump [1] whenever you think your process is using too much CPU
time. Better yet, take a few of them and compare. If you do have a
tight loop, you'll probably be able to see it because one thread will
be stuck in the same method for a while.

Taking a thread dump *should* be easy (not sure on AS/400) and it
doesn't take a long time to get one. That means you don't disturb
current users like taking a heap dump would (heap dumps in my
experience tend to pause the entire JVM). I suppose you're about to
take-down the JVM so user inconvenience isn't a huge deal.

You might also consider that high CPU usage isn't necessarily bad,
unless it's impacting the operation of one or more services. Assuming
that your suspected-tight-loop finally completes, it might be better
to just let it finish rather than taking-down the JVM entirely.

- -chris

[1]
http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk/rTZEACgkQ9CaO5/Lv0PDL5wCgu683MJwUBQzgn2HKPcDinUEF
PyYAnAvPWYtbSB8PiKF4OfFchPKbMTcL
=ETGk
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org