tomcat heap gc marksweep outage

[Charles Richard <ch...@thelearningbar.com>]

Hi,

I'm trying to help out my old company who has no IT staff to look at this.
This might be a bad coding issue but I'm hoping to be able to understand
this issue.

They are using Tomcat 6.0.35 and Java 1.6.0_26 . The application is a Java,
hibernate, c3p0 application, not really sure if it is Spring, Struts or
what the Java framework is.

The issue is the site gets unresponsive almost every day. The monitoring
tool shows that the heap reaches its limit and then the GC marksweep kicks
in.

My knowledge about Garbage Collection is not extensive but my understanding
is that once the heap reaches its limit or close to, the GC kicks in and
this is likely what is rendering the site unresponsive.

So my question is: Is there anything I can do with Tomcat to troubleshoot
this further?

Thanks,
Charles

Re: tomcat heap gc marksweep outage

[Christopher Schultz <ch...@christopherschultz.net>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Charles,

On 6/30/14, 9:14 AM, Charles Richard wrote:
> I'm trying to help out my old company who has no IT staff to look
> at this. This might be a bad coding issue but I'm hoping to be able
> to understand this issue.
> 
> They are using Tomcat 6.0.35 and Java 1.6.0_26 . The application is
> a Java, hibernate, c3p0 application, not really sure if it is
> Spring, Struts or what the Java framework is.
> 
> The issue is the site gets unresponsive almost every day. The
> monitoring tool shows that the heap reaches its limit and then the
> GC marksweep kicks in.
> 
> My knowledge about Garbage Collection is not extensive but my
> understanding is that once the heap reaches its limit or close to,
> the GC kicks in and this is likely what is rendering the site
> unresponsive.
> 
> So my question is: Is there anything I can do with Tomcat to
> troubleshoot this further?

It's going to take some work to figure this out.

The JVM naturally takes small pauses throughout time to clean-up
garbage. Basically, in a generational GC, the new space is scanned for
(soon-to-be) survivors, then they are copied to the survivor space and
everything leftover in the "old" new space is discarded. It's like
double-buffering.

Anyhow, those things can happen once every few seconds. Once an object
has survived a certain number of sweeps, it gets "tenured" and moves
up to another part of the heap that gets checked less frequently. When
that heap space fills up (or any other event that causes this to
happen), the tenured space is scanned and cleaned. The tenured space
is typically much larger than the new/survivor spaces and most objects
remain alive so it takes longer to scan and clean that space.

If you have a huge tenured heap space, a huge number of objects that
are still alive, and are always operating at close to the size limit
of that heap space, then you are going to constantly have the GC
working to reclaim tiny amounts of the tenured space.

Things to look for:

1. What are the heap space sizes? If you aren't explicitly setting a
heap size, you can use 'jinfo' and 'jmap' to get information about a
running JVM. Or you can connect using jconsole or something similar.

2. How much heap space is being used of those heap spaces? Take a look
after startup/warmup and compare that to what's going on when things
become unresponsive. Are there in fact large numbers of long-lived
objects in the heap?

3. Does the JVM ever throw OutOfMemoryError? If it does, often things
to downhill fast and may be unrecoverable (from an application point
of view).

4. Does the application legitimately have a need to store lots of
long-lived objects? If so, then you may just need a bigger heap.

5. Check your GC settings. If you have a bunch of them, review them to
make sure they make sense. Lots of folks are running heap settings
that they got from some blog 12 years ago and they are awful ideas
these days with new GC technology, better defaults, etc. If nobody can
remember why something is set the way it is, remove it and start with
a clean(er) configuration.

6. Always remember that sometimes expanding the heap can make things
worse. So, don't just blindly double the heap size and call the
problem fixed. What you might end up doing is just delaying the
unresponsiveness if there really is an underlying problem that hasn't
been solved.

Hope that helps,
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJTsbk5AAoJEBzwKT+lPKRYtiAP/j93JbplrBgzYzmIM0ehUaN3
Z/D9lazKivgDqNcAAMuhVXDapZygG7jvv9uNo6W4LVDr7b2z48f6DnDQ+UWtO0Dc
HAJwRncMqzdOvDGS/4VKfB/3PgG7Px6u9rup/4mMAAoZQ2CSbPyZ3Z61TaypIWLI
TSS+fRqJuznCYuwXhrwhifd4xxVVDmNYFTeKGSV0mF0siz5Zk9O+HbliXvj/2C0G
rtH2UfG+CNp5ZItpGp6kukFTQqMnUYwg83nC62zvdagMONQpnWaFFeuHMv1Brpzw
Ue5ph/wOKMiyJ2Y/U2hB6+VQgCP19saEGfrJDz8ifdJHtNDsYgXmnYch9w/TsiNs
UQWq1lx3058+giwpTC+FpsvkaDQIpNc7/anrsGttXQvIsejDd/7fxWM0i6SXqQCK
3VMkZrooN2OGAV21k9IiYBS/gJ9l8NQ2Zzl4ERiXyJIH94iraJR+pwGF5Z9HAT/J
3tifwqkprr2Zw9N3hSBYjPg8Qd+1eiYqqdxgkR5dY0g3z99rPn2htahWospR8Y/u
5QO956qAcYniAsovxooCkCO5TrRVpBmE+FszmWPQHUJn+ze6b8ZiCCsUPucfxbkg
1lC7w3W1DQ+ws5FkyQxLtNgoQpx32TJ8QFInKyXDOtnOS4u4rnG65qWSdpT2Wp8F
kK6rfpeDT6E8CphNryF3
=sGH1
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org