Re: tomcat full GC every 2 minutes for first 6 hours after war reload

[Rainer Jung <ra...@kippdata.de>]

On 29.05.2012 17:03, Kevin wrote:
> Hi
>
> We have a cluster of tomcat servers being used on a very high volume
> website. We've noticed that for the first 5-6 hours after an application
> re-load that Full GC will run every 2 minutes pausing the application for
> anywhere between 5 and 20 seconds. After 5-6 hours the full GC will no
> longer run until tomcat is reloaded.  This behavior is repeatable.
>
> We discounted our EhCache settings as potential issue because tomcat can
> be killed without this issue showing up.  It's not until we actually
> reload the WAR with changes (clearing the work directory) and restart
> tomcat that this issue shows up.  Traffic level isn't a factor we'll go
> through our peak hours with no problem.
>
>
> The servers are all dual quad core with 32GB of RAM running Centos 5 and
> tomcat 6.0.18.  We've attempted every suggested GC setting change
> suggested on the net and by developers and system administrators.  Our
> java opts are being played with on a daily basis however here are the
> settings that correspond to the sample GC log below:
>
>
> -server -Xmx27g -Xms27g  -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC
> -XX:+PrintTenuringDistribution  -Dsun.rmi.dgc.client.gcInterval=900000
> -Dsun.rmi.dgc.server.gcInterval=900000 -XX:NewSize=8g -XX:SurvivorRatio=16
> -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails

> 343.376: [Full GC 343.378: [CMS: 1158107K->1312570K(19922944K), 3.4129290
> secs] 2884580K->1312570K(27845568K), [CMS Perm : 83964K->47203K(83968K)],
> 3.4168600 secs] [Times: user=3.87 sys=0.02, real=3.41 secs]

> 20526.469: [Full GC 20526.475: [CMS: 9175755K->9210800K(19922944K),
> 33.1161300 secs] 13632232K->9210800K(27845568K), [CMS Perm :
> 83967K->53332K(83968K)], 33.1254170 secs] [Times: user=33.12 sys=0.02,
> real=33.12 secs]

Looks to me as the oerm gen fills up and triggers the Full GC. Then perm 
gen is back to only be used by about 60% but rapidly fills up again. So 
it seems your are using a lot of classes that are only used for a short 
period of time.

Add

-XX:+TraceClassUnloading

which adds the names of the classes which are unloaded to the GC output. 
Then check, whether the names of the classes give you an idea, why you 
load and unload that many classes for a long time (and why it finally 
gets stable).

Regards,

Rainer


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org