You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Darryl Pentz <dj...@yahoo.com> on 2009/03/10 00:40:14 UTC
Tracking down OOM - PermGen using jmap and jhat
I'm using this article: http://blogs.sun.com/fkieviet/entry/how_to_fix_the_dreaded to try and isolate an apparent memory leak in our web application. It has been functioning fine until a new release which we deployed over this last weekend. Now suddenly we're hitting PermGen OOM's within the first day.
According to the article above, I should be able to deploy my application, then undeploy it, and be able to then use the combination of jmap and jhat to identify classes that are hanging around in the VM when they should have been unloaded. I'm not having much success with this.
My first question is, how do I 'unload' a web application from a running Tomcat instance. I have used the manager app to stop and undeploy the app, but in both cases when I run jmap/jhat the resulting output shows the memory to still be littered with the application classes. Is there another way to 'unload' an app such that it will force TC to also unload all of its associated classes?
So that's possibly the second question, assuming I use the correct manager option, how will I know when it's classes have been unloaded as well?
Thanks,
Darryl
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tracking down OOM - PermGen using jmap and jhat
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Darryl Pentz [mailto:djpentz@yahoo.com]
> Subject: Re: Tracking down OOM - PermGen using jmap and jhat
>
> I found this thread:
> http://forum.springframework.org/printthread.php?t=21383&pp=40
There's a lot of real BS in that thread. There is one accurate and useful statement:
"What it boils down to is Tomcat's classloader not releasing objects that CGLIB holds in ThreadLocal variables during redeployment"
The classloader cannot, of course, release such objects - it has no knowledge of them. It really is a CGLIB problem, since it's failing to null out ThreadLocal values at the end of processing each request; such behavior is antisocial, at best.
Solving this problem would be difficult in any container using thread pooling. If CGLIB provides some means of cleaning up after itself, you might be able to write a filter that runs at the end of each request to make the necessary calls.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tracking down OOM - PermGen using jmap and jhat
Posted by Darryl Pentz <dj...@yahoo.com>.
Hi Chuck,
Thanks. I read that tutorial as you suggested. Very useful information. I think I understand the gist of it. After further reading it seems like there are some issues w.r.t. the use of Spring(and/or)Hibernate together with CGLib that particularly affect Tomcat. I found this thread: http://forum.springframework.org/printthread.php?t=21383&pp=40 which seems to try and point the finger at Tomcat. Does anybody concur with these findings or have you found ways around using Spring/Hibernate/CGLib/Tomcat so that they all get along? It's a bit late in the game for me to be making any major architectural changes and Tomcat has worked well for us in all other respects.
Thanks again,
Darryl
----- Original Message ----
From: "Caldarale, Charles R" <Ch...@unisys.com>
To: Tomcat Users List <us...@tomcat.apache.org>
Sent: Tuesday, March 10, 2009 5:52:29 PM
Subject: RE: Tracking down OOM - PermGen using jmap and jhat
> From: Darryl Pentz [mailto:djpentz@yahoo.com]
> Subject: Re: Tracking down OOM - PermGen using jmap and jhat
>
> I tried using JConsole's GC button, but
> clearly this didn't do the trick.
Did clicking the button run a major GC (aka PS MarkSweep)?
Are any classes at all being unloaded? (Look at the Classes tab in JConsole.)
> -XX:MaxPermSize=128m
That's actually not very big.
> So the issue right now is a plan of attack to properly unload
> classes in order to expose the likely culprit.
The culprit is what's keeping the classes from being unloaded.
Here's a short tutorial you might want to look at:
http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/html/memleaks.html
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tracking down OOM - PermGen using jmap and jhat
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Darryl Pentz [mailto:djpentz@yahoo.com]
> Subject: Re: Tracking down OOM - PermGen using jmap and jhat
>
> I tried using JConsole's GC button, but
> clearly this didn't do the trick.
Did clicking the button run a major GC (aka PS MarkSweep)?
Are any classes at all being unloaded? (Look at the Classes tab in JConsole.)
> -XX:MaxPermSize=128m
That's actually not very big.
> So the issue right now is a plan of attack to properly unload
> classes in order to expose the likely culprit.
The culprit is what's keeping the classes from being unloaded.
Here's a short tutorial you might want to look at:
http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/html/memleaks.html
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tracking down OOM - PermGen using jmap and jhat
Posted by Darryl Pentz <dj...@yahoo.com>.
Hi Charles,
Thanks for the info. I recall a post of yours I read on the Nabble list related to this stuff so I appreciate and value your feedback.
I think I misspoke earlier. When I said the memory is still littered with the application classes, I mean virtually everything, thousands of dynamic Proxy classes that seemed to be generated by Hibernate, all the Hibernate classes, the Spring classes, it was virtually the entire application. What I suspect I saw was the memory prior to GC. I had used the TC manager app to both stop the app and to undeploy. How do I force a GC though? I tried using JConsole's GC button, but clearly this didn't do the trick. Do I just leave Tomcat running for some period of time?
As to the other info:
Other information you need to supply on your next posting:
1) Tomcat version
Tomcat 6.0.18
2) JRE/JDK version (including 32- or 64-bit)
JRE 1.6.0 Update 12 32 bit
3) platform you're running on
Windows 2003 R2
4) JVM command-line options used when starting Tomcat
-Xms512
-Xmx1024
-XX:MaxPermSize=128m
The permsize was bumped up a long time ago because Tomcat runs both a pure Java app and a Grails application which generates a lot of dynamic classes, and Hibernate and Spring are used in both. Some other info on the application itself is that the customer instances where we're experiencing problems also have a high client load (we have client apps that connect to our pure Java server app) and they maintain long-lived socket connections to the server that ping every 30s on the socket as a heartbeat. I don't think the Grails application is causing a problem as the issue occurs even when the Grails app is not (or is hardly) used. So the main activity would be the N clients pinging away with each ping causing a DB roundtrip. At the moment the problem instances have 1800 and 2700 concurrent client connections to the server. So the leak is a LOT more noticeable than some of our customer instances where the client count is much lower.
So the issue right now is a plan of attack to properly unload classes in order to expose the likely culprit.
Many thanks,
Darryl Pentz
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tracking down OOM - PermGen using jmap and jhat
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Darryl Po force a entz [mailto:djpentz@yahoo.com]
> Subject: Tracking down OOM - PermGen using jmap and jhat
>
> in both cases when I run jmap/jhat the resulting output shows
> the memory to still be littered with the application classes.
Which is likely the exact memory leak you're looking for. However, you do need to force two full GCs to insure that unreferenced Class objects are scavenged. If you've done the GCs, then look for where the references to the classes are coming from; that's the cause of the leak.
> Is there another way to 'unload' an app such that it will
> force TC to also unload all of its associated classes?
TC does not unload classes, a full GC by the JVM does - but only when all references to the Class objects have been cleared.
Other information you need to supply on your next posting:
1) Tomcat version
2) JRE/JDK version (including 32- or 64-bit)
3) platform you're running on
4) JVM command-line options used when starting Tomcat
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org