You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Greg Vilardi <gv...@recurrentsoft.com> on 2007/11/02 00:11:14 UTC
What do I do with a heap dump? (OOM Permgen)
Hello everyone.
My team and I are trying to develop a new web application and the
tomcat JVM is crashing every few days. We are deploying our separate
versions of the application several times per hour, and by looking at
jprobe, I see that each deployment of a webapp consumes 440kb of
PermGen space. This space does not seem to be released, although I've
only been monitoring this for 4 hours now. I have a heap dump from the
last crash, (courtesy of +XXHeapDumpOnOutOfMemoryError) and have taken
two memory dumps using jmap before and after a redeploy on an otherwise
quiescent JVM.
Now, having done this, I've also looked at the dumps using jhat. There
seems to be a vast amount of data there, but I can't distill the
information I need out of it. I apologize if my Google-Fu is weak, but
I didn't find much real information on interpreting jhat data.
Preliminary examination of the jhat data does not show much of my
application or helper classes (mysql, jdbc).
So, my questions are:
How do I figure out what is in that 440kb per deployment?
Is there an FM for me to R on how to interpret jhat data?
What should I be looking for?
How do I break this problem down further?
The environment is:
OS: Debian etch 2.6.21 kernel
JVM: Sun Java 6 1.6.0-b105 Configured for use with jprobe and default
memory allocations. (This was done to try to make the problem easier to
reproduce/study.)
Tomcat: 6.0.14 fronted via Apache 2.2.4 using mod_proxy_ajp.
Database: MySQL 5.0.32 using the latest ConnectorJ.
The application is a combination of JSP and servlets with some AJAX but
no Comet, Hibernate, &c. Apache and Tomcat were hand installed (not
from Debian packages).
Any help that you can provide would be greatly appreciated.
-Greg
--
Gregory H. Vilardi 631-752-2701x240 gvilardi@recurrentsoft.com
Project Manager / Lead Software Engineer Fax: (631)752-3397
Recurrent Software Solutions, Inc. http://www.recurrentsoft.com
1 Huntington Quadrangle, Suite 1C02, Melville, NY 11747
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: What do I do with a heap dump? (OOM Permgen)
Posted by Diego Rodríguez Martín <dr...@altiria.com>.
Hi,
I found a memory leak in commons pool version 1.3 that is affecting
tomcat 6.0.x as it is the commons pool library embedded in it. I tried a
workaround using a factory with commons 1.2, but it also showed a memory
leak.
http://www.mail-archive.com/users@tomcat.apache.org/msg29820.html
As far as I know, it has not been resolved, so you could be running
into this
Diego
Greg Vilardi escribió:
> On 2 Nov 2007 at 7:24, Caldarale, Charles R wrote:
>
>
>>> From: Peter Crowther [mailto:Peter.Crowther@melandra.com]
>>> Subject: RE: What do I do with a heap dump? (OOM Permgen)
>>>
>>> As far as I know, public enemy #1 for eating PermGen space is
>>> still developers using the Singleton pattern in their code
>>> and not having listeners to null out the singleton instance
>>> when the webapp is undeployed.
>>>
>> Analagous use of ThreadLocal is also a subtle contributor to the
>> problem. The offending references must be cleared at the end of
>> processing of each request, since listeners don't have ready access to
>> the thread pool.
>>
>
> Thank you for the pointers Peter. We do not have any Singletons in the
> application nor do we use ThreadLocal. However, we do not seem to be
> cleaning up the JDBC classes on redeploy. I'll install the
> SessionListener code that is mentionned in the references in Peter's
> reply. I'm also using the eval version of yourkit now to see what
> exactly is attached to each of the 52 WebAppClassLoaders in the dump.
>
> I'll be back if I get stuck again further down the road. Thank you to
> all of the people who replied for the pointers and the tool
> recommendations.
>
> -Greg
>
--
-------------------------------------------------------------
Diego Rodríguez Martín (drodriguez@altiria.com)
ALTIRIA TIC - Servicios SMS - Desarrollo Web
Tel. +34 913311198 - Fax +34 915713993 - Móvil +34 610299750
www.altiria.com
-------------------------------------------------------------
RE: What do I do with a heap dump? (OOM Permgen)
Posted by Greg Vilardi <gv...@recurrentsoft.com>.
On 2 Nov 2007 at 7:24, Caldarale, Charles R wrote:
> > From: Peter Crowther [mailto:Peter.Crowther@melandra.com]
> > Subject: RE: What do I do with a heap dump? (OOM Permgen)
> >
> > As far as I know, public enemy #1 for eating PermGen space is
> > still developers using the Singleton pattern in their code
> > and not having listeners to null out the singleton instance
> > when the webapp is undeployed.
>
> Analagous use of ThreadLocal is also a subtle contributor to the
> problem. The offending references must be cleared at the end of
> processing of each request, since listeners don't have ready access to
> the thread pool.
Thank you for the pointers Peter. We do not have any Singletons in the
application nor do we use ThreadLocal. However, we do not seem to be
cleaning up the JDBC classes on redeploy. I'll install the
SessionListener code that is mentionned in the references in Peter's
reply. I'm also using the eval version of yourkit now to see what
exactly is attached to each of the 52 WebAppClassLoaders in the dump.
I'll be back if I get stuck again further down the road. Thank you to
all of the people who replied for the pointers and the tool
recommendations.
-Greg
--
Gregory H. Vilardi 631-752-2701x240 gvilardi@recurrentsoft.com
Project Manager / Lead Software Engineer Fax: (631)752-3397
Recurrent Software Solutions, Inc. http://www.recurrentsoft.com
1 Huntington Quadrangle, Suite 1C02, Melville, NY 11747
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: What do I do with a heap dump? (OOM Permgen)
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Peter Crowther [mailto:Peter.Crowther@melandra.com]
> Subject: RE: What do I do with a heap dump? (OOM Permgen)
>
> As far as I know, public enemy #1 for eating PermGen space is
> still developers using the Singleton pattern in their code
> and not having listeners to null out the singleton instance
> when the webapp is undeployed.
Analagous use of ThreadLocal is also a subtle contributor to the
problem. The offending references must be cleared at the end of
processing of each request, since listeners don't have ready access to
the thread pool.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: What do I do with a heap dump? (OOM Permgen)
Posted by Peter Crowther <Pe...@melandra.com>.
> From: Greg Vilardi [mailto:gvilardi@recurrentsoft.com]
> How do I figure out what is in that 440kb per deployment?
> What should I be looking for?
As far as I know, public enemy #1 for eating PermGen space is still developers using the Singleton pattern in their code and not having listeners to null out the singleton instance when the webapp is undeployed. I've no doubt I'll be yelled at by other list members if Java 6 or Tomcat 6 have dealt with this - I'm still on 5.5!
A description of the problem (there are plenty of others on the list over the years, and this may not be the best) can be found at:
http://readlist.com/lists/tomcat.apache.org/users/3/19056.html
You've probably seen the following two general links, but just in case:
http://tomcat.apache.org/faq/memory.html
and from that...
http://opensource.atlassian.com/confluence/spring/pages/viewpage.action?pageId=2669
- Peter
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: What do I do with a heap dump? (OOM Permgen)
Posted by Greg Vilardi <gv...@recurrentsoft.com>.
On 1 Nov 2007 at 18:32, Gabe Wong wrote:
> Greg Vilardi wrote:
> > Hello everyone.
> >
> > My team and I are trying to develop a new web application and the
> > tomcat JVM is crashing every few days. We are deploying our separate
> > versions of the application several times per hour, and by looking at
> Please elaborate, are you undeploying the same application, then
> redeploying several times per hour,
> are are you deploying several instances of the same application per
> hour.
>
> I am also assuming these are hot deploys (without shutting down the app
> server)?
Each of us uses our own sandbox directory that is linked into the
tomcat/webapps directory. We use make for our build management, and run
'make install' to copy the JSPs and jar files into the webapp
directory. Since Tomcat is set to dynamically look for new files, we
don't actually ever undeploy the application directly. Each developer
rebuilds their web application several times per hour into their own
sandbox. You assume correctly that weare not shutting down tomcat as
part of each deployment.
Thanks,
-Greg
--
Gregory H. Vilardi 631-752-2701x240 gvilardi@recurrentsoft.com
Project Manager / Lead Software Engineer Fax: (631)752-3397
Recurrent Software Solutions, Inc. http://www.recurrentsoft.com
1 Huntington Quadrangle, Suite 1C02, Melville, NY 11747
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: What do I do with a heap dump? (OOM Permgen)
Posted by Gabe Wong <ga...@ngasi.com>.
Greg Vilardi wrote:
> Hello everyone.
>
> My team and I are trying to develop a new web application and the
> tomcat JVM is crashing every few days. We are deploying our separate
> versions of the application several times per hour, and by looking at
Please elaborate, are you undeploying the same application, then
redeploying several times per hour,
are are you deploying several instances of the same application per hour.
I am also assuming these are hot deploys (without shutting down the app
server)?
--
Regards
Gabe Wong
NGASI AppServer Manager
Application server installation and configuration AUTOMATION
http://www.ngasi.com
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: What do I do with a heap dump? (OOM Permgen)
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
www.yourkit.com works, and I'm sure there are others
Filip
Greg Vilardi wrote:
> Hello everyone.
>
> My team and I are trying to develop a new web application and the
> tomcat JVM is crashing every few days. We are deploying our separate
> versions of the application several times per hour, and by looking at
> jprobe, I see that each deployment of a webapp consumes 440kb of
> PermGen space. This space does not seem to be released, although I've
> only been monitoring this for 4 hours now. I have a heap dump from the
> last crash, (courtesy of +XXHeapDumpOnOutOfMemoryError) and have taken
> two memory dumps using jmap before and after a redeploy on an otherwise
> quiescent JVM.
>
> Now, having done this, I've also looked at the dumps using jhat. There
> seems to be a vast amount of data there, but I can't distill the
> information I need out of it. I apologize if my Google-Fu is weak, but
> I didn't find much real information on interpreting jhat data.
> Preliminary examination of the jhat data does not show much of my
> application or helper classes (mysql, jdbc).
>
> So, my questions are:
> How do I figure out what is in that 440kb per deployment?
> Is there an FM for me to R on how to interpret jhat data?
> What should I be looking for?
> How do I break this problem down further?
>
> The environment is:
> OS: Debian etch 2.6.21 kernel
> JVM: Sun Java 6 1.6.0-b105 Configured for use with jprobe and default
> memory allocations. (This was done to try to make the problem easier to
> reproduce/study.)
> Tomcat: 6.0.14 fronted via Apache 2.2.4 using mod_proxy_ajp.
> Database: MySQL 5.0.32 using the latest ConnectorJ.
> The application is a combination of JSP and servlets with some AJAX but
> no Comet, Hibernate, &c. Apache and Tomcat were hand installed (not
> from Debian packages).
>
> Any help that you can provide would be greatly appreciated.
>
> -Greg
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org