You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Ken Bowen <kb...@als.com> on 2009/04/21 22:32:40 UTC
Headstart on "Resolving OOM-PermGen errors on webapp reload"
Mark,
Any chance we could make a headstart on "Resolving OOM-PermGen errors
on webapp reload" ??
Perhaps some general pointers, guidance etc. [to help you refine the
talk in advance :-) ]
The manager app is giving me more & more of:
FAIL - Application at context path /ctx could not be started
FAIL - Encountered exception java.lang.OutOfMemoryError: PermGen space
This is on a Tomcat 6.0.18 (java 1.5) running on a remote Linux CentOS
5, in a Parallels VM, with httpd + mod_jk, running two versions of the
app. Total memory available is 432 MB, with a number of non-system,
non-TC processes, including ActiveMQ and 9 Postres processes. But
there appears to be about 200MB of unused memory according to the
Parallels control panel.
I develop locally on a Mac OS/X 10.5.6 box with 4GBmem using
(My)Eclipse and Tomcat 6.0.18 directly. With lots & lots of reloads,
I'm not surprised that I eventually hit OOM PermGen space in this
setting, but it happens much less often than on the CentOS box (of
course 4GB != 432MB).
Thanks...Ken
Mark Thomas wrote:
> I was at a conference recently and rather rashly made the statement
> "OOM
> PermGen errors on reload are an application issue, not a Tomcat one".
> Several people came up to me afterwards with variations on a theme of
> "Our app has OOM on reload - put your money where your mouth is and
> show
> us where we have gone wrong"
>
> Having helped several people track down the causes of these errors
> (and
> yes they were all application issues) it occurred to me that there may
> be interest in this at ApacheCon. So my suggestion is:
>
> Title: Resolving OOM-PermGen errors on webapp reload
> Type: Session
> Abstract: What causes the error. Typical root causes.
> How to find and fix the root causes. With live demo.
> Audience: Developer/SysAdmin
>
> Thoughts?
>
> Mark
Re: Headstart on "Resolving OOM-PermGen errors on webapp reload"
Posted by Ken Bowen <kb...@als.com>.
Thanks Chris,
As in my previous thanks to Mark, I'll be slowly mastering how to get
into this.
Cheers,
Ken
On Apr 21, 2009, at 5:44 PM, Christopher Schultz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Ken,
>
> On 4/21/2009 4:32 PM, Ken Bowen wrote:
>> I develop locally on a Mac OS/X 10.5.6 box with 4GBmem using
>> (My)Eclipse
>> and Tomcat 6.0.18 directly. With lots & lots of reloads, I'm not
>> surprised that I eventually hit OOM PermGen space in this setting,
>> but
>> it happens much less often than on the CentOS box (of course 4GB !=
>> 432MB).
>
> Can you observe your heap and which ClassLoaders are still hanging
> around? You might want to check to see how many
> org/apache/catalina/loader/WebappClassLoader objects are still "live".
> In my web application, just after a Tomcat startup, I have 2 instances
> of this class, both of which are "live".
>
> After an application reload, I can see that I still have 2 live
> WebappClassLoaders: one of them is the same one that was live before
> the
> app reload (probably the "Shared" webapp ClassLoader - I'm using TC
> 5.5)
> and the other one is a new one that appears to have replaced the old
> one
> (which should be expected, since the old one should be discarded after
> the undeploy).
>
> If you find multiple WebappClassLoader objects still hanging around,
> you
> can probably start looking for the objects they actually own. Those
> will
> give you a clue as to what services or objects aren't being cleaned-up
> correctly when your application shuts down.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAknuPiUACgkQ9CaO5/Lv0PBZ9gCgwnO0rP3j40m32+vI1jmy0qaI
> 9w0AoMCt+AteT1uv+vvrgCbpteNV9rGb
> =X9YG
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Headstart on "Resolving OOM-PermGen errors on webapp reload"
Posted by Mark Thomas <ma...@apache.org>.
Caldarale, Charles R wrote:
>> From: Christopher Schultz [mailto:chris@christopherschultz.net]
>> Subject: Re: Headstart on "Resolving OOM-PermGen errors on webapp
>> reload"
>>
>> Can you observe your heap and which ClassLoaders are still hanging
>> around? You might want to check to see how many
>> org/apache/catalina/loader/WebappClassLoader objects are still "live".
>> In my web application, just after a Tomcat startup, I have 2 instances
>> of this class, both of which are "live".
>
> "Live" is a critical attribute here - unless a full GC (or two) is done, unused classes and classloaders will still be present. Use JConsole to trigger GCs, if needed.
A decent profiler - cue another YourKit advert :) will skip any
instances eligible for GC.
Mark
PS To be fair, all the profilers I have seen do this.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [OT] Headstart on "Resolving OOM-PermGen errors on webapp reload"
Posted by Pid <p...@pidster.com>.
Caldarale, Charles R wrote:
>> From: André Warnier [mailto:aw@ice-sa.com]
>> Subject: Re: Headstart on "Resolving OOM-PermGen errors on webapp
>> reload"
>>
>> You'll probably want mountains, and chocolate. Switzerland ?
>
> London. $ vs pound is pretty decent right now. It's been 45+ years since I've been on the Tube...
If you fancy a beer...
p
>> I recently came across this article :
>> http://en.wikipedia.org/wiki/Adaptive_Replacement_Cache
>
> I don't see the parallels; nothing in GC is LRU based that I can think of.
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: [OT] Headstart on "Resolving OOM-PermGen errors on webapp
reload"
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: André Warnier [mailto:aw@ice-sa.com]
> Subject: Re: Headstart on "Resolving OOM-PermGen errors on webapp
> reload"
>
> You'll probably want mountains, and chocolate. Switzerland ?
London. $ vs pound is pretty decent right now. It's been 45+ years since I've been on the Tube...
> I recently came across this article :
> http://en.wikipedia.org/wiki/Adaptive_Replacement_Cache
I don't see the parallels; nothing in GC is LRU based that I can think of.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
Re: Headstart on "Resolving OOM-PermGen errors on webapp reload"
Posted by André Warnier <aw...@ice-sa.com>.
Caldarale, Charles R wrote:
...
>
> It can't really; if I get time before going on vacation this Friday I'll look to see where that number comes from.
>
We'll miss you.
You'll probably want mountains, and chocolate. Switzerland ?
>> You *did* say it was unnecessarily complicated ;)
>
> Probably "seemed like a good idea at the time."
>
I recently came across this article :
http://en.wikipedia.org/wiki/Adaptive_Replacement_Cache
and find some eery parallels. Which makes me wonder about the patent
situation..
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Headstart on "Resolving OOM-PermGen errors on webapp reload"
Posted by János Löbb <ja...@yale.edu>.
>
> are there any good primers on eden,
Stanislaw Lem : Eden :)
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Headstart on "Resolving OOM-PermGen errors on webapp reload"
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Martin Gainty [mailto:mgainty@hotmail.com]
> Subject: RE: Headstart on "Resolving OOM-PermGen errors on webapp
> reload"
>
> we expect free technical support 24/7/365 so bring a blackberry w/ you
No thanks; I'll keep my iPhone (and Skype, so I don't have to pay AT&T's outrageous international charges).
> are there any good primers on eden,PermGen and general heap?
Start here:
http://java.sun.com/javase/technologies/hotspot/gc/index.jsp
Look at the Memory Management white paper and Garbage Collection Tuning to start.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Headstart on "Resolving OOM-PermGen errors on webapp reload"
Posted by Martin Gainty <mg...@hotmail.com>.
we expect free technical support 24/7/365 so bring a blackberry w/ you
are there any good primers on eden,PermGen and general heap?
(HF)
Martin
______________________________________________
Disclaimer and Confidentiality/Verzicht und Vertraulichkeitanmerkung / Note de déni et de confidentialité
This message is confidential. If you should not be the intended receiver, then we ask politely to report. Each unauthorized forwarding or manufacturing of a copy is inadmissible. This message serves only for the exchange of information and has no legal binding effect. Due to the easy manipulation of emails we cannot take responsibility over the the contents.
Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen.
Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni.
> From: Chuck.Caldarale@unisys.com
> To: users@tomcat.apache.org
> Date: Wed, 22 Apr 2009 14:19:16 -0500
> Subject: RE: Headstart on "Resolving OOM-PermGen errors on webapp reload"
>
> > From: Christopher Schultz [mailto:chris@christopherschultz.net]
> > Subject: Re: Headstart on "Resolving OOM-PermGen errors on webapp
> > reload"
> >
> > Does that mean that, technically speaking, PermGen is allowed
> > to grow to take over the whole heap?
>
> No, PermGen is independent of the general heap, limited by MaxPermSize and -Xmx respectively. They are allocated contiguously to insure that the underlying reference marking of HotSpot GC works properly.
>
> > Odd that the NewSize can exceed the maximum heap.
>
> It can't really; if I get time before going on vacation this Friday I'll look to see where that number comes from.
>
> > You *did* say it was unnecessarily complicated ;)
>
> Probably "seemed like a good idea at the time."
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
>
_________________________________________________________________
Windows Live™ Hotmail®:…more than just e-mail.
http://windowslive.com/online/hotmail?ocid=TXT_TAGLM_WL_HM_more_042009
RE: Headstart on "Resolving OOM-PermGen errors on webapp reload"
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Christopher Schultz [mailto:chris@christopherschultz.net]
> Subject: Re: Headstart on "Resolving OOM-PermGen errors on webapp
> reload"
>
> Does that mean that, technically speaking, PermGen is allowed
> to grow to take over the whole heap?
No, PermGen is independent of the general heap, limited by MaxPermSize and -Xmx respectively. They are allocated contiguously to insure that the underlying reference marking of HotSpot GC works properly.
> Odd that the NewSize can exceed the maximum heap.
It can't really; if I get time before going on vacation this Friday I'll look to see where that number comes from.
> You *did* say it was unnecessarily complicated ;)
Probably "seemed like a good idea at the time."
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
Re: Headstart on "Resolving OOM-PermGen errors on webapp reload"
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Chuck,
On 4/22/2009 12:16 PM, Caldarale, Charles R wrote:
>> From: Christopher Schultz [mailto:chris@christopherschultz.net]
>> Subject: Re: Headstart on "Resolving OOM-PermGen errors on webapp
>> reload"
>>
>> It's also a shame that the values for -Xmx aren't shown
>
> It is - it's the MaxHeapSize under Heap Configuration.
>
> The odd thing in your report is MaxNewSize, which is clearly out of
> whack; not sure at this point where that comes from.
I'm using:
java version "1.5.0_13"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_13-b05)
Java HotSpot(TM) Client VM (build 1.5.0_13-b05, mixed mode)
and the heap configuration, again, for reference is:
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 67108864 (64.0MB)
NewSize = 655360 (0.625MB)
MaxNewSize = 4294901760 (4095.9375MB)
OldSize = 1441792 (1.375MB)
NewRatio = 12
SurvivorRatio = 8
PermSize = 8388608 (8.0MB)
MaxPermSize = 67108864 (64.0MB)
Does that mean that, technically speaking, PermGen is allowed to grow to
take over the whole heap? Clearly, that isn't technically possible
(because the other heap sections will be non-zero) but it seems weird
that the max permgen is the same as the max heap.
I wonder if MaxNewSize is set to be the process max memory (4GB... or
just shy of that). That would be a *very* big NewSize. Odd that the
NewSize can exceed the maximum heap.
You *did* say it was unnecessarily complicated ;)
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAknva54ACgkQ9CaO5/Lv0PCwOwCglqyVZQVgBpbDMuKtTo77aQ7T
mNYAn2yb4DO7tq1pQuJ+a/iB4myz66fL
=hLg9
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Headstart on "Resolving OOM-PermGen errors on webapp reload"
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Christopher Schultz [mailto:chris@christopherschultz.net]
> Subject: Re: Headstart on "Resolving OOM-PermGen errors on webapp
> reload"
>
> It's also a shame that the values for -Xmx aren't shown
It is - it's the MaxHeapSize under Heap Configuration.
The odd thing in your report is MaxNewSize, which is clearly out of whack; not sure at this point where that comes from.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
Re: Headstart on "Resolving OOM-PermGen errors on webapp reload"
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Chuck,
On 4/21/2009 8:48 PM, Caldarale, Charles R wrote:
> It's really 64 MB, of which 32.5 MB is available for allocation. The
> 8 MB is the initial amount available for allocation. (If this sounds
> unnecessarily complicated, that's only because it is, with emphasis
> on the unnecessarily.)
Gotcha. I thought the "heap configuration" was the current config, but
it's the initial configuration. That information is ... not particularly
useful. It's too bad they don't have a summary of the /current/ heap
configuration anywhere. You have to read the details (which, I admit,
isn't that bad). It's also a shame that the values for -Xmx aren't
shown, so you'd know how big your heap could get. I'm not assigning any
specific values, so I'm getting the default for my jvm/client/physical
memory size, which I won't know unless I query the Runtime object.
>> I'd love some help interpreting the heap info I see above.
>
> Anything in particular?
Just what you already did. Thanks!
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAknvP24ACgkQ9CaO5/Lv0PCMMwCgmOXyyB9idWxQfDUMyPEQMo2D
dpMAoJQbj/YNaveHL67y2S7XZPYILTxR
=FABQ
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Headstart on "Resolving OOM-PermGen errors on webapp reload"
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Christopher Schultz [mailto:chris@christopherschultz.net]
> Subject: Re: Headstart on "Resolving OOM-PermGen errors on webapp
> reload"
>
> the "configuration" seems to indicate that I've
> got an 8MB PermGen space, but then the Permanent
> Generation says it's 32.5MB.
It's really 64 MB, of which 32.5 MB is available for allocation. The 8 MB is the initial amount available for allocation. (If this sounds unnecessarily complicated, that's only because it is, with emphasis on the unnecessarily.)
> I'd love some help interpreting the heap info I see above.
Anything in particular? The Heap Configuration values are those in play at JVM startup; the Heap Usage values reflect the state of the heap when the snapshot was taken. If not prohibited, the GC code will adjust the sizes of the eden, from/to, and tenured spaces dynamically as conditions change.
For those who haven't already seen it, reference information starts here:
http://java.sun.com/javase/technologies/hotspot/gc/index.jsp
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
Re: Headstart on "Resolving OOM-PermGen errors on webapp reload"
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Chuck,
On 4/21/2009 5:50 PM, Caldarale, Charles R wrote:
>> From: Christopher Schultz [mailto:chris@christopherschultz.net]
>> Subject: Re: Headstart on "Resolving OOM-PermGen errors on webapp
>> reload"
>>
>> Can you observe your heap and which ClassLoaders are still hanging
>> around? You might want to check to see how many
>> org/apache/catalina/loader/WebappClassLoader objects are still "live".
>> In my web application, just after a Tomcat startup, I have 2 instances
>> of this class, both of which are "live".
>
> "Live" is a critical attribute here - unless a full GC (or two) is
> done, unused classes and classloaders will still be present. Use
> JConsole to trigger GCs, if needed.
I haven't forced any full GCs (jconsole can't connect since I didn't
enable JMX when I started my server) but I only see 2 live
WebappClassLoaders (and 2 dead ones) after running this command:
for x in 1 2 3 4 5 ; \
do echo running ; \
touch lib/log4j-1.2.15.jar ; \
ant quick-install ; \
sleep 20s ; \
done
(Each touch/quick-install triggers an application reload)
Before my stress test, this was the output of 'jmap -heap':
(Hmm... didn't realize I was using -client in development... I should
fix that!)
Attaching to process ID 11255, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 1.5.0_13-b05
using thread-local object allocation.
Mark Sweep Compact GC
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 67108864 (64.0MB)
NewSize = 655360 (0.625MB)
MaxNewSize = 4294901760 (4095.9375MB)
OldSize = 1441792 (1.375MB)
NewRatio = 12
SurvivorRatio = 8
PermSize = 8388608 (8.0MB)
MaxPermSize = 67108864 (64.0MB)
Heap Usage:
New Generation (Eden + 1 Survivor Space):
capacity = 4128768 (3.9375MB)
used = 1943672 (1.8536300659179688MB)
free = 2185096 (2.0838699340820312MB)
47.0763191344246% used
Eden Space:
capacity = 3670016 (3.5MB)
used = 1606928 (1.5324859619140625MB)
free = 2063088 (1.9675140380859375MB)
43.785313197544646% used
- From Space:
capacity = 458752 (0.4375MB)
used = 336744 (0.32114410400390625MB)
free = 122008 (0.11635589599609375MB)
73.40436662946429% used
To Space:
capacity = 458752 (0.4375MB)
used = 0 (0.0MB)
free = 458752 (0.4375MB)
0.0% used
tenured generation:
capacity = 54755328 (52.21875MB)
used = 20156272 (19.222518920898438MB)
free = 34599056 (32.99623107910156MB)
36.81152635959007% used
Perm Generation:
capacity = 32505856 (31.0MB)
used = 23190896 (22.116561889648438MB)
free = 9314960 (8.883438110351562MB)
71.343748031124% used
After the test, I get this:
Attaching to process ID 11255, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 1.5.0_13-b05
using thread-local object allocation.
Mark Sweep Compact GC
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 67108864 (64.0MB)
NewSize = 655360 (0.625MB)
MaxNewSize = 4294901760 (4095.9375MB)
OldSize = 1441792 (1.375MB)
NewRatio = 12
SurvivorRatio = 8
PermSize = 8388608 (8.0MB)
MaxPermSize = 67108864 (64.0MB)
Heap Usage:
New Generation (Eden + 1 Survivor Space):
capacity = 3604480 (3.4375MB)
used = 1326112 (1.264678955078125MB)
free = 2278368 (2.172821044921875MB)
36.79066051136363% used
Eden Space:
capacity = 3211264 (3.0625MB)
used = 1004312 (0.9577865600585938MB)
free = 2206952 (2.1047134399414062MB)
31.274663185586736% used
- From Space:
capacity = 393216 (0.375MB)
used = 321800 (0.30689239501953125MB)
free = 71416 (0.06810760498046875MB)
81.83797200520833% used
To Space:
capacity = 393216 (0.375MB)
used = 0 (0.0MB)
free = 393216 (0.375MB)
0.0% used
tenured generation:
capacity = 47677440 (45.46875MB)
used = 40365360 (38.49540710449219MB)
free = 7312080 (6.9733428955078125MB)
84.66343830541237% used
Perm Generation:
capacity = 34078720 (32.5MB)
used = 33963000 (32.38964080810547MB)
free = 115720 (0.11035919189453125MB)
99.66043325570914% used
I'm a little confused... the "configuration" seems to indicate that I've
got an 8MB PermGen space, but then the Permanent Generation says it's
32.5MB.
Let's try to force a full GC by using System.gc() from a little JSP
file. Here's the heap description afterward:
Attaching to process ID 11255, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 1.5.0_13-b05
using thread-local object allocation.
Mark Sweep Compact GC
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 67108864 (64.0MB)
NewSize = 655360 (0.625MB)
MaxNewSize = 4294901760 (4095.9375MB)
OldSize = 1441792 (1.375MB)
NewRatio = 12
SurvivorRatio = 8
PermSize = 8388608 (8.0MB)
MaxPermSize = 67108864 (64.0MB)
Heap Usage:
New Generation (Eden + 1 Survivor Space):
capacity = 3604480 (3.4375MB)
used = 357840 (0.3412628173828125MB)
free = 3246640 (3.0962371826171875MB)
9.927645596590908% used
Eden Space:
capacity = 3211264 (3.0625MB)
used = 357840 (0.3412628173828125MB)
free = 2853424 (2.7212371826171875MB)
11.143275669642858% used
- From Space:
capacity = 393216 (0.375MB)
used = 0 (0.0MB)
free = 393216 (0.375MB)
0.0% used
To Space:
capacity = 393216 (0.375MB)
used = 0 (0.0MB)
free = 393216 (0.375MB)
0.0% used
tenured generation:
capacity = 47677440 (45.46875MB)
used = 17904184 (17.07476043701172MB)
free = 29773256 (28.39398956298828MB)
37.55273773088488% used
Perm Generation:
capacity = 20185088 (19.25MB)
used = 20101336 (19.170127868652344MB)
free = 83752 (0.07987213134765625MB)
99.58507983715504% used
... and checking for WebappClassLoader objects after that shows 2 live
WebappClassLoaders none dead. Looks like my webapp is clean, at least
when it first comes up.
We don't have any weird stuff like Quartz, etc. and all our caching is
done using Maps in the application scope or within a webapp-loaded class.
I also have a pair of ServletContextListeners that shut down JULI and
log4j as appropriate (if they're been loaded by the webapp and not a
parent). I'm not sure if they actually help (and it doesn't matter since
we never bounce our application in production without also bouncing
Tomcat itself.
I'd love some help interpreting the heap info I see above.
Thanks!
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAknuR4sACgkQ9CaO5/Lv0PAnRgCgjHrQWJsj4hDG2ys9OcXE8ftf
8p8AoIVhWdvPJSTpldVCmsolEQe6K8wq
=DW+t
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Headstart on "Resolving OOM-PermGen errors on webapp reload"
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Christopher Schultz [mailto:chris@christopherschultz.net]
> Subject: Re: Headstart on "Resolving OOM-PermGen errors on webapp
> reload"
>
> Can you observe your heap and which ClassLoaders are still hanging
> around? You might want to check to see how many
> org/apache/catalina/loader/WebappClassLoader objects are still "live".
> In my web application, just after a Tomcat startup, I have 2 instances
> of this class, both of which are "live".
"Live" is a critical attribute here - unless a full GC (or two) is done, unused classes and classloaders will still be present. Use JConsole to trigger GCs, if needed.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
Re: Headstart on "Resolving OOM-PermGen errors on webapp reload"
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Ken,
On 4/21/2009 4:32 PM, Ken Bowen wrote:
> I develop locally on a Mac OS/X 10.5.6 box with 4GBmem using (My)Eclipse
> and Tomcat 6.0.18 directly. With lots & lots of reloads, I'm not
> surprised that I eventually hit OOM PermGen space in this setting, but
> it happens much less often than on the CentOS box (of course 4GB != 432MB).
Can you observe your heap and which ClassLoaders are still hanging
around? You might want to check to see how many
org/apache/catalina/loader/WebappClassLoader objects are still "live".
In my web application, just after a Tomcat startup, I have 2 instances
of this class, both of which are "live".
After an application reload, I can see that I still have 2 live
WebappClassLoaders: one of them is the same one that was live before the
app reload (probably the "Shared" webapp ClassLoader - I'm using TC 5.5)
and the other one is a new one that appears to have replaced the old one
(which should be expected, since the old one should be discarded after
the undeploy).
If you find multiple WebappClassLoader objects still hanging around, you
can probably start looking for the objects they actually own. Those will
give you a clue as to what services or objects aren't being cleaned-up
correctly when your application shuts down.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAknuPiUACgkQ9CaO5/Lv0PBZ9gCgwnO0rP3j40m32+vI1jmy0qaI
9w0AoMCt+AteT1uv+vvrgCbpteNV9rGb
=X9YG
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Headstart on "Resolving OOM-PermGen errors on webapp reload"
Posted by Mark Thomas <ma...@apache.org>.
Ken Bowen wrote:
> Hey Mark,
>
> This is really good.
> Presumably this is an outline for success
It has a 100% success rate so far on a admittedly small sample size of
around 5 (I can't remember the exact number).
Mark
>-- and it is definitely an
> outline for improvement of my rather improvished skills in this area.
> Many thanks!
>
> Ken
>
> On Apr 21, 2009, at 5:54 PM, Mark Thomas wrote:
>
>> Ken Bowen wrote:
>>> Mark,
>>>
>>> Any chance we could make a headstart on "Resolving OOM-PermGen errors on
>>> webapp reload" ??
>>> Perhaps some general pointers, guidance etc. [to help you refine the
>>> talk in advance :-) ]
>>
>> The very short version.
>>
>> 1. Find an app that you can't reload without OOME
>> 2. Get a profiler - I like Yourkit
>> Full disclosure: they give ASF committers a free copy
>> 3. Reload you app once
>> 4. Use the profiler to look for instances of WebappClassLoader
>> 5. Look for the one with the started attribute == false
>> 6. Trace the GC roots for this instance
>> 7. Fix whatever is hanging on to references to the instance - there
>> should not be any
>>
>> The session (assuming it goes ahead) will do a demo of the above and
>> expand on 7 in terms of typical culprits and possible workarounds. It
>> will also explain what is going on and why this happens.
>>
>> HTH,
>>
>> Mark
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Headstart on "Resolving OOM-PermGen errors on webapp reload"
Posted by Ken Bowen <kb...@als.com>.
Hey Mark,
This is really good.
Presumably this is an outline for success -- and it is definitely an
outline for improvement of my rather improvished skills in this area.
Many thanks!
Ken
On Apr 21, 2009, at 5:54 PM, Mark Thomas wrote:
> Ken Bowen wrote:
>> Mark,
>>
>> Any chance we could make a headstart on "Resolving OOM-PermGen
>> errors on
>> webapp reload" ??
>> Perhaps some general pointers, guidance etc. [to help you refine the
>> talk in advance :-) ]
>
> The very short version.
>
> 1. Find an app that you can't reload without OOME
> 2. Get a profiler - I like Yourkit
> Full disclosure: they give ASF committers a free copy
> 3. Reload you app once
> 4. Use the profiler to look for instances of WebappClassLoader
> 5. Look for the one with the started attribute == false
> 6. Trace the GC roots for this instance
> 7. Fix whatever is hanging on to references to the instance - there
> should not be any
>
> The session (assuming it goes ahead) will do a demo of the above and
> expand on 7 in terms of typical culprits and possible workarounds. It
> will also explain what is going on and why this happens.
>
> HTH,
>
> Mark
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Headstart on "Resolving OOM-PermGen errors on webapp reload"
Posted by Mark Thomas <ma...@apache.org>.
Christopher Schultz wrote:
> Mark,
>
> On 4/21/2009 5:54 PM, Mark Thomas wrote:
>> 1. Find an app that you can't reload without OOME
>> 2. Get a profiler - I like Yourkit
>> Full disclosure: they give ASF committers a free copy
>> 3. Reload you app once
>> 4. Use the profiler to look for instances of WebappClassLoader
>> 5. Look for the one with the started attribute == false
>
> I'm using YourKit and I have located the two WebappClassLoader object
> currently in memory. I'm using TC 5.5.26 and I don't see any "started"
> attribute. Is "started" in 6.x while 5.5 has something else? Or, do I
> have to dig deeper into the tree to find the "started" attribute?
Looks lie a difference between 1.5 and 1.6 JVMs. With 1.6 I can see the
primitives when I use object explorer. With 1.5 I can't.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Headstart on "Resolving OOM-PermGen errors on webapp reload"
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Mark,
On 4/21/2009 5:54 PM, Mark Thomas wrote:
> 1. Find an app that you can't reload without OOME
> 2. Get a profiler - I like Yourkit
> Full disclosure: they give ASF committers a free copy
> 3. Reload you app once
> 4. Use the profiler to look for instances of WebappClassLoader
> 5. Look for the one with the started attribute == false
I'm using YourKit and I have located the two WebappClassLoader object
currently in memory. I'm using TC 5.5.26 and I don't see any "started"
attribute. Is "started" in 6.x while 5.5 has something else? Or, do I
have to dig deeper into the tree to find the "started" attribute?
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAknwosQACgkQ9CaO5/Lv0PCOaACfasn/yTePFbRAXzopZIX3UKYl
jysAnj7Thed2QQJGV2OTKX0b/aHkUQL2
=d3p2
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Headstart on "Resolving OOM-PermGen errors on webapp reload"
Posted by Mark Thomas <ma...@apache.org>.
Ken Bowen wrote:
> Mark,
>
> Any chance we could make a headstart on "Resolving OOM-PermGen errors on
> webapp reload" ??
> Perhaps some general pointers, guidance etc. [to help you refine the
> talk in advance :-) ]
The very short version.
1. Find an app that you can't reload without OOME
2. Get a profiler - I like Yourkit
Full disclosure: they give ASF committers a free copy
3. Reload you app once
4. Use the profiler to look for instances of WebappClassLoader
5. Look for the one with the started attribute == false
6. Trace the GC roots for this instance
7. Fix whatever is hanging on to references to the instance - there
should not be any
The session (assuming it goes ahead) will do a demo of the above and
expand on 7 in terms of typical culprits and possible workarounds. It
will also explain what is going on and why this happens.
HTH,
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Headstart on "Resolving OOM-PermGen errors on webapp reload"
Posted by Ma...@McAfee.com.
I don't doubt that jmap/jhat would be able to give you more detailed information. My exact goal was to come up with something for automated testing that would help prevent classloader leaks from making it into production. If someone can think of a programmatic way to do that with jmap/jhat, please share!
Mark
-----Original Message-----
From: Caldarale, Charles R [mailto:Chuck.Caldarale@unisys.com]
Sent: Wednesday, April 22, 2009 10:30 AM
To: Tomcat Users List
Subject: RE: Headstart on "Resolving OOM-PermGen errors on webapp reload"
> From: Mark_Despain@McAfee.com [mailto:Mark_Despain@McAfee.com]
> Subject: RE: Headstart on "Resolving OOM-PermGen errors on webapp
> reload"
>
> Yeah, Insane just using reflection and a graph traversal algorithm to
> get the job done. It looks like this is implemented by
> org.netbeans.insane.impl.InsaneEngine.
Other than being programmable for automated testing purposes, does this provide any more or different information than a jmap/jhat combo?
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
RE: Headstart on "Resolving OOM-PermGen errors on webapp reload"
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Mark_Despain@McAfee.com [mailto:Mark_Despain@McAfee.com]
> Subject: RE: Headstart on "Resolving OOM-PermGen errors on webapp
> reload"
>
> Yeah, Insane just using reflection and a graph traversal algorithm to
> get the job done. It looks like this is implemented by
> org.netbeans.insane.impl.InsaneEngine.
Other than being programmable for automated testing purposes, does this provide any more or different information than a jmap/jhat combo?
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
RE: Headstart on "Resolving OOM-PermGen errors on webapp reload"
Posted by Ma...@McAfee.com.
Yeah, Insane just using reflection and a graph traversal algorithm to get the job done. It looks like this is implemented by org.netbeans.insane.impl.InsaneEngine.
Oh, and I found my copy of the Insane source. The third argument to ScannerUtils.scan() should be true since that is what signals to InsaneEngine that static fields should be traversed during the heap walk.
~Mark
-----Original Message-----
From: Christopher Schultz [mailto:chris@christopherschultz.net]
Sent: Wednesday, April 22, 2009 9:05 AM
To: Tomcat Users List
Subject: Re: Headstart on "Resolving OOM-PermGen errors on webapp reload"
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Mark,
On 4/21/2009 10:27 PM, Mark_Despain@McAfee.com wrote:
> Ok, so my wife actually wrote a couple of month ago in Japanese about
> using strategy for leveraging the Insane library and a continuous
> integration server in order to prevent webapp classloader leakage
> issues from creeping in.
I'll definitely take a look at this (in English -- tell her thanks!).
> With this in place, you can then setup your test environment to
> exercise a given webapp, shut it down, and then invoke your
> ScannerUtils code to see if that the webapp's classloader is still
> hanging around.
This is super sexy! What a nice job. I'll have to read-up on the Insane
library, but my suspicion is that you probably don't really need it...
all the RTTI information is available from the objects themselves, and
the code should be relatively simple.... just tons and tons of loops and
recursive calls.
> A word of warning... this is a very heavy weight operation.
Heh, you think? That's why this type of testing should be done in
development and not in production ;
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAknvQCkACgkQ9CaO5/Lv0PC5OwCeONLPIu7BAaBiwGhEbuYm4caf
d/4An2TpoymWDAi2/o4fi/sRwNpqxROy
=sL8m
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Headstart on "Resolving OOM-PermGen errors on webapp reload"
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Mark,
On 4/21/2009 10:27 PM, Mark_Despain@McAfee.com wrote:
> Ok, so my wife actually wrote a couple of month ago in Japanese about
> using strategy for leveraging the Insane library and a continuous
> integration server in order to prevent webapp classloader leakage
> issues from creeping in.
I'll definitely take a look at this (in English -- tell her thanks!).
> With this in place, you can then setup your test environment to
> exercise a given webapp, shut it down, and then invoke your
> ScannerUtils code to see if that the webapp's classloader is still
> hanging around.
This is super sexy! What a nice job. I'll have to read-up on the Insane
library, but my suspicion is that you probably don't really need it...
all the RTTI information is available from the objects themselves, and
the code should be relatively simple.... just tons and tons of loops and
recursive calls.
> A word of warning... this is a very heavy weight operation.
Heh, you think? That's why this type of testing should be done in
development and not in production ;
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAknvQCkACgkQ9CaO5/Lv0PC5OwCeONLPIu7BAaBiwGhEbuYm4caf
d/4An2TpoymWDAi2/o4fi/sRwNpqxROy
=sL8m
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [OT] Headstart on "Resolving OOM-PermGen errors on webapp reload"
Posted by André Warnier <aw...@ice-sa.com>.
Mark_Despain@McAfee.com wrote:
...
Being named DeSpain, having a wife able to write about Java GC in
Japanese and English, and being oneself able to write eloquently about
an Insane Java library and its usage with Tomcat..
This world is full of wonders.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Headstart on "Resolving OOM-PermGen errors on webapp reload"
Posted by Ma...@McAfee.com.
Ok, so my wife actually wrote a couple of month ago in Japanese about using strategy for leveraging the Insane library and a continuous integration server in order to prevent webapp classloader leakage issues from creeping in. If you can read Japanese, check out
http://pdxwhitebox.blog118.fc2.com/blog-entry-33.html
For those of us who can't read Japanese, today she posted the English text it was based upon here:
http://pdxwhitebox.blog118.fc2.com/blog-entry-161.html
The information there is a little light on the specifics, but it does a great job at describing the strategy, the rationale, where to find Insane, and which API to use. When you read it, treat "plugins" as interchangeable with "webapps".
Getting a little more specific, we leveraged Insane's ScannerUtils.scan() method to search for webapp classloaders that should not be reachable via a strong reference. Doing this involved (1) implementing a org.netbeans.insane.scanner.Visitor that gathers relevant classloaders and (2) implementing a org.netbeans.insane.scanner.Filter that causes weak/soft/phantom references to be skipped over by ScannerUtils.scan().
(1) To implement the Visitor class, just have visitObject(ObjectMap map, Object object) collect relevant classloaders passed as the second argument (probably instances of org.apache.catalina.loader.WebappClassLoader). All other methods can have empty implementations.
(2) To implement the Filter class, have accept(Object obj, Object referredFrom, Field reference) return false if the second argument is an instance of SoftReference, WeakReference, or PhantomReference. Return true for everything else.
With the Visitor and Filter implemented, the scanning code would look something like the following:
Set<Object> roots = ScannerUtils.interestingRoots();
// add to this set anything else you'd like to make sure gets visits
MyWebAppClassLoaderGatherer visitor = new MyWebAppClassLoaderGatherer();
MySkipWeakReferencesFilter filter = new MySkipWeakReferencesFilter();
ScannerUtils.scan( filter, visitor, roots, true );
// do whatever is needed with the classloaders gathered by the visitor
Unfortunately, I don't remember why the last argument to ScannerUtils.scan() should be true, and I don't have the Insane source or javadoc handy to remind me. Sorry about that...
With this in place, you can then setup your test environment to exercise a given webapp, shut it down, and then invoke your ScannerUtils code to see if that the webapp's classloader is still hanging around. This seems to work well with in combination with a continuous integration (CI) server whose job is to run tests against checkins all day. If your CI server tracks who checked it what, you'll be able to quickly narrow down what may have caused the leak, assuming that your web app wasn't leaking a classloader to begin with.
A word of warning... this is a very heavy weight operation. If there are better ways this can be done, I'd love to hear about it!
Best regards,
Mark DeSpain
-----Original Message-----
From: Despain, Mark
Sent: Tuesday, April 21, 2009 4:01 PM
To: users@tomcat.apache.org
Subject: RE: Headstart on "Resolving OOM-PermGen errors on webapp reload"
Hi Chris,
I'll follow up later tonight. Hopefully I'll have less typos then, but don't be surprised if I just go triple-negative instead :)
~Mark
-----Original Message-----
From: Christopher Schultz [mailto:chris@christopherschultz.net]
Sent: Tuesday, April 21, 2009 3:47 PM
To: Tomcat Users List
Subject: Re: Headstart on "Resolving OOM-PermGen errors on webapp reload"
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Mark,
On 4/21/2009 6:17 PM, Mark_Despain@McAfee.com wrote:
> None of the issues I've looked into have never been attributed to
> Tomcat.
You mean "ever" attributed to Tomcat, right? Good. ;)
> * A webapp registering an object with another object that outlives
> the webapp and forgetting to unregister it webapp shutdown. As a
> result, that object cannot be garbage collected, which also prevents
> that webapp's classloader from being garbage collected.
It's worse than that: even if most of the objects created from classes
loaded by that ClassLoader have gone out of scope and can be GC'd, the
java.lang.Class, java.lang.Package, and java.lang.reflect.* objects that
are created from those loaded classes won't be released until the
ClassLoader is released. So if you have a single object that is being
held by an object loaded by another ClassLoader, you can have
potentially hundreds or thousands of objects left around in memory just
wasting space.
> * Instantiating a new Thread somehow (e.g. directly via its
> constructor, or indirectly via a thread pool, a Timer, or a
> ScheduledExecutorService) in response to a request to a webapp. This
> is because, by default, the new Thread inherits the parent thread's
> context classloader, which in Tomcat (5.5.x) is set to that webapp's
> classloader.
This is a very good point to consider, because many folks aren't aware
of the thread->classloader relationship. If you don't kill your threads,
you will keep your whole app's ClassLoader around forever. The
'daemonness' of the thread is not relevant. I'm looking at you, Quartz!
> To help pro-actively detect webapp classloader garbage collection
> issues, I've leveraged the Insane library (a library I found from
> Netbeans while researching the topic) to write a utility that
> searches for webapp classloaders that should have been garbage
> collected. Using the utility in combination with automated tests has
> been definitely helped catch and diagnose issues.
Could you post some more info on this? I'm sure a lot of folks would be
interested in this kind of thing.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAknuTNYACgkQ9CaO5/Lv0PCN6wCgkDKwhzpRB7re4StuClVe1Rt/
3K0Anj5eXjLiTql97dxbhrNFavPXGIvC
=Iuos
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Headstart on "Resolving OOM-PermGen errors on webapp reload"
Posted by Ma...@McAfee.com.
Hi Chris,
I'll follow up later tonight. Hopefully I'll have less typos then, but don't be surprised if I just go triple-negative instead :)
~Mark
-----Original Message-----
From: Christopher Schultz [mailto:chris@christopherschultz.net]
Sent: Tuesday, April 21, 2009 3:47 PM
To: Tomcat Users List
Subject: Re: Headstart on "Resolving OOM-PermGen errors on webapp reload"
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Mark,
On 4/21/2009 6:17 PM, Mark_Despain@McAfee.com wrote:
> None of the issues I've looked into have never been attributed to
> Tomcat.
You mean "ever" attributed to Tomcat, right? Good. ;)
> * A webapp registering an object with another object that outlives
> the webapp and forgetting to unregister it webapp shutdown. As a
> result, that object cannot be garbage collected, which also prevents
> that webapp's classloader from being garbage collected.
It's worse than that: even if most of the objects created from classes
loaded by that ClassLoader have gone out of scope and can be GC'd, the
java.lang.Class, java.lang.Package, and java.lang.reflect.* objects that
are created from those loaded classes won't be released until the
ClassLoader is released. So if you have a single object that is being
held by an object loaded by another ClassLoader, you can have
potentially hundreds or thousands of objects left around in memory just
wasting space.
> * Instantiating a new Thread somehow (e.g. directly via its
> constructor, or indirectly via a thread pool, a Timer, or a
> ScheduledExecutorService) in response to a request to a webapp. This
> is because, by default, the new Thread inherits the parent thread's
> context classloader, which in Tomcat (5.5.x) is set to that webapp's
> classloader.
This is a very good point to consider, because many folks aren't aware
of the thread->classloader relationship. If you don't kill your threads,
you will keep your whole app's ClassLoader around forever. The
'daemonness' of the thread is not relevant. I'm looking at you, Quartz!
> To help pro-actively detect webapp classloader garbage collection
> issues, I've leveraged the Insane library (a library I found from
> Netbeans while researching the topic) to write a utility that
> searches for webapp classloaders that should have been garbage
> collected. Using the utility in combination with automated tests has
> been definitely helped catch and diagnose issues.
Could you post some more info on this? I'm sure a lot of folks would be
interested in this kind of thing.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAknuTNYACgkQ9CaO5/Lv0PCN6wCgkDKwhzpRB7re4StuClVe1Rt/
3K0Anj5eXjLiTql97dxbhrNFavPXGIvC
=Iuos
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Headstart on "Resolving OOM-PermGen errors on webapp reload"
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Mark,
On 4/21/2009 6:17 PM, Mark_Despain@McAfee.com wrote:
> None of the issues I've looked into have never been attributed to
> Tomcat.
You mean "ever" attributed to Tomcat, right? Good. ;)
> * A webapp registering an object with another object that outlives
> the webapp and forgetting to unregister it webapp shutdown. As a
> result, that object cannot be garbage collected, which also prevents
> that webapp's classloader from being garbage collected.
It's worse than that: even if most of the objects created from classes
loaded by that ClassLoader have gone out of scope and can be GC'd, the
java.lang.Class, java.lang.Package, and java.lang.reflect.* objects that
are created from those loaded classes won't be released until the
ClassLoader is released. So if you have a single object that is being
held by an object loaded by another ClassLoader, you can have
potentially hundreds or thousands of objects left around in memory just
wasting space.
> * Instantiating a new Thread somehow (e.g. directly via its
> constructor, or indirectly via a thread pool, a Timer, or a
> ScheduledExecutorService) in response to a request to a webapp. This
> is because, by default, the new Thread inherits the parent thread's
> context classloader, which in Tomcat (5.5.x) is set to that webapp's
> classloader.
This is a very good point to consider, because many folks aren't aware
of the thread->classloader relationship. If you don't kill your threads,
you will keep your whole app's ClassLoader around forever. The
'daemonness' of the thread is not relevant. I'm looking at you, Quartz!
> To help pro-actively detect webapp classloader garbage collection
> issues, I've leveraged the Insane library (a library I found from
> Netbeans while researching the topic) to write a utility that
> searches for webapp classloaders that should have been garbage
> collected. Using the utility in combination with automated tests has
> been definitely helped catch and diagnose issues.
Could you post some more info on this? I'm sure a lot of folks would be
interested in this kind of thing.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAknuTNYACgkQ9CaO5/Lv0PCN6wCgkDKwhzpRB7re4StuClVe1Rt/
3K0Anj5eXjLiTql97dxbhrNFavPXGIvC
=Iuos
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Headstart on "Resolving OOM-PermGen errors on webapp reload"
Posted by Ma...@McAfee.com.
Hi Ken (and Mark),
Different "Mark" here... I'm new to this mailing list and am not a Tomcat developer. Forgive me if I'm interrupting your thread, but I am very interested in this topic, since I've spent a fair amount of time debugging OOM-PermGen errors within Tomcat (5.5.x). I would be interested in hearing any tips or tricks regarding OM-PermGen, too, and am happy to share my experiences regarding this.
None of the issues I've looked into have never been attributed to Tomcat. They have all been application issues related to an inability to garbage collect a webapp's classloader. This root causes were split between one of the following:
* A webapp registering an object with another object that outlives the webapp and forgetting to unregister it webapp shutdown. As a result, that object cannot be garbage collected, which also prevents that webapp's classloader from being garbage collected. Examples of things that outlive a webapp or objects from libraries within Tomcat's shared library folder, Tomcat itself, and the JRE itself. For example, if a webapp registers a JDBC driver or a JCA/JCE provider, it should take care to unregister it on shutdown.
* Instantiating a new Thread somehow (e.g. directly via its constructor, or indirectly via a thread pool, a Timer, or a ScheduledExecutorService) in response to a request to a webapp. This is because, by default, the new Thread inherits the parent thread's context classloader, which in Tomcat (5.5.x) is set to that webapp's classloader. If that Thread keeps the context classloader reference and outlives the webapp, the webapp's classloader cannot be garbage collected.
As an aside, those causes highlight a reason why people should be weary of putting a new JAR files into one of Tomcat's shared library folders. Doing so requires that a webapp be vigilant about cleaning up any interactions with that library on shutdown. Said library also needs to provide a means to do the cleanup, as well as be sensitive to context classloader issues.
When trying to debug these issues, I typically load things up in a profiler, stop the webapp and see of its classloader still resides in memory. When it exists, I use the profiler's "find the garbage collection root" to help determine why the classloader still resides in memory and figure out how to fix the issue. If using JProfiler (and probably other profilers), I think you have to tell it to go not stop when it reaches the Class object along the reference path.
To help pro-actively detect webapp classloader garbage collection issues, I've leveraged the Insane library (a library I found from Netbeans while researching the topic) to write a utility that searches for webapp classloaders that should have been garbage collected. Using the utility in combination with automated tests has been definitely helped catch and diagnose issues.
Apologies if this wasn't helpful. Please let me know if I'm wrong, outdated, or if there is a better way!
Best regards,
Mark DeSpain
-----Original Message-----
From: Ken Bowen [mailto:kbowen@als.com]
Sent: Tuesday, April 21, 2009 1:33 PM
To: Tomcat Users List
Subject: Headstart on "Resolving OOM-PermGen errors on webapp reload"
Mark,
Any chance we could make a headstart on "Resolving OOM-PermGen errors
on webapp reload" ??
Perhaps some general pointers, guidance etc. [to help you refine the
talk in advance :-) ]
The manager app is giving me more & more of:
FAIL - Application at context path /ctx could not be started
FAIL - Encountered exception java.lang.OutOfMemoryError: PermGen space
This is on a Tomcat 6.0.18 (java 1.5) running on a remote Linux CentOS
5, in a Parallels VM, with httpd + mod_jk, running two versions of the
app. Total memory available is 432 MB, with a number of non-system,
non-TC processes, including ActiveMQ and 9 Postres processes. But
there appears to be about 200MB of unused memory according to the
Parallels control panel.
I develop locally on a Mac OS/X 10.5.6 box with 4GBmem using
(My)Eclipse and Tomcat 6.0.18 directly. With lots & lots of reloads,
I'm not surprised that I eventually hit OOM PermGen space in this
setting, but it happens much less often than on the CentOS box (of
course 4GB != 432MB).
Thanks...Ken
Mark Thomas wrote:
> I was at a conference recently and rather rashly made the statement
> "OOM
> PermGen errors on reload are an application issue, not a Tomcat one".
> Several people came up to me afterwards with variations on a theme of
> "Our app has OOM on reload - put your money where your mouth is and
> show
> us where we have gone wrong"
>
> Having helped several people track down the causes of these errors
> (and
> yes they were all application issues) it occurred to me that there may
> be interest in this at ApacheCon. So my suggestion is:
>
> Title: Resolving OOM-PermGen errors on webapp reload
> Type: Session
> Abstract: What causes the error. Typical root causes.
> How to find and fix the root causes. With live demo.
> Audience: Developer/SysAdmin
>
> Thoughts?
>
> Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org