You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Christopher Schultz <ch...@christopherschultz.net> on 2012/11/05 21:11:34 UTC
WebappClassLoader weak/soft reachable
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
All,
I was playing around with the manager webapp today and just for grins,
I redeployed my webapp and then hit the "Find Leaks" button to see how
long it would take Tomcat to decide that my webapp reloads cleanly.
It didn't. That is, my webapp does not appear to cleanly reload. Which
was a surprise to me.
I waited a few moments and tried again: Find Leaks still says "there's
a leak, confirm with a profiler". So I attached with YourKit, took a
memory snapshot and started looking around: I found 4
WebappClassLoaders (1 for manager, 1 for my webapp, 1 for a dummy ROOT
webapp I have and finally a started=false WebappClassLoader for the
old pre-reload of my webapp).
Interestingly, the started=false WebappClassLoader says that it is
Weak/Soft Reachable. That means that tracing GC roots doesn't really work.
Requesting GC via the manager changes nothing. Requesting GC via the
profiler changes nothing.
Does anyone have any good ideas for tracking-down what is keeping this
WebappClassLoader in memory? Do I just have to intentionally exhaust
the heap to convince the MM/GC that it should toss-out this WCL?
Any suggestions would be greatly appreciated.
Thanks,
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
iEYEARECAAYFAlCYHXYACgkQ9CaO5/Lv0PBOGwCguV4xkhvHZNGtpNEVy3nNUht0
X4cAoLB/OTKvHNMln2Jx3bfrK1TZ4PFo
=D11z
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: WebappClassLoader weak/soft reachable
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Konstantin,
On 11/6/12 5:14 PM, Konstantin Kolinko wrote:
> You are on your own with this. I would not have much time to dig
> into this further. Maybe others can help.
Okay, thanks for the help thus far.
> I think those chains of weak references are not an issue.
They shouldn't be, but Yourkit says there are no other paths from GC
roots to the WebappClassLoader, so its the only thing I have to go on
for now.
> Maybe you could remove parts of your configuration, of your webapp,
> until you have some minimal configuration that still reproduces
> the issue?
Of course, that's the plan from here on out. :)
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
iEYEARECAAYFAlCahkcACgkQ9CaO5/Lv0PAHcgCdFU9drCpuo04+Jayyuiln77gZ
4CsAn3ewEkxYsQmnilxdjvX3mTmSnfvP
=JybX
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: WebappClassLoader weak/soft reachable
Posted by Konstantin Kolinko <kn...@gmail.com>.
2012/11/7 Christopher Schultz <ch...@christopherschultz.net>:
>
>> I do not know about the dbcp one. It might be an issue. It
>> certainly will contain a stack trace (it is what it is used for).
>> I usually run with logAbandoned being turned off. Do you have
>> opened connections in your pool? Do you have closeMethod="close" on
>> your pool definition?
>
> Here's my <Resource>:
> <Resource name="..."
> description="..."
> auth="Container"
> type="javax.sql.DataSource"
> maxActive="1"
> maxIdle="1"
> maxWait="10000"
> url="..."
> username="..."
> password="..."
> driverClassName="com.mysql.jdbc.Driver"
> removeAbandoned="true"
> removeAbandonedTimeout="30"
> logAbandoned="true"
> testOnBorrow="true"
> validationQuery="/* ping */ SELECT 1"
> />
>
> So, I have no closeMethod defined for my <Resource>, so I suspect that
> I get the default. I couldn't find any documentation on DBCP
> closeMethod :(
>
It is an attribute used by Tomcat itself, not DBCP:
http://tomcat.apache.org/tomcat-7.0-doc/config/context.html#Resource_Definitions
(FIXME: should be mentioned here as well, as well as the "singleton" attribute:
http://tomcat.apache.org/tomcat-7.0-doc/config/globalresources.html#Resource_Definitions
)
> I'm happy to provide a memory snapshot or even a screenshot ...
You are on your own with this. I would not have much time to dig into
this further. Maybe others can help. I think those chains of weak
references are not an issue.
Maybe you could remove parts of your configuration, of your webapp,
until you have some minimal configuration that still reproduces the
issue?
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: WebappClassLoader weak/soft reachable
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Konstantin,
On 11/6/12 4:19 PM, Konstantin Kolinko wrote:
> 2012/11/7 Christopher Schultz <ch...@christopherschultz.net>:
>>
>> These are the actual Exception objects I can see:
>>
>> 3 instances of
>> org.apache.tomcat.util.buf.UDecoder$DecodeException
>>
>> 1 instance of
>> org.apache.tomcat.dbcp.dbcp.AbandonedTrace$AbandonedObjectException
>>
>>
>>
(I am using a Tomcat-configured JNDI DataSource)
>>
>> 1 instance of
>> org.apache.naming.resources.ImmutableNameNotFoundException
>>
>
> The first and the last one were dealt by
> https://issues.apache.org/bugzilla/show_bug.cgi?id=50460
Okay. Would those be triggered even when no requests go to my webapp?
Also, given that 50460 was fixed in 7.0.6 and I'm running 7.0.32, does
that suggest a regression or an unlikely cause for what I'm observing?
> I do not know about the dbcp one. It might be an issue. It
> certainly will contain a stack trace (it is what it is used for).
> I usually run with logAbandoned being turned off. Do you have
> opened connections in your pool? Do you have closeMethod="close" on
> your pool definition?
Here's my <Resource>:
<Resource name="..."
description="..."
auth="Container"
type="javax.sql.DataSource"
maxActive="1"
maxIdle="1"
maxWait="10000"
url="..."
username="..."
password="..."
driverClassName="com.mysql.jdbc.Driver"
removeAbandoned="true"
removeAbandonedTimeout="30"
logAbandoned="true"
testOnBorrow="true"
validationQuery="/* ping */ SELECT 1"
/>
So, I have no closeMethod defined for my <Resource>, so I suspect that
I get the default. I couldn't find any documentation on DBCP
closeMethod :(
>> jdk1.7.0_09.jdk Tomcat 7.0.32
>>
>
> I wonder whether there are any differences wrt to jdk 1.6.
I can run under jdk1.6 if that would help narrow the problem.
Back to the session issue, I was able to determine that my ClientPrefs
object is *not* the problem (or, at least, not the *only* problem)
because I removed the Filter that causes it to be placed into the
session and reloading still caused leaks. My preferences class had no
objects in memory at all (neither the data object nor the filter
itself, since I had commented it out entirely in web.xml).
Back to the WebappClassLoader, I have this as the shortest (Weak/Soft)
GC path for it:
[Weak/Soft Reachable] org.apache.catalina.loader.WebappClassLoader
<- referent of java.util.WeakHashMap$Entry
<- [15] of java.util.WeakHashMap$Entry[16]
<- table of java.util.WeakHashMap
<- loaderToCache of java.lang.reflect.Proxy [Class]
There are several other references that look like the above, and the
first object in the list (after the WebappClassLoader) does *not* say
"referent of [Weak/Soft Reachable] java.util.WeakHashMap$Entry" like
pretty much all the other objects that refer to the WebappClassLoader
(mostly classloader fields for java.lang.Class objects.
But, since the whole point of WeakHashMap is supposed to be its use of
weak references, perhaps that's not a problem?
I'm happy to provide a memory snapshot or even a screenshot if that
would help illustrate things a bit more.
Thanks for your help.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
iEYEARECAAYFAlCZg+MACgkQ9CaO5/Lv0PC8YgCgmoe2JDfd23ySxBG+V04byDDg
7hUAoIMsRdPn5IoCY+Dl3KZi+8qpnc3U
=Tdc3
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: WebappClassLoader weak/soft reachable
Posted by Konstantin Kolinko <kn...@gmail.com>.
2012/11/7 Christopher Schultz <ch...@christopherschultz.net>:
>
> These are the actual Exception objects I can see:
>
> 3 instances of org.apache.tomcat.util.buf.UDecoder$DecodeException
>
> 1 instance of
> org.apache.tomcat.dbcp.dbcp.AbandonedTrace$AbandonedObjectException
>
> (I am using a Tomcat-configured JNDI DataSource)
>
> 1 instance of
> org.apache.naming.resources.ImmutableNameNotFoundException
>
The first and the last one were dealt by
https://issues.apache.org/bugzilla/show_bug.cgi?id=50460
I do not know about the dbcp one. It might be an issue. It certainly
will contain a stack trace (it is what it is used for). I usually run
with logAbandoned being turned off. Do you have opened connections in
your pool? Do you have closeMethod="close" on your pool definition?
>
> jdk1.7.0_09.jdk
> Tomcat 7.0.32
>
I wonder whether there are any differences wrt to jdk 1.6.
(...)
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: WebappClassLoader weak/soft reachable
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Konstantin,
On 11/6/12 2:59 PM, Konstantin Kolinko wrote:
> 2012/11/6 Christopher Schultz <ch...@christopherschultz.net>:
>> On 11/6/12 10:37 AM, Konstantin Kolinko wrote:
>>> 2012/11/6 Christopher Schultz <ch...@christopherschultz.net>:
>>
>>> 2) I know of a bug in JVM when a certain kind of leak goes
>>> undetected. I mean the leak through a shared Exception
>>> instance and its native stacktrace. See
>>>
>>> https://issues.apache.org/bugzilla/show_bug.cgi?id=50460
>>> https://issues.apache.org/bugzilla/show_bug.cgi?id=53936
>>> (Linked from
>>> https://wiki.apache.org/tomcat/MemoryLeakProtection)
>>
>> I don't create any of my own Exception objects that get shared.
>> Could this be something introduced by Tomcat itself, or more
>> likely some dependent library of mine?
>>
>> Since I can't readily trace the WebappClassLoader to its GC
>> roots, do you have any suggestions for identifying the specific
>> cause in my case?
>>
>
> IIRC, I found that issue by looking for Exception objects that
> were present in the memory. I was using Eclipse MAT. IIRC I
> filtered the living objects by their class name.
Okay, the good news is that there are very few objects with the word
"Exception" in their name. Better still, many of them are classes like
ExceptionHandler and thus not likely to be exceptions themselves.
These are the actual Exception objects I can see:
3 instances of org.apache.tomcat.util.buf.UDecoder$DecodeException
1 instance of
org.apache.tomcat.dbcp.dbcp.AbandonedTrace$AbandonedObjectException
(I am using a Tomcat-configured JNDI DataSource)
1 instance of
org.apache.naming.resources.ImmutableNameNotFoundException
The few other Exception objects that I can see, I inspected and they
said "no GC paths found" even when including soft/weak/finalizer
references.
Does that help at all? I can provide details about the GC paths for
the above Tomcat-related exceptions if that helps.
jdk1.7.0_09.jdk
Tomcat 7.0.32
I realized when disclosing the Tomcat version that I had been running
off Tomcat 7.0.x-trunk, so I have repeated my tests on Tomcat 7.0.32
with a bit more information:
1. Tomcat start
2. List applications in manager - no active sessions in main webapp
3. Reload target webapp
4. Find leaks: none reported
5. Reload webapp again
6. Find leaks again: none reported
So far, so good.
7. Access protected page in target webapp (session is created),
no authentication, yet (Tomcat redirects to login page, which
invokes some Apache Struts routing code and a simple Velocity
template)
8. Reload & find leaks: found a leak
Performed 3 full GCs from within Yourkit, then took a memory snapshot
for inspection. I see the aforementioned Exception objects in memory,
but they all have o.a.c.loader.StandardClassLoader as their parent
ClassLoader.
Yourkit lists the following interesting memory tidbits:
Objects unreachable from GC roots, but not yet collected: 68358
Objects reachable from GC roots via weak and/or soft references only:
35215
Objects pending finalization (finalizer queue objects unreachable via
strong references): 288
It still looks like there is quite a bit of garbage laying around,
even after 3 full GC requests (with a noticeable drop in memory usage
when having performed my 3-GCs-in-a-row above, so the JVM isn't
ignoring GC requests).
Let's keep going. There's a session in memory.
9. Stop Tomcat (sessions are persisted to disk)
10. Start Tomcat (sessions are loaded from disk)
11. List Applications, confirm 1 active session
12. Inspect session. It contains the following objects:
java.util.Locale
A ClientPreferences object for our own application (basically a
wrapper around Map<String,String>)
13. Reload, find leaks: found a leak
So, it doesn't even require that I actually access a resource within
my webapp: it is enough to have an active session.
14. Expire all sessions
15. Reload, find leaks: found *2* leaks
16. Stop Tomcat (no sessions persisted: all expired in step #14)
17. Start Tomcat
18. Reload, find leaks: none found
So something my session loading triggers is causing the leak it seems.
I'll take a closer look at the ClientPrefs class to see if there's
anything odd going on in there.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
iEYEARECAAYFAlCZevkACgkQ9CaO5/Lv0PCRmACfV22N21CSLqp6O+Y5i7kQ0ftc
GNUAn01EXXhGfByIwoCZPcjocG0iC9hB
=kXGO
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: WebappClassLoader weak/soft reachable
Posted by Konstantin Kolinko <kn...@gmail.com>.
2012/11/6 Christopher Schultz <ch...@christopherschultz.net>:
> On 11/6/12 10:37 AM, Konstantin Kolinko wrote:
>> 2012/11/6 Christopher Schultz <ch...@christopherschultz.net>:
>
>> 2) I know of a bug in JVM when a certain kind of leak goes
>> undetected. I mean the leak through a shared Exception instance
>> and its native stacktrace. See
>>
>> https://issues.apache.org/bugzilla/show_bug.cgi?id=50460
>> https://issues.apache.org/bugzilla/show_bug.cgi?id=53936 (Linked
>> from https://wiki.apache.org/tomcat/MemoryLeakProtection)
>
> I don't create any of my own Exception objects that get shared. Could
> this be something introduced by Tomcat itself, or more likely some
> dependent library of mine?
>
> Since I can't readily trace the WebappClassLoader to its GC roots, do
> you have any suggestions for identifying the specific cause in my case?
>
IIRC, I found that issue by looking for Exception objects that were
present in the memory. I was using Eclipse MAT. IIRC I filtered the
living objects by their class name.
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: WebappClassLoader weak/soft reachable
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Konstantin,
On 11/6/12 10:37 AM, Konstantin Kolinko wrote:
> 2012/11/6 Christopher Schultz <ch...@christopherschultz.net>:
>>
>> I was playing around with the manager webapp today and just for
>> grins, I redeployed my webapp and then hit the "Find Leaks"
>> button to see how long it would take Tomcat to decide that my
>> webapp reloads cleanly.
>>
>> It didn't. That is, my webapp does not appear to cleanly reload.
>> Which was a surprise to me.
>>
>> I waited a few moments and tried again: Find Leaks still says
>> "there's a leak, confirm with a profiler". So I attached with
>> YourKit, took a memory snapshot and started looking around: I
>> found 4 WebappClassLoaders (1 for manager, 1 for my webapp, 1
>> for a dummy ROOT webapp I have and finally a started=false
>> WebappClassLoader for the old pre-reload of my webapp).
>>
>> Interestingly, the started=false WebappClassLoader says that it
>> is Weak/Soft Reachable. That means that tracing GC roots doesn't
>> really work.
>>
>> Requesting GC via the manager changes nothing. Requesting GC via
>> the profiler changes nothing.
>>
>> Does anyone have any good ideas for tracking-down what is
>> keeping this WebappClassLoader in memory? Do I just have to
>> intentionally exhaust the heap to convince the MM/GC that it
>> should toss-out this WCL?
>>
>> Any suggestions would be greatly appreciated.
>>
>
> 1) There exists an -X option that disables explicit GC calls. I
> hope that you are not using it.
No, I don't do any of that foolishness.
> 2) I know of a bug in JVM when a certain kind of leak goes
> undetected. I mean the leak through a shared Exception instance
> and its native stacktrace. See
>
> https://issues.apache.org/bugzilla/show_bug.cgi?id=50460
> https://issues.apache.org/bugzilla/show_bug.cgi?id=53936 (Linked
> from https://wiki.apache.org/tomcat/MemoryLeakProtection)
I don't create any of my own Exception objects that get shared. Could
this be something introduced by Tomcat itself, or more likely some
dependent library of mine?
Since I can't readily trace the WebappClassLoader to its GC roots, do
you have any suggestions for identifying the specific cause in my case?
Thanks,
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
iEYEARECAAYFAlCZZTsACgkQ9CaO5/Lv0PCcvgCgh3/8/66jfTVNzsTtd2M1S8vY
Ze4An0k9BLL6AydveTuakEjRTy3xzkb1
=+FsG
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: WebappClassLoader weak/soft reachable
Posted by Konstantin Kolinko <kn...@gmail.com>.
2012/11/6 Christopher Schultz <ch...@christopherschultz.net>:
>
> I was playing around with the manager webapp today and just for grins,
> I redeployed my webapp and then hit the "Find Leaks" button to see how
> long it would take Tomcat to decide that my webapp reloads cleanly.
>
> It didn't. That is, my webapp does not appear to cleanly reload. Which
> was a surprise to me.
>
> I waited a few moments and tried again: Find Leaks still says "there's
> a leak, confirm with a profiler". So I attached with YourKit, took a
> memory snapshot and started looking around: I found 4
> WebappClassLoaders (1 for manager, 1 for my webapp, 1 for a dummy ROOT
> webapp I have and finally a started=false WebappClassLoader for the
> old pre-reload of my webapp).
>
> Interestingly, the started=false WebappClassLoader says that it is
> Weak/Soft Reachable. That means that tracing GC roots doesn't really work.
>
> Requesting GC via the manager changes nothing. Requesting GC via the
> profiler changes nothing.
>
> Does anyone have any good ideas for tracking-down what is keeping this
> WebappClassLoader in memory? Do I just have to intentionally exhaust
> the heap to convince the MM/GC that it should toss-out this WCL?
>
> Any suggestions would be greatly appreciated.
>
1) There exists an -X option that disables explicit GC calls. I hope
that you are not using it.
2) I know of a bug in JVM when a certain kind of leak goes undetected.
I mean the leak through a shared Exception instance and its native
stacktrace. See
https://issues.apache.org/bugzilla/show_bug.cgi?id=50460
https://issues.apache.org/bugzilla/show_bug.cgi?id=53936
(Linked from https://wiki.apache.org/tomcat/MemoryLeakProtection)
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org