You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Christopher Schultz <ch...@christopherschultz.net> on 2013/05/31 17:39:57 UTC
WebappClassLoaders pinned in memory by java.security.ProtectionDomain
objects
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
All,
My primary webapp (and ClassLoader) used to unload cleanly but now I
get WebappClassLoaders still stuck in memory after a reload. I'm
having a little trouble figuring out exactly what is holding onto
references. Here's what I know:
- - Tomcat 7.0.40
- - Running Sun/Oracle Java 1.6.0_26 64-bit Server JVM
- - Running on Linux 2.6.32 kernel
- - Using MySQL Connector/J 5.1.24 (important later)
- - Not running under a SecurityManager
- - JreMemoryLeakPreventionListener is enabled
- - <Context> has no "clearReferences...*" attributes set (all default
to "false" save for HttpClientKeepAlive)
I took a heap dump, and I can see that I have a WebappClassLoader
pinned in memory, "started=false", etc.
When looking at the GC roots, I can see the following references to
the WebappClassLoader:
WebappClassLoader
^- classloader of java.security.ProtectionDomain
^- [7] of java.security.ProtectionDomain[8]
^- context of java.security.AccessControlContext
^- inheritedAccessControlContext of
com.mysql.jdbc.AbandonedConnectionCleanupThread
There are a few other roots, but they are either (almost) identical to
the above or similar enough to the above that I figure they are pretty
much the same: this Connector/J thread has an indirect reference to my
WebappClassLoader for some reason.
Here's what I know about that thread:
- - It's class's ClassLoader is an o.a.c.loader.StandardClassLoader (yay!)
- - The Thread's Class's ProtectionDomain's ClasSLoader is
StandardClassLoader (yay!)
- - I'm not sure what that Thread's current context class loader is (any
idea how to find out? Browsing the object's members isn't leading me
anywhere)
- - The thread is running.
AbandonedConnectionCleanupThread has a member called
inheritedAccessControlContext which is a wrapper around
ProtectionDomain[]. Some elements in that array have a ClassLoader of
my WebappClassLoader, while others have StandardClassLoader and still
others have the JVM's application ClassLoader (the one once-removed
from boot, Chuck ;).
So, it looks like the ProtectionDomain objects themselves are what are
trapping me, here.
I'm not an expert at the internals of Java's security infrastructure,
nor have I dived into the Connector/J code (yet). I was wondering if
anyone had any guidance for me before I proceed.
(There is an updated 5.1.25 Connector/J driver that has a note in the
ChangeLog about ClassLoader references specifically-related to Tomcat.
I'll try with the latest -- which is likely to have a regression in it
because of the stupid fix to http://bugs.mysql.com/bug.php?id=68556 --
and see what the differences are).
At any rate, I'm curious as to where the ProtectionDomain objects are
actually created.
Thanks,
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJRqMRMAAoJEBzwKT+lPKRYPpcQAIxX984igfcVyfHxuwE11Dd1
XRXfMCyGX9RAQLgXWtbjV8iROpUat2AsBrYeJeZ/4UnohnJAbkaVe66NronSDX0P
ZVRcFPViz/PjXwRQOV81Jo+2clGv3jpgLwszF+QxQpLjixRYmgLSHXk/u+zgOeqH
m2KE3t/mrpuSAbboVjucepdd7Lc1MtY59wl9NVu2WgAWzTAnJ6LyUKTPGJToD5+/
H7/THkmXcoxitqYPvwLiTlZtzds7eoooc+WZpMqMDZ6Cspi12M/gkefb1MTu3Ufu
9EsLa6ZvrY2taySYS8yFQUfmM11BJzoiK9GlHRAHa/x/9zhztE+chzvpyLgdCMxg
6A4jY4RZHkXr9z8T/tvmSoT8lIE1/RY+M1l86sjQMBSUGlJZJi/WhjM2G0EV5MEr
DArBpV8fiovb6cV+aqQQ1rce3tfar1amUt9LzGfBethqzpZ64zqsTqEpXCTxxqXV
7kNZSmB009ziP6xat9kzqPmwcTV97YQFxDDLOAs7O6Z9Lj1OXBkCA/Jy1NRMJWmn
UiDbER2zc3SS2+QpRuzJt+z0LADP4/wlM3qJDbyLSWJP3QkQlDILjKB5xGXs3Z3Q
Y70qzwsBVV6svB9zyqE5AMfMP+K60CwX2qWm4NiBTskIwcmJ1TTNvrsjFoI2Dzri
dKYpPlsk3eMufOS5s16Q
=s4fy
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: WebappClassLoaders pinned in memory by java.security.ProtectionDomain
objects
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
All,
Also worth noting is that it appears only the first WebappClassLoader
is leaked: multiple subsequent webapp re-loads do not leak additional
WebappClassLoaders (which is kind of nice).
- -chris
On 5/31/13 2:10 PM, Christopher Schultz wrote:
> Chuck,
>
> On May 31, 2013, at 12:02, "Caldarale, Charles R"
> <Ch...@unisys.com> wrote:
>
>>> From: Christopher Schultz
>>> [mailto:chris@christopherschultz.net] Subject:
>>> WebappClassLoaders pinned in memory by
>>> java.security.ProtectionDomain objects
>>
>>> Using MySQL Connector/J 5.1.24 (important later)
>>
>> Where is the JDBC jar located? Is the observed behavior
>> different if it's in tomcat/lib versus the webapp's WEB-INF/lib?
>
> Duh, should've mentioned that: it's in CATALINA_HOME/lib
>
>>> I'm not sure what that Thread's current context class loader is
>>> (any idea how to find out? Browsing the object's members isn't
>>> leading me anywhere)
>>
>> Thread.getContextClassLoader() will tell you, if you can get into
>> a debugger that will evaluate it for you. Can't seem to find an
>> MBean attribute for it, unfortunately.
>
> I've been looking at an off-line heap dump. YourKit doesn't seem to
> let me see the contextClassLoader member for some reason.
>
> I'll try to attach to the running process later.
>
> -Chris
> ---------------------------------------------------------------------
>
>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJRqPpcAAoJEBzwKT+lPKRYwVAP/0ia+gUZ5SRQi6JYL6nkU0ws
yYh0t1/J36t5CsshJMfpOtuVcy72VYZKPZ58Rc6AnRSXNxzevLmgHaW1xeFIYxPf
IWftnDUhcvvjw+nFfSpuQhRoAqC7vixUFtHP2ljVi/JTFV3nXHrCn/4tuvMb7qF+
Mqvk6l3dsZ6YlMu1s1sS7P1JRw96dl/hliTjiekQpVgBKQczlWto6B32JVEtL7El
sN3BXrFkwbMl6hBkhBBYi85WXEReTHNdMC4uqVPKUrLTE1d4tMKFyQsQ3t7BysIG
QkrR0MIIfoe+5EflFBWbKEC+IunjEybydzQOVafHLqGERQ8AR5U8e05/xv+Kz/A0
UiTizXXMnlKQRl7Ue73CKhmCwmPGn6HjbY2M56bbeQtL1R1ZopcVjuv41A3gRTWn
BpBNsMA9dZ0v6qxEd5bjW4M1wuPcitUOA44mOGIwOvDFUGN/2chKRRHWjwEdo1sq
UvVjDcoBBbThr+dRIxOToEzSEZ6o3IG3LRQzF7DGy6xzruoShFld4KVcPljKJ14g
kuNa0NnDSwecQV08aIODvxcXGus43O6IT2EaWM9C/1m1F0bCjxfB3iLtsxWKiOJF
u+KaNM8e7KzNQfGsw/OjHtTZihX1XQPjlapTLONoczI6CyYxPJ839X/baf/7icre
2atH0XUb0pUVw9cJOxGe
=lsW+
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: WebappClassLoaders pinned in memory by java.security.ProtectionDomain objects
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Chuck,
On May 31, 2013, at 12:02, "Caldarale, Charles R" <Ch...@unisys.com> wrote:
>> From: Christopher Schultz [mailto:chris@christopherschultz.net]
>> Subject: WebappClassLoaders pinned in memory by java.security.ProtectionDomain objects
>
>> Using MySQL Connector/J 5.1.24 (important later)
>
> Where is the JDBC jar located? Is the observed behavior different if it's in tomcat/lib versus the webapp's WEB-INF/lib?
Duh, should've mentioned that: it's in CATALINA_HOME/lib
>> I'm not sure what that Thread's current context class loader is (any
>> idea how to find out? Browsing the object's members isn't leading me
>> anywhere)
>
> Thread.getContextClassLoader() will tell you, if you can get into a debugger that will evaluate it for you. Can't seem to find an MBean attribute for it, unfortunately.
I've been looking at an off-line heap dump. YourKit doesn't seem to let me see the contextClassLoader member for some reason.
I'll try to attach to the running process later.
-Chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: WebappClassLoaders pinned in memory by
java.security.ProtectionDomain objects
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Christopher Schultz [mailto:chris@christopherschultz.net]
> Subject: WebappClassLoaders pinned in memory by java.security.ProtectionDomain objects
> Using MySQL Connector/J 5.1.24 (important later)
Where is the JDBC jar located? Is the observed behavior different if it's in tomcat/lib versus the webapp's WEB-INF/lib?
> I'm not sure what that Thread's current context class loader is (any
> idea how to find out? Browsing the object's members isn't leading me
> anywhere)
Thread.getContextClassLoader() will tell you, if you can get into a debugger that will evaluate it for you. Can't seem to find an MBean attribute for it, unfortunately.
> still others have the JVM's application ClassLoader (the one once-removed
> from boot, Chuck ;).
That would more properly be termed the system class loader.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: WebappClassLoaders pinned in memory by java.security.ProtectionDomain
objects
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
All,
On 5/31/13 11:39 AM, Christopher Schultz wrote:
> - Using MySQL Connector/J 5.1.24 (important later) - Not running
> under a SecurityManager - <Context> has no "clearReferences...*"
> attributes set (all default to "false" save for
> HttpClientKeepAlive)
>
> [...]
>
> WebappClassLoader ^- classloader of java.security.ProtectionDomain
> ^- [7] of java.security.ProtectionDomain[8] ^- context of
> java.security.AccessControlContext ^- inheritedAccessControlContext
> of com.mysql.jdbc.AbandonedConnectionCleanupThread
It seems that the AbandonedConnectionCleanupThread itself is not to
blame: it hardly has any code in it at all, other than this
astoundingly ineffective attempt to avoid ClassLoader leakage in the
run() method:
threadRef.setContextClassLoader(null);
Connector/J 5.1.25 (one patch-version after the one I'm currently
running has removed that line, so there's now a direct link from the
thread to the WebappClassLoader. Brilliant.)
Anyhow, everything crystallized in Thread.init(), called from
Thread.<init>, so is happens even before MySQL's thread had a chance
to do anything (such as change the context ClassLoader).
The context class loader needs to be set /before/ the Thread is
created, and that happens in MySQL's JDBC driver's static initializer,
like this:
static {
AbandonedConnectionCleanupThread referenceThread = new
AbandonedConnectionCleanupThread();
referenceThread.setDaemon(true);
referenceThread.start();
}
*sigh*
I've commented on the bug... we'll see if anything else happens.
It appears that using the JreMemoryLeakPreventionListener's
classesToInitialize set to "com.mysql.jdbc.NonRegisteringDriver" seems
to do the trick: clean reloads again.
/me sighs satisfactorily
Then again, using classesToInitialize is a total hack.
Would it be reasonable to have the context ClassLoader of the thread
Tomcat is using to create the JNDI DataSource be temporarily set (if
necessary) to the ClassLoader that actually loaded the Driver itself?
That way, if the driver is loaded from WEB-INF/lib, then these kinds
of threads will get the WebappClassLoader as their context ClassLoader
(which I believe is appropriate), and those loaded from
CATALINA_BASE/lib will get the container's ClassLoader (which I
believe if appropriate, since the driver shouldn't be loading
webapp-specific classes).
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJRqQZAAAoJEBzwKT+lPKRYhIAP/1fAwuGhQoeepqkbw8ed4XvL
G4m80P6XC81er9KFZ0A8FImQ/Ua6w0t168TV7d6roqWl3voNC/Zvi+s1BIzmLmtX
Fvs66j6sGzUdYuE6Hu8B4joyG1o4e/fJRb5IA9zdLI/cbna8n5+tFhDPtly3vnuN
iJQgyVEYfdhej62lwLG/gGwIC+qUHxVgt0Z0MbRVEyKYvWX7RUWonYTrVHoH9+hG
ycf6yjUQXT3DFKno9FA8cUbDZwwpMeKXz7bK2oEdfFiAVWSFp63Y5KR6RSYYNwCG
DPY8POuwBGufzvQH6xRBb4TtRyZ1gt35bR4raOxK0Ls0YnnJYZHlzb41HxrQXMSP
lBQpnsneQhvNURGZUKw6VU2aCdg5l+OQbWhXJF/wzFz6B2M3u9ARwvY7h/iCJx0V
KECmprP3dZ717FtvVNNAs19ncr7fOVRhJqv2fRmDU+t5iklD0HROmCk00FcRUuub
kJVqvOeNjE8Tjy9lZ3MviLD/Xq7CIdKfIKghHwwdB/+Mm5kXOP2DodORBScZK1ad
s9X92mtbrNF5pMwL9eZJycjSZYxVoDTXqISNOjUWIIWpbz0piv4JReZrTHx2EFxw
bOjEKW8K5NGrIy2xRLUjmgCl70smUqmUvPrjbIbT+EjwOe1nphi68as2aj4i9hU2
S3kJo2iGjjEbkVwLjHoy
=9YG6
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org