You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Gary Weaver <ga...@gmail.com> on 2011/05/10 21:55:53 UTC
Suggestions for really getting to the bottom of memory leak issues on hot-deploys
Here's what I did and what little I came up with:
http://stufftohelpyouout.blogspot.com/2011/05/diagnosing-webappportlet-hot-deploy.html
I'm definitely not an expert at diagnosing leaks, so if you have any
recommendations/comments, please let me know.
I unfortunately started off just thinking I could tune JVM settings, not
just by bumping up permgen but by adding things like:
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSPermGenSweepingEnabled
-XX:+CMSClassUnloadingEnabled
But, I was (really) wrong as you can see in the post, along with the various
tools used to try to identify what was causing the issue(s).
It doesn't seem like determining exactly what is leaking memory (at least in
the case of permgen and hot-redeploys/hot-deploys) is a whole lot easier
than it was years ago, unless I'm doing something wrong, which I most likely
am. Thanks in advance for any feedback!
Gary
Re: Suggestions for really getting to the bottom of memory leak issues on hot-deploys
Posted by Sylvain Laurent <sy...@m4x.org>.
[from your blog entry]
> I also think that logging leaks may be more helpful than using part of manager, because many don't deploy manager (they remove it).
it is logged when the application is stopped, on recent tomcat 6 and 7.
I think your approach to finding leaks by performing comparisons is not the most efficient. What I usually do to find the leak is to use MAT (eclipse memory analyzer, as you used too) and its "class loader explorer". There I find the instance of org.apache.catalina.loader.WebappClassLoader that corresponds to my webapp and right-click to ask "path to GC roots"/"exclude all phantom/weak/soft, etc. references".
There I usually see the cause of the leak vey quickly.
I you find no strong reference to your classloaders then you might have encountered some JDK bug... You should use a recent one to avoid some (at least 6.0u21).
Sylvain
On 10 mai 2011, at 21:55, Gary Weaver wrote:
> Here's what I did and what little I came up with:
> http://stufftohelpyouout.blogspot.com/2011/05/diagnosing-webappportlet-hot-deploy.html
>
> I'm definitely not an expert at diagnosing leaks, so if you have any
> recommendations/comments, please let me know.
>
> I unfortunately started off just thinking I could tune JVM settings, not
> just by bumping up permgen but by adding things like:
>
> -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC
> -XX:+CMSPermGenSweepingEnabled
> -XX:+CMSClassUnloadingEnabled
>
> But, I was (really) wrong as you can see in the post, along with the various
> tools used to try to identify what was causing the issue(s).
>
> It doesn't seem like determining exactly what is leaking memory (at least in
> the case of permgen and hot-redeploys/hot-deploys) is a whole lot easier
> than it was years ago, unless I'm doing something wrong, which I most likely
> am. Thanks in advance for any feedback!
>
> Gary
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Suggestions for really getting to the bottom of memory leak issues
on hot-deploys
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Gary,
On 5/10/2011 4:33 PM, Gary Weaver wrote:
> Chris,
>
> On Tue, May 10, 2011 at 4:06 PM, Christopher Schultz <
> chris@christopherschultz.net> wrote:
>
>>
>> http://people.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf
>
> Thanks! That is a great presentation!
It really is. Having a decent memory profiler can really help things out.
If you can, try upgrading to the latest version of either Tomcat 6 or
Tomcat 7... there are some speculative memory leak prevention techniques
that they use to try to eliminate some memory leaks that webapps cannot
hope to eliminate by themselves -- things like JVM-provided classes that
store references to the context class loader.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk3KmBQACgkQ9CaO5/Lv0PBRdQCeMAdx6jA+RasVei8e1cprTXoy
mWEAoIlg5+VY+ZXQptXkdVsh8OV8gGdo
=8hjI
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Suggestions for really getting to the bottom of memory leak
issues on hot-deploys
Posted by Gary Weaver <ga...@gmail.com>.
Chris,
On Tue, May 10, 2011 at 4:06 PM, Christopher Schultz <
chris@christopherschultz.net> wrote:
>
> http://people.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf
>
Thanks! That is a great presentation!
Gary
Re: Suggestions for really getting to the bottom of memory leak issues
on hot-deploys
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Gary,
On 5/10/2011 3:55 PM, Gary Weaver wrote:
> http://stufftohelpyouout.blogspot.com/2011/05/diagnosing-webappportlet-hot-deploy.html
Generally speaking, when you have a permgen "leak" across webapp
redeployments, this is the situation:
1. Webapp invokes some code that stores a reference to the webapp's
WebappClassLoader
2. That reference is not cleared during undeploy [this is the leak]
The big problem is that the WebappClassLoader keeps a reference to all
java.lang.Class objects it loads, so it can be quite bulky when it
leaks, depending on exactly how many classes your webapp has locally.
markt has a great presentation which lays out the anatomy of these types
of memory leaks, how to diagnose them, and often how to fix them. These
are slides from a presentation at last year's ApacheCon NA. It's /very/
readable.
http://people.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk3JmrQACgkQ9CaO5/Lv0PAwWwCfQNuuTZnw9JYqgVSdQ/daVABG
6jQAnRb1VJScgOjH8J73EKaWZLukKcqf
=Bvw2
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Suggestions for really getting to the bottom of memory leak issues
on hot-deploys
Posted by Pid <pi...@pidster.com>.
On 5/10/11 8:55 PM, Gary Weaver wrote:
> Here's what I did and what little I came up with:
> http://stufftohelpyouout.blogspot.com/2011/05/diagnosing-webappportlet-hot-deploy.html
>
> I'm definitely not an expert at diagnosing leaks, so if you have any
> recommendations/comments, please let me know.
>
> I unfortunately started off just thinking I could tune JVM settings, not
> just by bumping up permgen but by adding things like:
>
> -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC
> -XX:+CMSPermGenSweepingEnabled
> -XX:+CMSClassUnloadingEnabled
>
> But, I was (really) wrong as you can see in the post, along with the various
> tools used to try to identify what was causing the issue(s).
>
> It doesn't seem like determining exactly what is leaking memory (at least in
> the case of permgen and hot-redeploys/hot-deploys) is a whole lot easier
> than it was years ago, unless I'm doing something wrong, which I most likely
> am. Thanks in advance for any feedback!
>
> Gary
>
Did you examine the Tomcat 7 logs during redeploy cycles?
VisualVM can also take and/or open a heap dump.
p