You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Dale Ogilvie <Da...@trimble.com> on 2013/11/06 23:50:03 UTC
Fix your web application so it can cleanly un-deploy and re-deploy - how?
Chris made the following good suggestion in another thread:
"Can I make a suggestion? Fix your web application so it can cleanly un-deploy and re-deploy and then simply do a hot deployment?"
I've been down this track with our own Spring web apps and found it to be quite a deep rabbit hole where a number of 3rd party libs are used. We get the issue where the webapp classloader is not GC'ed due to classes in the libraries we use not being terminated cleanly. Which means we get a big permgen memory leak when we redeploy the app. The "occasional tomcat restart" workaround is effective, if nasty.
I did what Chris suggested for one of our apps and I think I got to 3rd party library problem number FIVE (an oracle jdbc driver connection timer) before I gave up in disgust. As I recall undisposed thread locals were a common theme. I used various strategies to resolve the prior issues in things as simple as logging frameworks, JMS queuing libraries, underlying http client code etc. Strategies such as:
1. Specifically calling a low level library finalization routine in a context listener or Spring lifecycle bean
2. Updating the 3rd party library to a later version which fixed the leak
3. Including Mattias Jiderhamn's active leak prevention library
I would so love it if Tomcat could just throw away the entire webapp memory footprint on undeploy... Tomcat 7x memory leak protection wasn't good enough for our app a few months ago.
Or failing that, if anyone can share successful strategies for "Fixing your web application so it can cleanly un-deploy and re-deploy" please do.
Dale
Ref: http://wiki.apache.org/tomcat/MemoryLeakProtection
Ref: https://github.com/mjiderhamn/classloader-leak-prevention
-----Original Message-----
From: Christopher Schultz [mailto:chris@christopherschultz.net]
Sent: Thursday, 7 November 2013 10:44 a.m.
To: Tomcat Users List
Subject: Re: how to bounce tomcat remote?
<snip>
Can I make a suggestion? Fix your web application so it can cleanly un-deploy and re-deploy and then simply do a hot deployment?
<snip>
Re: Fix your web application so it can cleanly un-deploy and re-deploy
- how?
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Dale,
On 11/6/13, 5:50 PM, Dale Ogilvie wrote:
> Chris made the following good suggestion in another thread:
>
> "Can I make a suggestion? Fix your web application so it can
> cleanly un-deploy and re-deploy and then simply do a hot
> deployment?"
>
> I've been down this track with our own Spring web apps and found it
> to be quite a deep rabbit hole where a number of 3rd party libs are
> used. We get the issue where the webapp classloader is not GC'ed
> due to classes in the libraries we use not being terminated
> cleanly. Which means we get a big permgen memory leak when we
> redeploy the app. The "occasional tomcat restart" workaround is
> effective, if nasty.
Chuck and (especially) Mark have given good replies. I have some other
thoughts.
> I did what Chris suggested for one of our apps and I think I got
> to 3rd party library problem number FIVE (an oracle jdbc driver
> connection timer) before I gave up in disgust.
As for JDBC drivers, you can force the JDBC driver to be loaded before
the webapp classloader by using the JreMemoryLeakPreventionListener.
It looks like the Tomcat documentation might even have the exact name
of the class you might want to use:
http://tomcat.apache.org/tomcat-7.0-doc/config/listeners.html#JRE_Memory_Leak_Prevention_Listener_-_org.apache.catalina.core.JreMemoryLeakPreventionListener
(See the attribute "classesToInitialize").
> As I recall undisposed thread locals were a common theme.
These are taken care-of by ... some component, I think. I can't
remember which one.
> I used various strategies to resolve the prior issues in things as
> simple as logging frameworks, JMS queuing libraries, underlying
> http client code etc. Strategies such as:
>
> 1. Specifically calling a low level library finalization routine in
> a context listener or Spring lifecycle bean
>
> 2. Updating the 3rd party library to a later version which fixed
> the leak
>
> 3. Including Mattias Jiderhamn's active leak prevention library
If you find a bug in a 3rd-party library, you can usually contact the
vendor (or, better yet, contribute a patch to an OSS library) and get
it fixed. Mark's work with Tomcat's memory-leak detection code has
prompted a bunch of fixes in both ASF- and non-ASF-related libraries.
> I would so love it if Tomcat could just throw away the entire
> webapp memory footprint on undeploy... Tomcat 7x memory leak
> protection wasn't good enough for our app a few months ago.
As Mark says, that's not really possible. I mean I suppose
theoretically, one could use JVMTI or whatever and attach to the JVM's
profiling interface, crawl every object in the heap and null-out any
reference to an object whose class was loaded by the
WebappClassLoader, but that would probably cause a few minor disasters
every time you did it (plus it would be hideously inefficient).
> Or failing that, if anyone can share successful strategies for
> "Fixing your web application so it can cleanly un-deploy and
> re-deploy" please do.
If you haven't already, enable the JreMemoryLeakPreventionListener and
make sure you pay attention to it when it complains.
Also, read Mark's slides on tracking-down the problems using a heap
profiler:
http://people.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.15 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJSfAdUAAoJEBzwKT+lPKRYhBcP/RN2eL2qmxdUATpTfQxbs8Ec
PLG59WyQcUBJiKdhmenviTXf6EIcrBCAWmVXlw3+l52y6HvbnfuOl3LlX4grdKUO
EWpwg5RJ0dtkgCBpXHBHBdqM2pJ5zcRgGhFB2bgSUT8qo8uB1F8tjbtHAfRsupqc
2RSZXpvAlVNoxQFAt8zBLLYHVzoOTM0s30iE8gAo+xWPReUSQ2TIFe6I1tZYp2on
JsSAaiYmRpVgd2x/0Baw0xqIOVHmJr5zaoDKiePivTLKZAHPHDSeDWSJjHWpyan/
l6g+Q/cMwzP7SraaKVVnz0QshEMujYK/HkesI9Kmux99qGjR0c8aWTPnrxm3RcHd
hI2PCPNe7mAZNVN9nfEnHylloPCWdtlmyHMmLrXBYeUGbtBxZnehWXGgUSiSqh6U
7/fwLPNpUUssNmt7Y95+5E8qSj+9M22mPK58oGb5+D8Ih0rohrpvWeiuF8AIa6mc
v+PF0elb6ximmxvuf9aR1wrjsTCIV1tsqlJWkEgR9aK+DYEeZ5a7IyDxP8kB15zE
pydGSk3IRx6pQqWefMeU+r+74KQMPLC4heRHOaFHYtNtQCGOds075BLjajBPpMXY
vhPxvRlg2SXbRw4Flk3nFaVYyQbddun6bYwQKlHXcBi8qjqILHR+rCxV6aDVqW99
mjjBFDjhOP2t1QDTTRg0
=vize
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Fix your web application so it can cleanly un-deploy and
re-deploy - how?
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Dale Ogilvie [mailto:Dale_Ogilvie@trimble.com]
> Subject: Fix your web application so it can cleanly un-deploy and re-deploy - how?
> As I recall undisposed thread locals were a common theme.
Have you enabled this Listener?
http://tomcat.apache.org/tomcat-7.0-doc/config/listeners.html#ThreadLocal_Leak_Prevention_Listener_-_org.apache.catalina.core.ThreadLocalLeakPreventionListener
> I would so love it if Tomcat could just throw away the entire webapp memory
> footprint on undeploy...
Last time I looked, Java didn't have a free() API...
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Fix your web application so it can cleanly un-deploy and re-deploy - how?
Posted by Dale Ogilvie <Da...@trimble.com>.
Mark Thomas [mailto:markt@apache.org] wrote:
<snip other good stuff>
>In short, if there is something the Tomcat developers can do to help then we would love to do so - just let us know.
Many thanks Mark, I'll definitely take careful notes next time I look at this and report back.
Dale
Re: Fix your web application so it can cleanly un-deploy and re-deploy
- how?
Posted by Mark Thomas <ma...@apache.org>.
On 06/11/2013 22:50, Dale Ogilvie wrote:
> Chris made the following good suggestion in another thread:
>
> "Can I make a suggestion? Fix your web application so it can cleanly
> un-deploy and re-deploy and then simply do a hot deployment?"
>
> I've been down this track with our own Spring web apps and found it
> to be quite a deep rabbit hole where a number of 3rd party libs are
> used. We get the issue where the webapp classloader is not GC'ed due
> to classes in the libraries we use not being terminated cleanly.
> Which means we get a big permgen memory leak when we redeploy the
> app. The "occasional tomcat restart" workaround is effective, if
> nasty.
>
> I did what Chris suggested for one of our apps and I think I got to
> 3rd party library problem number FIVE (an oracle jdbc driver
> connection timer) before I gave up in disgust. As I recall undisposed
> thread locals were a common theme. I used various strategies to
> resolve the prior issues in things as simple as logging frameworks,
> JMS queuing libraries, underlying http client code etc. Strategies
> such as:
>
> 1. Specifically calling a low level library finalization routine in a
> context listener or Spring lifecycle bean 2. Updating the 3rd party
> library to a later version which fixed the leak 3. Including Mattias
> Jiderhamn's active leak prevention library
>
> I would so love it if Tomcat could just throw away the entire webapp
> memory footprint on undeploy...
Dale,
Me too. Unfortunately, if references to the web app class loader are
held outside of the web app then this just isn't possible.
> Tomcat 7x memory leak protection
> wasn't good enough for our app a few months ago.
That protection is never going to be the 100% solution for every
application. What we try and do is cover the common causes and provide
enough debug information to point to the root causes of as many of the
other issues as we can. If you are able to share more details of the
leaks you found and how you fixed them there may be something that we
can do to improve the built-in protection.
> Or failing that, if anyone can share successful strategies for
> "Fixing your web application so it can cleanly un-deploy and
> re-deploy" please do.
It sounds as if you are heading in the right direction. It isn't an easy
task. One of the goals of the leak prevention listener was education.
The more folks that are aware of the issues then, hopefully, the less
prevalent they will be come. The more folks like you educate your
internal development teams and raise bugs with the third-party libraries
then the better things should get.
As an aside, I suspect that the Tomcat developers and most library
developers have a very different view on the correct use of thread
locals. Fortunately, that is one of those issues we can work-around
(albeit at the expense of renewing the thread pool).
In short, if there is something the Tomcat developers can do to help
then we would love to do so - just let us know.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Fix your web application so it can cleanly un-deploy and re-deploy - how?
Posted by Dale Ogilvie <Da...@trimble.com>.
Martin Gainty wrote:
>How did Mattias Jiderhamn's library help?
It resolved one of the memory leaks, which allowed me to progress on to the next leak :-(
Dale
RE: Fix your web application so it can cleanly un-deploy and
re-deploy - how?
Posted by Martin Gainty <mg...@hotmail.com>.
Springs over-use of CGLib for Interfaces is a memory consumer
Retask CGLib Proxy to JDKProxy to create your Impl classes for @Before advised methods
proxyTargetClass: false
Similarly using JavaAssist with Hibernate reduced memory footprint over CGLib significantly
http://docs.spring.io/spring/docs/3.0.0.M4/reference/html/ch08s05.html
http://what-when-how.com/Tutorial/SpringFramework3/SpringFramework300224.html
Dale: How did Mattias Jiderhamn's library help?
Martin
> Subject: Fix your web application so it can cleanly un-deploy and re-deploy - how?
> Date: Thu, 7 Nov 2013 11:50:03 +1300
> From: Dale_Ogilvie@trimble.com
> To: users@tomcat.apache.org
>
> Chris made the following good suggestion in another thread:
>
> "Can I make a suggestion? Fix your web application so it can cleanly un-deploy and re-deploy and then simply do a hot deployment?"
>
> I've been down this track with our own Spring web apps and found it to be quite a deep rabbit hole where a number of 3rd party libs are used. We get the issue where the webapp classloader is not GC'ed due to classes in the libraries we use not being terminated cleanly. Which means we get a big permgen memory leak when we redeploy the app. The "occasional tomcat restart" workaround is effective, if nasty.
>
> I did what Chris suggested for one of our apps and I think I got to 3rd party library problem number FIVE (an oracle jdbc driver connection timer) before I gave up in disgust. As I recall undisposed thread locals were a common theme. I used various strategies to resolve the prior issues in things as simple as logging frameworks, JMS queuing libraries, underlying http client code etc. Strategies such as:
>
> 1. Specifically calling a low level library finalization routine in a context listener or Spring lifecycle bean
> 2. Updating the 3rd party library to a later version which fixed the leak
> 3. Including Mattias Jiderhamn's active leak prevention library
>
> I would so love it if Tomcat could just throw away the entire webapp memory footprint on undeploy... Tomcat 7x memory leak protection wasn't good enough for our app a few months ago.
>
> Or failing that, if anyone can share successful strategies for "Fixing your web application so it can cleanly un-deploy and re-deploy" please do.
>
> Dale
>
> Ref: http://wiki.apache.org/tomcat/MemoryLeakProtection
> Ref: https://github.com/mjiderhamn/classloader-leak-prevention
>
> -----Original Message-----
> From: Christopher Schultz [mailto:chris@christopherschultz.net]
> Sent: Thursday, 7 November 2013 10:44 a.m.
> To: Tomcat Users List
> Subject: Re: how to bounce tomcat remote?
>
> <snip>
> Can I make a suggestion? Fix your web application so it can cleanly un-deploy and re-deploy and then simply do a hot deployment?
> <snip>
>
> B�KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKCB�� [��X��ܚX�KK[XZ[� \�\��][��X��ܚX�P �X�] �\X�K�ܙ�B��܈Y][ۘ[��[X[� �K[XZ[� \�\��Z[ �X�] �\X�K�ܙ�B�