You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Dale Ogilvie <Da...@trimble.com> on 2012/03/21 01:40:51 UTC
Tomcat Classloader leaks
In our webapps running on Tomcat 7 we have seen quite a number of
classloader memory leaks. The end result is that after a number of
application reloads and redeploys, we run out of permanent generation
memory and Tomcat stops responding. We can see the evidence of this sort
of issue thanks to the new memory leak reporting in TC6+
Generally we can trace this back to service semantics library code,
ActiveMQ, Quartz, etc. I spent a fair bit of time hunting out these sort
of issues, squashing some of them by explicitly calling specific
finalization routines in each library, on context closure.
My question is, why is it that we have to be bothered by this
show-stopper memory leak? When the application is unloaded by Tomcat, it
should go away completely, regardless of (for example) unstopped threads
in library foo.
My semi-rant is inspired by the following recent comment by Mladen Turk.
"Entire IIS worker process recycling is meant for handling crappy .NET
applications with memory and resource leaks."
So, why can't Tomcat completely clean up our crappy java apps ;-) when a
reload is requested?
Thanks
Dale
RE: Tomcat Classloader leaks
Posted by Dale Ogilvie <Da...@trimble.com>.
Mark Thomas wrote:
>Your rant would be better aimed at the developers of the third party
libraries and JVMs that create the problems in the first place rather
than at the community that has worked hard to:
>- prevent them causing an issue in the first place [1], [2]
>- cleans up the mess they leave behind where it can [3]
>- is trying to educate folks on what the problems are [4]
Thanks for that info Mark, that looks like a great summary of the
problem and what Tomcat is doing to help. Some reading for me later.
Dale
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Classloader leaks
Posted by Mark Thomas <ma...@apache.org>.
On 21/03/2012 00:40, Dale Ogilvie wrote:
> In our webapps running on Tomcat 7 we have seen quite a number of
> classloader memory leaks. The end result is that after a number of
> application reloads and redeploys, we run out of permanent generation
> memory and Tomcat stops responding. We can see the evidence of this sort
> of issue thanks to the new memory leak reporting in TC6+
Glad to see that the memory leak reporting is doing its job.
> Generally we can trace this back to service semantics library code,
> ActiveMQ, Quartz, etc. I spent a fair bit of time hunting out these sort
> of issues, squashing some of them by explicitly calling specific
> finalization routines in each library, on context closure.
>
> My question is, why is it that we have to be bothered by this
> show-stopper memory leak? When the application is unloaded by Tomcat, it
> should go away completely, regardless of (for example) unstopped threads
> in library foo.
Because there is no safe API available to stop a running thread.
> My semi-rant is inspired by the following recent comment by Mladen Turk.
Your rant would be better aimed at the developers of the third party
libraries and JVMs that create the problems in the first place rather
than at the community that has worked hard to:
- prevent them causing an issue in the first place [1], [2]
- cleans up the mess they leave behind where it can [3]
- is trying to educate folks on what the problems are [4]
> So, why can't Tomcat completely clean up our crappy java apps ;-) when a
> reload is requested?
It does what it can based on the examples we have seen so far. I am sure
there is scope for improvement. Patches welcome - or at least a simple
test case that demonstrates the issue.
Mark
[1]
http://svn.apache.org/viewvc/tomcat/trunk/java/org/apache/catalina/core/JreMemoryLeakPreventionListener.java?view=log
[2]
http://svn.apache.org/viewvc/tomcat/trunk/java/org/apache/catalina/core/ThreadLocalLeakPreventionListener.java?view=log
[3]
http://svn.apache.org/viewvc/tomcat/trunk/java/org/apache/catalina/loader/WebappClassLoader.java?view=annotate
(start at line 1984)
[4]
http://people.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Classloader leaks
Posted by Mark Thomas <ma...@apache.org>.
On 21/03/2012 08:25, Leon Rosenberg wrote:
> On Wed, Mar 21, 2012 at 7:52 AM, Pid * <pi...@pidster.com> wrote:
>> On 21 Mar 2012, at 02:41, Dale Ogilvie <Da...@trimble.com> wrote:
>
>>
>>> When the application is unloaded by Tomcat, it
>>> should go away completely, regardless of (for example) unstopped threads
>>> in library foo.
>>
>> There is an aggressive thread killer option but it is unreliable,
>> because it is hard to stop a Java thread from outside the thread.
>
> How about having an explicit executor pool per webapp? That would
> allow to throw this pool away on reload, and prevent pollution of
> general thread pool by one-webapp's thread locals.
That would be overly complex to implement given how Tomcat's internals
are structured.
There is already a solution available the achieves the same result:
http://svn.apache.org/viewvc/tomcat/trunk/java/org/apache/catalina/core/ThreadLocalLeakPreventionListener.java?view=log
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Classloader leaks
Posted by Leon Rosenberg <ro...@gmail.com>.
On Wed, Mar 21, 2012 at 7:52 AM, Pid * <pi...@pidster.com> wrote:
> On 21 Mar 2012, at 02:41, Dale Ogilvie <Da...@trimble.com> wrote:
>
>> When the application is unloaded by Tomcat, it
>> should go away completely, regardless of (for example) unstopped threads
>> in library foo.
>
> There is an aggressive thread killer option but it is unreliable,
> because it is hard to stop a Java thread from outside the thread.
How about having an explicit executor pool per webapp? That would
allow to throw this pool away on reload, and prevent pollution of
general thread pool by one-webapp's thread locals.
Leon
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Classloader leaks
Posted by Pid * <pi...@pidster.com>.
On 21 Mar 2012, at 02:41, Dale Ogilvie <Da...@trimble.com> wrote:
> In our webapps running on Tomcat 7 we have seen quite a number of
> classloader memory leaks. The end result is that after a number of
> application reloads and redeploys, we run out of permanent generation
> memory and Tomcat stops responding. We can see the evidence of this sort
> of issue thanks to the new memory leak reporting in TC6+
>
>
>
> Generally we can trace this back to service semantics library code,
> ActiveMQ, Quartz, etc. I spent a fair bit of time hunting out these sort
> of issues, squashing some of them by explicitly calling specific
> finalization routines in each library, on context closure.
>
>
>
> My question is, why is it that we have to be bothered by this
> show-stopper memory leak?
You can turn the error notices and prevention functions off if you prefer?
Otherwise, you're bothered because of the reason you describe above -
your app causes a permgen OOM after N reloads.
> When the application is unloaded by Tomcat, it
> should go away completely, regardless of (for example) unstopped threads
> in library foo.
There is an aggressive thread killer option but it is unreliable,
because it is hard to stop a Java thread from outside the thread.
> My semi-rant is inspired by the following recent comment by Mladen Turk.
>
> "Entire IIS worker process recycling is meant for handling crappy .NET
> applications with memory and resource leaks."
>
> So, why can't Tomcat completely clean up our crappy java apps ;-) when a
> reload is requested?
That is what Tomcat is trying to do.
It unloads DB drivers that you haven't cleaned up, it prevents other
JVM leaks, it recycles threads in the connector pool to prevent
ThreadLocal leaks.
If you can find a way to make it go more, patches are always welcome.
Which reminds me...
p
> Thanks
>
>
>
> Dale
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org