You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by liny <in...@gmail.com> on 2012/07/13 10:09:41 UTC
[Old issue] Memory leak after stop web application in Tomcat
Hi,
It's old issue, but I still can't or find any solution so far.
After stop my web application, Tomcat told me some memory leak.
I use VisualVM to check, and yes, below caused memory leak.
Does anyone find any solutions?
Can ActiveMQ team try to solve it?
Thank you for any advice.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Old-issue-Memory-leak-after-stop-web-application-in-Tomcat-tp4653959.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: [Old issue] Memory leak after stop web application in Tomcat
Posted by liny <in...@gmail.com>.
Hi gtully:
I made a war with source code to reproduce the problem, please download it
from https://docs.google.com/open?id=0B-ZFcczN9qxYZmh5dDdoMFVCUkE My
GoogleDoc .
The Tomcat version is 6.0.33 or above, but I use Tomcat 7.0.28.
Before starting this web app, you need to setup a broker TCP address in
WEB-INF\classes\jms-spring-client.xml:
Change foo.com:61616 to your own broker.
After web app is started, you can see below main page:
http://activemq.2283324.n4.nabble.com/file/n4654028/0.png
And live threads are:
http://activemq.2283324.n4.nabble.com/file/n4654028/1.png
The reproduce steps are:
1. Click [Start getting text msg from topic], it will start listening msg
from simpleTopic topic. And some ActiveMQ threads are shown as below:
http://activemq.2283324.n4.nabble.com/file/n4654028/2.png
2. You can go to ActiveMQ web console to send fake msg into simpleTopic.
(optional) Like below:
http://activemq.2283324.n4.nabble.com/file/n4654028/3.png
3. Click [Stop getting text msg from topic] will do below methods (in stop()
of \WEB-INF\classes\com\foo\MainApplication.java):
4. Go to Tomcat manager to stop this web app. Tomcat will show some logs
like below:
After web app is stopped, these 4 ActiveMQ threads are still running:
http://activemq.2283324.n4.nabble.com/file/n4654028/4.png
Hope these can help analysis.
Let me know if any!
Thank you very much.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Old-issue-Memory-leak-after-stop-web-application-in-Tomcat-tp4653959p4654028.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: [Old issue] Memory leak after stop web application in Tomcat
Posted by Gary Tully <ga...@gmail.com>.
connection pool stop needs to be called before the static shutdown methods.
Because stop is called via a destroy-method, you must only call the
static shutdowns from your shutdown bean, and only after the
connection pool has stopped. So there should be a depends-on relation
between the beans (shutdown depends on pool). I think they will be
stopped in reverse order.
Every call to close results in a call to stop on the inactivity
monitor which tracks the inactivity monitor executor and calls
shutdown when the reference count drops.
Looking at it now, it may be that it needs to call
java.util.concurrent.ThreadPoolExecutor#shutdownNow for that case
where some tasks are executing.
On 20 July 2012 03:41, liny <in...@gmail.com> wrote:
> Thanks, Gary.
>
> I created a bean like you said:
>
> And added it to context xml:
>
>
> If I don't call static shutdown() in MainApplication#stop(), like below:
>
>
> After manual stop web app from Tomcat web manager, the memory leaks still
> are there. Logs as below:
>
> you can see the AbstractInactivityMonitor is still running.
>
> If I call static shutdown() in MainApplication#stop(), and also in
> MyDisposableBean, logs look different:
>
>
> After double call static shutdown(),
> thread [writeCheck] is gone after "java.lang.ClassNotFoundException:
> org.apache.activemq.transport.AbstractInactivityMonitor$3".
> Thread [InactivityMonitor ReadCheck] and [ActiveMQ Transport:
> tcp://foo/15.87.14.93:61616] are still there, but after few seconds, they
> are gone around 10:06:44.170 finally.
> At the end, FailoverTransport even tried to reconnect...weird.
>
> I hope there is better solution on this issue.
> Thank you again, Gary.
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Old-issue-Memory-leak-after-stop-web-application-in-Tomcat-tp4653959p4654187.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
--
http://fusesource.com
http://blog.garytully.com
Re: [Old issue] Memory leak after stop web application in Tomcat
Posted by liny <in...@gmail.com>.
Thanks, Gary.
I created a bean like you said:
And added it to context xml:
If I don't call static shutdown() in MainApplication#stop(), like below:
After manual stop web app from Tomcat web manager, the memory leaks still
are there. Logs as below:
you can see the AbstractInactivityMonitor is still running.
If I call static shutdown() in MainApplication#stop(), and also in
MyDisposableBean, logs look different:
After double call static shutdown(),
thread [writeCheck] is gone after "java.lang.ClassNotFoundException:
org.apache.activemq.transport.AbstractInactivityMonitor$3".
Thread [InactivityMonitor ReadCheck] and [ActiveMQ Transport:
tcp://foo/15.87.14.93:61616] are still there, but after few seconds, they
are gone around 10:06:44.170 finally.
At the end, FailoverTransport even tried to reconnect...weird.
I hope there is better solution on this issue.
Thank you again, Gary.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Old-issue-Memory-leak-after-stop-web-application-in-Tomcat-tp4653959p4654187.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: [Old issue] Memory leak after stop web application in Tomcat
Posted by Gary Tully <ga...@gmail.com>.
Two suggestions:
introduce a DisposableBean to call the static shutdown methods on amq
so that they don't occur too early. Calling them from the main app
stop is too early as they will stop a close from being successful.
something like:
public class Shutdown implements DisposableBean {
private static final Logger log = LoggerFactory.getLogger(Shutdown.class);
@Override
public void destroy() throws Exception {
log.info("Doing amq static shutdown");
try {
DefaultThreadPools.getDefaultTaskRunnerFactory().shutdown();
} finally {
DefaultThreadPools.shutdown();
}
}
}
and add to your context like: <bean id="shutdown" class="com.foo.Shutdown" />
Also ensure that every call to getConnection has a corresponding call
to close, even if the call to connection pool stop has this covered,
it may be that some interaction causes a close from pool.stop to clash
and fail. But in that case I would expect to see something to that
effect in the log.
On 18 July 2012 06:32, liny <in...@gmail.com> wrote:
> Hi gtully:
>
> I changed to use 5.7-SNAPSHOT and did below in com.foo.MainApplication#stop:
>
>
> But still not work for me.
> Did ActiveMQ call AbstractInactivityMonitor#stopMonitorThreads while
> Connection#close?
> Or anything missing?
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Old-issue-Memory-leak-after-stop-web-application-in-Tomcat-tp4653959p4654109.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
--
http://fusesource.com
http://blog.garytully.com
Re: [Old issue] Memory leak after stop web application in Tomcat
Posted by liny <in...@gmail.com>.
Hi gtully:
I changed to use 5.7-SNAPSHOT and did below in com.foo.MainApplication#stop:
But still not work for me.
Did ActiveMQ call AbstractInactivityMonitor#stopMonitorThreads while
Connection#close?
Or anything missing?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Old-issue-Memory-leak-after-stop-web-application-in-Tomcat-tp4653959p4654109.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: [Old issue] Memory leak after stop web application in Tomcat
Posted by Gary Tully <ga...@gmail.com>.
hmm, I gave your war a whirl with 5.7-SNAPSHOT (current trunk) and it
worked and undeployed once I added the call to stop the pool before
the calls to shutdown the executors in com.foo.MainApplication#stop
Really the correct place for the calls to shutdown is when the spring
context is destroyed, so after the call to stop the pool from the
destroy-method. In that way they cannot interfere with normal
connection close etc.
There may be some change on trunk that resolved this issue. Can you do
some validation with a 5.7-SNAPSHOT.
On 17 July 2012 12:06, liny <in...@gmail.com> wrote:
> Hi, gtully:
>
> Thanks for reply.
> The solution that is suggested at
> https://issues.apache.org/jira/browse/AMQ-3451?focusedCommentId=13416044&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13416044
> AMQ-3451 can't solve the problem.
> I did specify "destroy-method" in my WEB-INF/classes/jms-spring-client.xml
> as below:
>
>
> And after stop web application by Tomcat manager, spring did call "stop" of
> PooledConnectionFactory as below:
>
>
> And "InactivityMonitor" is still running in the background.
> These logs can be found in
> http://activemq.2283324.n4.nabble.com/file/n4654084/tomcat_log.TXT
> tomcat_log.TXT .
>
> Really appreciated.
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Old-issue-Memory-leak-after-stop-web-application-in-Tomcat-tp4653959p4654084.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
--
http://fusesource.com
http://blog.garytully.com
Re: [Old issue] Memory leak after stop web application in Tomcat
Posted by liny <in...@gmail.com>.
Hi, gtully:
Thanks for reply.
The solution that is suggested at
https://issues.apache.org/jira/browse/AMQ-3451?focusedCommentId=13416044&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13416044
AMQ-3451 can't solve the problem.
I did specify "destroy-method" in my WEB-INF/classes/jms-spring-client.xml
as below:
And after stop web application by Tomcat manager, spring did call "stop" of
PooledConnectionFactory as below:
And "InactivityMonitor" is still running in the background.
These logs can be found in
http://activemq.2283324.n4.nabble.com/file/n4654084/tomcat_log.TXT
tomcat_log.TXT .
Really appreciated.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Old-issue-Memory-leak-after-stop-web-application-in-Tomcat-tp4653959p4654084.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: [Old issue] Memory leak after stop web application in Tomcat
Posted by liny <in...@gmail.com>.
I also caught some logs, doesn't know whether it's related or not.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Old-issue-Memory-leak-after-stop-web-application-in-Tomcat-tp4653959p4654076.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: [Old issue] Memory leak after stop web application in Tomcat
Posted by Gary Tully <ga...@gmail.com>.
This is https://issues.apache.org/jira/browse/AMQ-3451
can you validate that the problem is the same with a current
5.7-SNAPSHOT and that you are calling the static shutdown methods as
outlined in that jira.
If I want to quickly reproduce, what steps should I follow?
On 13 July 2012 09:09, liny <in...@gmail.com> wrote:
> Hi,
>
> It's old issue, but I still can't or find any solution so far.
> After stop my web application, Tomcat told me some memory leak.
> I use VisualVM to check, and yes, below caused memory leak.
> Does anyone find any solutions?
> Can ActiveMQ team try to solve it?
> Thank you for any advice.
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Old-issue-Memory-leak-after-stop-web-application-in-Tomcat-tp4653959.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
--
http://fusesource.com
http://blog.garytully.com