You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by khandelwalanuj <kh...@gmail.com> on 2013/12/26 10:08:32 UTC
Hung ActiveMQ broker and processes are blocking
JStack.txt <http://activemq.2283324.n4.nabble.com/file/n4675874/JStack.txt>
Hi,
I am using ActiveMQv5.8. Yesterday I saw some weird issue happening with
broker where:
ActiveMQ broker running on cluster machine was in hung state and not
responding to the client request. It was blocking processes. Also there are
lots of connection from a single client (more than 100)
I took jstack on the hung process. Attaching the same with this request.
Many of the blocked threads indicate that they were doing some inactivity
monitoring related processing.
I am seeing lot of warning in my broker log related to Inactivity
monitoring:
""
20131214 01:14:37:740 IST (ActiveMQ InactivityMonitor Worker)
org.apache.activemq.broker.TransportConnection#serviceTransportException 238
WARN - Transport Connection to: tcp://...:1373 failed:
org.apache.activemq.transport.InactivityIOException: Channel was inactive
for too (>30000) long: tcp://...:21373
""
My Observations:
I suspect that the ActiveMQ broker is not handling momentary cpu spikes very
well – it goes into a state where messages get piled up and
expensive protocols like closing/reopening all connections are triggered due
to heartbeat timeouts from clients. The current heartbeat/inactivity timeout
is 30 seconds which may be too low when handing more than 250 clients.
Please help me to understand and handle this issue because it is affecting
the critical production environment.
I don't know how to handle this and what to do but I think I should provide
Inactivity timeout period is 120 seconds which is enough time to do the read
check. So Are there any side effects of this Inactivity monitoring time
increase ?
Thanks,
Anuj
--
View this message in context: http://activemq.2283324.n4.nabble.com/Hung-ActiveMQ-broker-and-processes-are-blocking-tp4675874.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Hung ActiveMQ broker and processes are blocking
Posted by xabhi <xa...@gmail.com>.
I looked at stack trace and it has something to do with inactivity monitor.
Some one from ActiveMQ development team can help us here.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Hung-ActiveMQ-broker-and-processes-are-blocking-tp4675874p4675967.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Hung ActiveMQ broker and processes are blocking
Posted by khandelwalanuj <kh...@gmail.com>.
Please respond...
--
View this message in context: http://activemq.2283324.n4.nabble.com/Hung-ActiveMQ-broker-and-processes-are-blocking-tp4675874p4675928.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Hung ActiveMQ broker and processes are blocking
Posted by artnaseef <ar...@artnaseef.com>.
According to this page,
http://stackoverflow.com/questions/3334911/what-does-jvm-flag-cmsclassunloadingenabled-actually-do,
the following is also needed for permgen sweeping to work:
-XX+UseConcMarkSweepGC
We are using that one as well.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Hung-ActiveMQ-broker-and-processes-are-blocking-tp4675874p4676080.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Hung ActiveMQ broker and processes are blocking
Posted by artnaseef <ar...@artnaseef.com>.
I need to research the "used heap size" to understand what are the right
numbers that way. Using jconsole or visualvm, the total heap size is
displayed and graphed in an easy-to-use format, and the total never
exceeds the max (when it hits, GC kicks in).
BTW, another great tool for looking at JVM stats from the command-line is
jstat:
jstat -gccapacity <pid>
Here's a page that lists all of the output formats:
http://docs.oracle.com/javase/1.5.0/docs/tooldocs/share/jstat.html.
Here is the option that enables GC for permgen:
-XX:+CMSPermGenSweepingEnabled
We also use this one:
-XX:+CMSClassUnloadingEnabled
The JVM puts out this warning with the former, so these may be equivalent:
Please use CMSClassUnloadingEnabled in place of
CMSPermGenSweepingEnabled in the future
Note that, if your JVM is running out of Heap, and not permgen, then it
will be necessary to determine what's causing the broker JVM to run out of
memory. One common cause is slow consumers without PFC, or using a
configuration for which PFC does not work.
A good heap dump and analysis tool are the best bet in that case.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Hung-ActiveMQ-broker-and-processes-are-blocking-tp4675874p4676079.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Hung ActiveMQ broker and processes are blocking
Posted by khandelwalanuj <kh...@gmail.com>.
One more thing.
While my broker was hung, my total used heap is exceeded max heap size. I
don;t understand why ?
I calculated heap using "JMAP -heap PID" command.
where:
used heap size = used (PS Old Generation) + used(Eden Space)
how can this value be more than maxHeapSpace ?
Thanks,
Anuj
--
View this message in context: http://activemq.2283324.n4.nabble.com/Hung-ActiveMQ-broker-and-processes-are-blocking-tp4675874p4676070.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Hung ActiveMQ broker and processes are blocking
Posted by artnaseef <ar...@artnaseef.com>.
The broker runs inside a JVM. If the JVM runs out of memory, the broker is
impacted.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Hung-ActiveMQ-broker-and-processes-are-blocking-tp4675874p4676081.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Hung ActiveMQ broker and processes are blocking
Posted by Rodrigo Ramos <cr...@gmail.com>.
GC is JVM option, so, It should not affect the broker.
2014/1/7 khandelwalanuj <kh...@gmail.com>
> I checked the usage and also increased the heap size for my broker.
>
> But will you please let me know about "enabling permgen space garbage
> collection". How to do it and are there any side effects after this ?
>
> Thanks,
> Anuj
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Hung-ActiveMQ-broker-and-processes-are-blocking-tp4675874p4676069.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Hung ActiveMQ broker and processes are blocking
Posted by khandelwalanuj <kh...@gmail.com>.
I checked the usage and also increased the heap size for my broker.
But will you please let me know about "enabling permgen space garbage
collection". How to do it and are there any side effects after this ?
Thanks,
Anuj
--
View this message in context: http://activemq.2283324.n4.nabble.com/Hung-ActiveMQ-broker-and-processes-are-blocking-tp4675874p4676069.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Hung ActiveMQ broker and processes are blocking
Posted by artnaseef <ar...@artnaseef.com>.
Have you checked the memory use of the process? Use a JMX client (such as
jconsole or visualvm) to watch the memory use on the broker until it
freezes. Watch permgen space and heap space especially.
High connection counts point to a possible permgen space issue.
Most commonly the entire broker only freezes when the JVM is out of memory.
If the permgen is running out of space, there may be clients creating and
closing connections rapidly - a very bad practice for JMS. Anyway, besides
fixing the clients, enabling permgen space garbage collection may work for
you (it did for me).
--
View this message in context: http://activemq.2283324.n4.nabble.com/Hung-ActiveMQ-broker-and-processes-are-blocking-tp4675874p4676011.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.