Consumers not consuming though Queue has items- ActiveMQ 5.5.1 and Spring DMLC/JMStemplate

[Preetam <ra...@gmail.com>]

Hi,

We use Spring DMLC to consume from one or more Queues. Async send and auto
acknowledge are used. KahaDB is used for persistence. StoreUsage is set to
1g. But when this happens, the size is only around 200mb. So does not appear
to be a flow control induced deadlock. There are multiple DMLC running on
the same host. Each listener, consumes a message and depending the type of
message, may again push few more messages to the queue. The input rate is
around 200 rps, and each consumed message pushing 10 or more messages, the
load on broker may be 2000 rps. The consumer pushes the messages back to
broker using JMSTemplate. Both DMLC and JMSTemplate share the same
PooledConnectionFactory. Did not see any exception on either the broker or
consumer logs. Spring version is 2.5.6. ActiveMQ version is 5.5.1. Both are
running on rhel 5.

Here are the consumer thread dumps:

Of the 1173 threads 1013 are in this state:

Thread 31315: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be
imprecise)
 - java.lang.Object.wait() @bci=2, line=485 (Compiled frame)
 - org.apache.activemq.SimplePriorityMessageDispatchChannel.dequeue(long)
@bci=46, line=87 (Compiled frame)
 - org.apache.activemq.ActiveMQMessageConsumer.dequeue(long) @bci=19,
line=452 (Compiled frame)
 - org.apache.activemq.ActiveMQMessageConsumer.receive(long) @bci=44,
line=575 (Compiled frame)
 -
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(javax.jms.MessageConsumer)
@bci=23, line=405 (Compiled frame)
 -
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(java.lang.Object,
javax.jms.Session, javax.jms.MessageConsumer,
org.springframework.transaction.TransactionStatus) @bci=122, line=308
(Compiled frame)
 -
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(java.lang.Object,
javax.jms.Session, javax.jms.MessageConsumer) @bci=97, line=261 (Interpreted
frame)
 -
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener()
@bci=17, line=982 (Interpreted frame)
 -
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop()
@bci=130, line=974 (Interpreted frame)
 -
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run()
@bci=51, line=876 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)

Thread dump from broker:

There are around 40 pairs of these:

Thread 9762: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be
imprecise)
 - java.lang.Object.wait() @bci=2, line=485 (Compiled frame)
 - org.apache.activemq.thread.DedicatedTaskRunner.runTask() @bci=134,
line=111 (Compiled frame)
 - org.apache.activemq.thread.DedicatedTaskRunner$1.run() @bci=11, line=42
(Interpreted frame)


Thread 9760: (state = IN_NATIVE)
 - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[],
int, int, int) @bci=0 (Compiled frame; information may be imprecise)
 - java.net.SocketInputStream.read(byte[], int, int) @bci=84, line=129
(Compiled frame)
 - org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill() @bci=31,
line=50 (Compiled frame)
 - org.apache.activemq.transport.tcp.TcpTransport$2.fill() @bci=9, line=576
(Compiled frame)
 - org.apache.activemq.transport.tcp.TcpBufferedInputStream.read() @bci=12,
line=58 (Compiled frame)
 - org.apache.activemq.transport.tcp.TcpTransport$2.read() @bci=9, line=561
(Compiled frame)
 - java.io.DataInputStream.readInt() @bci=4, line=370 (Compiled frame)
 - org.apache.activemq.openwire.OpenWireFormat.unmarshal(java.io.DataInput)
@bci=10, line=269 (Compiled frame)
 - org.apache.activemq.transport.tcp.TcpTransport.readCommand() @bci=8,
line=227 (Compiled frame)
 - org.apache.activemq.transport.tcp.TcpTransport.doRun() @bci=1, line=219
(Compiled frame)
 - org.apache.activemq.transport.tcp.TcpTransport.run() @bci=47, line=202
(Compiled frame)
 - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
 
Then this:

Thread 21583: (state = IN_JAVA)
 - java.util.AbstractList$Itr.next() @bci=12, line=345 (Compiled frame;
information may be imprecise)
 - org.apache.activemq.broker.region.Queue.doActualDispatch(java.util.List)
@bci=171, line=1822 (Compiled frame)
 - org.apache.activemq.broker.region.Queue.doDispatch(java.util.List)
@bci=54, line=1772 (Compiled frame)
 - org.apache.activemq.broker.region.Queue.pageInMessages(boolean) @bci=6,
line=1932 (Compiled frame)
 - org.apache.activemq.broker.region.Queue.iterate() @bci=502, line=1440
(Compiled frame)
 - org.apache.activemq.thread.DedicatedTaskRunner.runTask() @bci=67,
line=104 (Compiled frame)
 - org.apache.activemq.thread.DedicatedTaskRunner$1.run() @bci=11, line=42
(Interpreted frame)

Have attached the thread dumps for both consumer and broker. Does not appear
to be any deadlocks  looking at thread dumps. But I may be wrong. 

Any pointers would be of great help.

Thanks
Preetam

broker_threads.txt
<http://activemq.2283324.n4.nabble.com/file/n4657796/broker_threads.txt>  
consumer_threads.txt
<http://activemq.2283324.n4.nabble.com/file/n4657796/consumer_threads.txt>  



--
View this message in context: http://activemq.2283324.n4.nabble.com/Consumers-not-consuming-though-Queue-has-items-ActiveMQ-5-5-1-and-Spring-DMLC-JMStemplate-tp4657796.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.