ActiveMQ 5.4.2 ActiveMQSession.createConsumer() blocks/does not receive a response

[H C <am...@gmail.com>]

Hello,

I am working on a client/server product where the server component has an
embedded ActiveMQ broker. After upgrading the ActiveMQ from 5.3.1 to 5.4.2
we always hit a server side problem where the call to create a consumer gets
stuck in waiting for a response:

"MonitorServer-RoundRobinMessageHandlerThread-1" prio=10
tid=0x00007fd0d8696800 nid=0x7b29 waiting on condition [0x00007fcecd0cf000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00007fcfe489edb0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
    at
org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:40)
    at
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:87)
    at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1276)
    at
org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1874)
    at
org.apache.activemq.ActiveMQMessageConsumer.<init>(ActiveMQMessageConsumer.java:254)
    at
org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1116)
    at
org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1060)
    at
org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:973)
    at
org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:946)
    at
com.mycompany.monitor.server.MonitorServer.handleMessagesFromAllConsumers(MonitorServer.java:167)
    at
com.mycompany.monitor.server.MonitorServer.receiveMessages(MonitorServer.java:122)
    at
com.mycompany.monitor.server.MonitorServer.access$000(MonitorServer.java:31)
    at
com.mycompany.monitor.server.MonitorServer$1.run(MonitorServer.java:84)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:619)

To give you some background on how things are organized: the clients create
queues and send messages to them. The server listens for created
destinations it creates and caches a consumer once a new destination is
found and then tries to consume the messages. The above thread dump shows
the code getting stuck in the createConsumer() part. The problem outlined
here is only seen in production, so we have limited time to
debug/investigate. All our attempts to replicate the problem on a
"non-production" machine have failed so far (we tried simulating a large
number of clients, messages per seconds, using client/server disconnects
etc).

In terms of environment, the clients run on separate machines that are
either Fedora 13 or Windows, while the server is a Fedora 13, running Sun
JDK 1.6.0_20.

Does anyone know what could be the problem and how to find the root cause?

Thank you!

Best Regards,

Horace