You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by zyc1984 <yi...@murex.com> on 2018/02/07 13:47:51 UTC
Deadlock in ActiveMQ server between BrokerFilter and Producer
creation with VM connection
Hi,
I'm not if this topic should be posted in User forum or Dev one, as this
issue is purely technical, i post it here.
We have identified a potential "deadlock" between BrokerFilter and Producer
creation with VM connection
This block is random, it happened on one given machine but not the others.
ActiveMQ version 5.9.0
Java version 1.8.0_111
Thread #310 : notified by BrokerFilter.addConsumer() and it creates a
Producer with local VM connection
Thread #308 : notifed by BrokerFilter.messageConsumed() and it tries to get
Consumer info in our application with read lock
Thread 380 : it is treating the Producer creation command but it's blocked
by Thread #308 a lock inside held by ActiveMQ BrokerFilter stack.
As this issue is not always happening, i guess it is related to the
Threading model in ActiveMQ.
And I would like to know in which case VM connection handling thread will be
blocked by BrokerFilter and in which case it is not. Is it related to jvm
arguments ? because the configuration of ActiveMQ is hard coded in our case.
Thanks,
zyc1984
"ActiveMQ Transport: tcp:///22.0.195.149:64315@64078" #310 daemon prio=5
os_prio=64 tid=0x000000010274e000 nid=0x14d waiting on condition
[0xfffffffe76ffd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xfffffffe8335fb60> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at
org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:40)
at
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:87)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1394)
at
org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1925)
at
org.apache.activemq.ActiveMQMessageProducer.<init>(ActiveMQMessageProducer.java:125)
at
org.apache.activemq.ActiveMQSession.createProducer(ActiveMQSession.java:969)
at EventFeeder.start(GenericEventFeeder.java:63)
at DefaultConsumerManager.startFeed(DefaultConsumerManager.java:192)
at
DefaultConsumerAdministration.consumerConnected(DefaultConsumerAdministration.java:249)
at ConsumerListener.onConsumerConnect(ConsumerListener.java:31)
at ActiveMQInterceptor.addConsumer(ActiveMQInterceptor.java:80)
at
org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:102)
at
org.apache.activemq.broker.TransportConnection.processAddConsumer(TransportConnection.java:587)
at org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:347)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:149)
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
at
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
at java.lang.Thread.run(Thread.java:745)
"ActiveMQ Transport: tcp:///22.0.195.149:64313@64078" #308 daemon prio=5
os_prio=64 tid=0x00000001028d5000 nid=0x14b waiting on condition
[0xfffffffe773fd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xfffffffe81003250> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at
DefaultConsumerAdministration.consumed(DefaultConsumerAdministration.java:367)
at ConsumerListener.onMessageConsumed(ConsumerListener.java:45)
at ActiveMQInterceptor.messageConsumed(ActiveMQInterceptor.java:123)
at
org.apache.activemq.broker.MutableBrokerFilter.messageConsumed(MutableBrokerFilter.java:350)
at
org.apache.activemq.broker.region.BaseDestination.messageConsumed(BaseDestination.java:465)
at org.apache.activemq.broker.region.Queue.acknowledge(Queue.java:917)
at org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1703)
at
org.apache.activemq.broker.region.QueueSubscription.acknowledge(QueueSubscription.java:63)
at
org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:235)
- locked <0xfffffffe833683f0> (a java.lang.Object)
at
org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:412)
at
org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:458)
at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:82)
at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:82)
at
org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:277)
at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:82)
at
org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:92)
at
org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:476)
at org.apache.activemq.command.MessageAck.visit(MessageAck.java:236)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:149)
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
at
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"ActiveMQ VMTransport: vm://BOS_JMS_BROKER#1-3" #380 daemon prio=5
os_prio=64 tid=0x0000000102bad000 nid=0x18a waiting for monitor entry
[0xfffffffe763fd000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:654)
- waiting to lock <0xfffffffe833683f0> (a java.lang.Object)
- locked <0xfffffffe83368400> (a java.lang.Object)
at
org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:161)
at
org.apache.activemq.broker.region.Queue.doActualDispatch(Queue.java:1998)
at org.apache.activemq.broker.region.Queue.doDispatch(Queue.java:1946)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2086)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1581)
- locked <0xfffffffe83368658> (a java.lang.Object)
at org.apache.activemq.broker.region.Queue.wakeup(Queue.java:1803)
at org.apache.activemq.broker.region.Queue.messageSent(Queue.java:1797)
at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:866)
at org.apache.activemq.broker.region.Queue.send(Queue.java:700)
at
org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:394)
at
org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:442)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:147)
at
org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
at
org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:307)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:147)
at
org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:152)
at
org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:467)
at
org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:751)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:149)
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at
org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:247)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:129)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0xfffffffe833687b8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
- <0xfffffffe83407310> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-f2368404.html