You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Timothy Bish (JIRA)" <ji...@apache.org> on 2016/03/08 19:47:41 UTC
[jira] [Updated] (AMQ-5785) Deadlock between
FilePendingMessageCursor usage change and incoming send operations.
[ https://issues.apache.org/jira/browse/AMQ-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Timothy Bish updated AMQ-5785:
------------------------------
Summary: Deadlock between FilePendingMessageCursor usage change and incoming send operations. (was: Deadlock between NIO worker and Broker.Servic threads)
> Deadlock between FilePendingMessageCursor usage change and incoming send operations.
> ------------------------------------------------------------------------------------
>
> Key: AMQ-5785
> URL: https://issues.apache.org/jira/browse/AMQ-5785
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.10.0
> Environment: Physical Machine (192 GB RAM, 24 VCPU), RHEL 5.9, Java 1.7
> ActiveMQ runs on 4 GB heap
> Reporter: Sree Panchajanyam D
> Assignee: Timothy Bish
> Priority: Critical
> Attachments: threaddump16214.txt
>
>
> During the peak loads we are encountering a recurring deadlock issue in ActiveMQ broker. The threads that are deadlocked are
> ActiveMQ NIO Worker - trying to add message to FilePendingCursor
> Broker.Service Worker - that is trying to expire message from FilePendingCursor.
> =============================
> Found one Java-level deadlock:
> =============================
> "ActiveMQ NIO Worker 1003":
> waiting to lock monitor 0x00002aeeb515a4f8 (object 0x00000007807da3e8, a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor),
> which is held by "ActiveMQ BrokerService.worker.1"
> "ActiveMQ BrokerService.worker.1":
> waiting for ownable synchronizer 0x000000077ac84b40, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
> which is held by "ActiveMQ NIO Worker 1003"
> Java stack information for the threads listed above:
> ===================================================
> "ActiveMQ NIO Worker 1003":
> at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.addMessageLast(FilePendingMessageCursor.java:207)
> - waiting to lock <0x00000007807da3e8> (a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.addMessageLast(StoreQueueCursor.java:96)
> - locked <0x00000007784e8c88> (a org.apache.activemq.broker.region.cursors.StoreQueueCursor)
> at org.apache.activemq.broker.region.Queue.sendMessage(Queue.java:1855)
> at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:939)
> at org.apache.activemq.broker.region.Queue.send(Queue.java:733)
> at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:424)
> at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:445)
> at org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:297)
> at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
> at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:307)
> at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:147)
> at org.apache.activemq.broker.UserIDBroker.send(UserIDBroker.java:56)
> at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:152)
> at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:496)
> at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:756)
> at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)
> at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:148)
> at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
> at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> at org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:138)
> at org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:69)
> at org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:94)
> at org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:119)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> "ActiveMQ BrokerService.worker.1":
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x000000077ac84b40> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
> at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945)
> at org.apache.activemq.broker.region.Queue.messageExpired(Queue.java:1841)
> at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.discardExpiredMessage(FilePendingMessageCursor.java:474)
> at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.expireOldMessages(FilePendingMessageCursor.java:420)
> - locked <0x00000007807da3e8> (a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
> at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.onUsageChanged(FilePendingMessageCursor.java:398)
> - locked <0x00000007807da3e8> (a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
> at org.apache.activemq.usage.Usage$1.run(Usage.java:304)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)