You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@activemq.apache.org by "mattrpav (via GitHub)" <gi...@apache.org> on 2026/03/10 02:04:52 UTC
[I] Possible deadlock with ConcurrentStoreAndDispatch with useCache=true [activemq]
mattrpav opened a new issue, #1747:
URL: https://github.com/apache/activemq/issues/1747
Version: v5.19.2 (w/ recent KahaDB updates)
Notes: this only occurs when useCache=true. When disabled, there is no deadlock.
Producer thread:
```
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.2/Native Method)
- parking to wait for <0x000000078b900220> (a org.apache.activemq.store.kahadb.KahaDBStore$StoreQueueTask$InnerFutureTask)
at java.util.concurrent.locks.LockSupport.park(java.base@21.0.2/LockSupport.java:221)
at java.util.concurrent.FutureTask.awaitDone(java.base@21.0.2/FutureTask.java:500)
at java.util.concurrent.FutureTask.get(java.base@21.0.2/FutureTask.java:190)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.waitForAsyncMessage(AbstractStoreCursor.java:575)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.syncWithStore(AbstractStoreCursor.java:337)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.tryAddMessageLast(AbstractStoreCursor.java:269)
- locked <0x000000078b900260> (a org.apache.activemq.broker.region.cursors.QueueStorePrefetch)
at org.apache.activemq.broker.region.cursors.AbstractPendingMessageCursor.addMessageLast(AbstractPendingMessageCursor.java:93)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.tryAddMessageLast(StoreQueueCursor.java:105)
- locked <0x000000078b900398> (a org.apache.activemq.broker.region.cursors.StoreQueueCursor)
at org.apache.activemq.broker.region.cursors.AbstractPendingMessageCursor.addMessageLast(AbstractPendingMessageCursor.java:93)
at org.apache.activemq.broker.region.Queue.doPendingCursorAdditions(Queue.java:827)
at org.apache.activemq.broker.region.Queue.tryOrderedCursorAdd(Queue.java:938)
at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:917)
at org.apache.activemq.broker.region.Queue.send(Queue.java:767)
...
Locked ownable synchronizers:
- ...
- ...
- <0x00000007bf78a970> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
```
ConcurrentQueueStoreAndDispatch thread:
```
"ConcurrentQueueStoreAndDispatch" #657 [122627] daemon prio=8 os_prio=31 cpu=890.98ms elapsed=815.63s tid=0x000000014d982600 nid=122627 waiting on condition [0x000000035faca000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.2/Native Method)
- parking to wait for <0x00000007bf78a970> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@21.0.2/LockSupport.java:221)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@21.0.2/AbstractQueuedSynchronizer.java:754)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@21.0.2/AbstractQueuedSynchronizer.java:990)
at java.util.concurrent.locks.ReentrantLock$Sync.lock(java.base@21.0.2/ReentrantLock.java:153)
at java.util.concurrent.locks.ReentrantLock.lock(java.base@21.0.2/ReentrantLock.java:322)
at org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:1264)
at org.apache.activemq.store.kahadb.MessageDatabase$6.visit(MessageDatabase.java:1198)
at org.apache.activemq.store.kahadb.data.KahaAddMessageCommand.visit(KahaAddMessageCommand.java:241)
at org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:1195)
at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:1089)
at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:1065)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.addMessage(KahaDBStore.java:537)
at org.apache.activemq.store.kahadb.KahaDBStore$StoreQueueTask.run(KahaDBStore.java:1669)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.2/ThreadPoolExecutor.java:1144)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.2/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.2/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.2/Thread.java:1583)
Locked ownable synchronizers:
- <0x00000007bf614ef0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@activemq.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@activemq.apache.org
For additional commands, e-mail: gitbox-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
Re: [I] Possible deadlock with ConcurrentStoreAndDispatch with useCache=true [activemq]
Posted by "cshannon (via GitHub)" <gi...@apache.org>.
cshannon commented on issue #1747:
URL: https://github.com/apache/activemq/issues/1747#issuecomment-4031245391
@mattrpav - It looks like you cut out stuff from the stack traces, do you have any custom plugins or code or anything else that might be causing this?
The reason I am asking is because I took a look and the following appears to be going on.
1. Thread number 1 is adding a new message to the cache but detects that the cache needs to be disabled because of memory. It first has to wait for any outstanding pending messages by waiting on their future where it's stuck. It's waiting for the future for a message which is presumably being handled by thread 2.
2. Thread number 2 is trying to add the message but is stuck waiting to acquire the KahaDB index lock.
What's interesting is the output shows that Thread number 1 holds the index lock which thread 2 needs (ie deadlock). However in the stack trace you posted, I don't see anywhere that thread 1 would actually acquire the index lock. Nothing in that trace would grab that lock, it's internal to KahaDB. So that's why I am wondering if you cut out something relevant here that would show where that lock was acquired.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@activemq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@activemq.apache.org
For additional commands, e-mail: gitbox-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
Re: [I] Possible deadlock with ConcurrentStoreAndDispatch with useCache=true [activemq]
Posted by "mattrpav (via GitHub)" <gi...@apache.org>.
mattrpav commented on issue #1747:
URL: https://github.com/apache/activemq/issues/1747#issuecomment-4033575558
Our report is invalid, we identified the root cause. It was circumstantial that useCache=true bubbled up the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@activemq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@activemq.apache.org
For additional commands, e-mail: gitbox-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
Re: [I] Possible deadlock with ConcurrentStoreAndDispatch with useCache=true [activemq]
Posted by "mattrpav (via GitHub)" <gi...@apache.org>.
mattrpav closed issue #1747: Possible deadlock with ConcurrentStoreAndDispatch with useCache=true
URL: https://github.com/apache/activemq/issues/1747
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@activemq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@activemq.apache.org
For additional commands, e-mail: gitbox-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact