You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@artemis.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/08/11 08:35:00 UTC
[jira] [Work logged] (ARTEMIS-6179) Deadlock in QueueImpl between management thread (removeMessage) and paging thread when JMX attribute Paused is scraped concurrently
[ https://issues.apache.org/jira/browse/ARTEMIS-6179?focusedWorklogId=1035016&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-1035016 ]
ASF GitHub Bot logged work on ARTEMIS-6179:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 11/Aug/26 08:34
Start Date: 11/Aug/26 08:34
Worklog Time Spent: 10m
Work Description: shivantaher-oviva opened a new pull request, #6606:
URL: https://github.com/apache/artemis/pull/6606
https://issues.apache.org/jira/browse/ARTEMIS-6179
`QueueImpl#deleteReference()` is still `synchronized` and calls `iterQueue()`, which acquires `depageLock` internally, while `depage()` acquires `depageLock` first and then enters a `synchronized(this)` block. The reversed lock order deadlocks `QueueControl#removeMessage()` against the paging executor whenever they race on an actively paging queue. In production this also wedges every other thread that needs the same monitor (e.g. JMX scrapes of `isPaused()`), and since `removeMessage()` holds the broker-wide management lock for the duration of the call, the entire JMX/management surface stops responding too.
This mirrors the deadlock already fixed for `copyReference()` in [ARTEMIS-5376](https://issues.apache.org/jira/browse/ARTEMIS-5376), fixed by removing `synchronized` there in apache/artemis@30c8fc7b10555478dfe6f912b38f312e3a39b165. `iterQueue()` already provides its own synchronization via `depageLock`, so the outer `synchronized` on `deleteReference()` is redundant and unsafe. `deleteReference()` wasn't touched by that fix because it didn't call `iterQueue()` yet at the time; it was moved onto `iterQueue()` two weeks later and the `synchronized` modifier was kept, reintroducing the same bug class.
Checked that this doesn't introduce a new race: `deleteReference()`'s callback (`incDelivering()` + `acknowledge()`) now runs in the exact synchronization context that `moveReference()` and `sendMessageToDeadLetterAddress()` already use unsynchronized since ARTEMIS-5376, calling the same underlying `acknowledge()`/`move()` machinery.
Verified with a dedicated reproduction test (`testRemoveMessageWhilstPagingAndConsuming`, racing `removeMessage()` against `depage()` while consuming) that reliably deadlocks without this change and passes cleanly with it, plus the existing `QueueControlTest`, `QueueControlUsingCoreTest`, `ManagementWithPagingServerTest`, and `LVQTest` suites, all green. The reproduction test itself is in the first commit on this branch and removed again in the third, since it's a probabilistic race rather than a deterministic test and isn't suited to the permanent suite.
Issue Time Tracking
-------------------
Worklog Id: (was: 1035016)
Remaining Estimate: 0h
Time Spent: 10m
> Deadlock in QueueImpl between management thread (removeMessage) and paging thread when JMX attribute Paused is scraped concurrently
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: ARTEMIS-6179
> URL: https://issues.apache.org/jira/browse/ARTEMIS-6179
> Project: Artemis
> Issue Type: Bug
> Components: Management
> Affects Versions: 2.54.0
> Environment: - ActiveMQ Artemis 2.54.0
> - OpenJDK 25.0.3+9-LTS (G1 GC)
> - JMX Prometheus Exporter scraping QueueControlImpl.isPaused() every 60 seconds
> - Active paging on affected queue
> Reporter: Moritz Schmitz von Hülst
> Priority: Major
> Labels: pull-request-available
> Attachments: dump.log
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> A three-thread deadlock forms when a management API call to removeMessage coincides with an active paging operation and a concurrent JMX scrape of the Paused attribute.
> Lock cycle (from JVM deadlock detector output):
> 1. Thread-3 (activemq-paging) — holds ReentrantLock 0x0000000586216410 (internal queue lock, acquired during depage()), then blocks on QueueImpl.depage():3288 waiting for the outer synchronized monitor 0x0000000586215490.
> 2. Thread-1 (activemq-web) — holds outer synchronized monitor 0x0000000586215490 (via QueueControlImpl.removeMessage() → deleteReference() → iterQueue():2071), then parks inside iterQueue() waiting for the same ReentrantLock that Thread-3 holds.
> 3. 10 × prometheus-http threads — all BLOCKED on QueueImpl.isPaused():2774 waiting for the outer monitor that Thread-1 holds.
> iterQueue() acquires the synchronized outer monitor first, then attempts to acquire the internal ReentrantLock with no timeout. depage() holds the ReentrantLock and then attempts to acquire the outer monitor. Classic lock-ordering inversion.
> Relevant call stacks (abbreviated):
> {code:java}
> "Thread-1 (activemq-web)" BLOCKED:
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:323)
> QueueImpl.iterQueue(QueueImpl.java:2071)
> QueueImpl.deleteReference(QueueImpl.java:2227)
> - locked <0x0000000586215490> (QueueImpl outer monitor)
> QueueControlImpl.removeMessage(QueueControlImpl.java:1169)
> "Thread-3 (activemq-paging-eb1150542a87)" BLOCKED:
> QueueImpl.depage(QueueImpl.java:3288)
> - waiting to lock <0x0000000586215490>
> (holds ReentrantLock 0x0000000586216410)
> "prometheus-http-1-1" (and 9 others) BLOCKED:
> QueueImpl.isPaused(QueueImpl.java:2774)
> - waiting to lock <0x0000000586215490>
> QueueControlImpl.isPaused(QueueControlImpl.java:1725) {code}
> Impact:
> - Paging completely frozen (consumers on affected queues receive nothing)
> - All JMX scrapes blocked indefinitely
> - Management API (removeMessage) frozen
> - Observed elapsed time before detection: ~314,000 seconds (3.6 days). Only a broker restart clears it.
> Root cause: iterQueue() acquires the synchronized outer monitor unconditionally, then blocks on ReentrantLock.lock() with no timeout. depage() acquires the ReentrantLock then tries to acquire the outer monitor. Lock ordering is not consistent between these two paths.
> Trigger: Requires all three of: active paging on the queue, a concurrent removeMessage management call, and JMX scraping of Paused (which piles up threads on the outer monitor, increasing the window).
> Workaround: Remove the Paused attribute from JMX exporter scrape patterns. This eliminates the prometheus-http pile-up, but does not fix the underlying lock-ordering inversion between the management thread and the paging thread.
> Steps to reproduce:
> 1. Configure a queue with paging enabled and active in-flight paging
> 2. Trigger a concurrent QueueControlImpl.removeMessage() via JMX or web console
> 3. (Optional) Have JMX scraping of Paused attribute active at 60s interval
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@artemis.apache.org
For additional commands, e-mail: issues-help@artemis.apache.org