[jira] [Commented] (ARTEMIS-6179) Deadlock in QueueImpl between management thread (removeMessage) and paging thread when JMX attribute Paused is scraped concurrently

["Moritz Schmitz von Hülst (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARTEMIS-6179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18103500#comment-18103500 ] 

Moritz Schmitz von Hülst commented on ARTEMIS-6179:
---------------------------------------------------

*Standalone reproduction added*

A self-contained Maven project that reliably triggers this deadlock has been created and verified on both 2.53.0 (first affected release) and 2.55.0 (latest release as of this comment).

*How to run*

{code}
git clone https://github.com/moritzschmitz-oviva/artemis.git
cd artemis
git checkout artemis-6179-deadlock-repro
mvn -pl tests/integration-tests -Dtest=DeadlockDepageLockDeleteReferenceTest test
{code}

The test is at:
tests/integration-tests/src/test/java/org/apache/activemq/artemis/tests/integration/paging/DeadlockDepageLockDeleteReferenceTest.java

It extends ActiveMQTestBase and uses @Timeout(60) — on a fixed version it passes cleanly; on affected versions it fails with "isPaused() blocked for >5000ms".

*What it does*

1. Starts an embedded broker with persistence enabled and a 512 KB address limit, forcing immediate paging.
2. Pre-fills the queue with 100 × 10 KB messages (1 MB >> limit), then a background sender at ~100 msg/s keeps paging active continuously.
3. A consumer at ~10 msg/s keeps depage() running (holds depageLock, then blocks on synchronized(QueueImpl.this)).
4. Five management threads loop on QueueControl.removeMessage(), which calls deleteReference() — synchronized(QueueImpl.this) — then iterQueue() — depageLock.lock().
5. A probe thread calls isPaused() every second with a 5 s timeout; if it doesn't return, the test fails.

*Observed deadlock (JVM-confirmed, findDeadlockedThreads)*

{code}
Thread "repro-management" [BLOCKED]
  waiting on: java.util.concurrent.locks.ReentrantLock$NonfairSync (depageLock)
  owned monitors:      [QueueImpl@...]          <- deleteReference synchronized(this)
  owned synchronizers: [ReentrantLock$NonfairSync@...]
  at QueueImpl.iterQueue(QueueImpl.java)
  at QueueImpl.deleteReference(QueueImpl.java)

Thread "activemq-paging" [BLOCKED]
  waiting on: QueueImpl@...                     <- depage() wants synchronized(this)
  owned synchronizers: [ReentrantLock$NonfairSync@...] <- holds depageLock
  at QueueImpl.depage(QueueImpl.java)
{code}

*Version matrix*

||Version||Affected||
|2.52.0|No — release branch cut before commit e9dbc11 (Feb 27 2026)|
|2.53.0|*Yes* — first release containing e9dbc11 (ARTEMIS-5376)|
|2.55.0|*Yes* — latest release, confirmed with this reproducer|

Root cause: commit e9dbc11 ("ARTEMIS-5376 Using iterQueue on deleteReferences") introduced the lock-order inversion. iterQueue() acquires depageLock inside a method that is already synchronized(this); depage() does the opposite.


> Deadlock in QueueImpl between management thread (removeMessage) and paging thread when JMX attribute Paused is scraped concurrently
> -----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: ARTEMIS-6179
>                 URL: https://issues.apache.org/jira/browse/ARTEMIS-6179
>             Project: Artemis
>          Issue Type: Bug
>          Components: Management
>    Affects Versions: 2.54.0
>         Environment: - ActiveMQ Artemis 2.54.0
>  - OpenJDK 25.0.3+9-LTS (G1 GC)
>  - JMX Prometheus Exporter scraping QueueControlImpl.isPaused() every 60 seconds
>  - Active paging on affected queue
>            Reporter: Moritz Schmitz von Hülst
>            Priority: Major
>         Attachments: dump.log
>
>
> A three-thread deadlock forms when a management API call to removeMessage coincides with an active paging operation and a concurrent JMX scrape of the Paused attribute.
> Lock cycle (from JVM deadlock detector output):
> 1. Thread-3 (activemq-paging) — holds ReentrantLock 0x0000000586216410 (internal queue lock, acquired during depage()), then blocks on QueueImpl.depage():3288 waiting for the outer synchronized monitor 0x0000000586215490.
> 2. Thread-1 (activemq-web) — holds outer synchronized monitor 0x0000000586215490 (via QueueControlImpl.removeMessage() → deleteReference() → iterQueue():2071), then parks inside iterQueue() waiting for the same ReentrantLock that Thread-3 holds.
> 3. 10 × prometheus-http threads — all BLOCKED on QueueImpl.isPaused():2774 waiting for the outer monitor that Thread-1 holds.
> iterQueue() acquires the synchronized outer monitor first, then attempts to acquire the internal ReentrantLock with no timeout. depage() holds the ReentrantLock and then attempts to acquire the outer monitor. Classic lock-ordering inversion.
> Relevant call stacks (abbreviated):
> {code:java}
> "Thread-1 (activemq-web)" BLOCKED:
>   java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:323)
>   QueueImpl.iterQueue(QueueImpl.java:2071)
>   QueueImpl.deleteReference(QueueImpl.java:2227)
>   - locked <0x0000000586215490> (QueueImpl outer monitor)
>   QueueControlImpl.removeMessage(QueueControlImpl.java:1169)
> "Thread-3 (activemq-paging-eb1150542a87)" BLOCKED:
>   QueueImpl.depage(QueueImpl.java:3288)
>   - waiting to lock <0x0000000586215490>
>   (holds ReentrantLock 0x0000000586216410)
> "prometheus-http-1-1" (and 9 others) BLOCKED:
>   QueueImpl.isPaused(QueueImpl.java:2774)
>   - waiting to lock <0x0000000586215490>
>   QueueControlImpl.isPaused(QueueControlImpl.java:1725) {code}
> Impact:
>  - Paging completely frozen (consumers on affected queues receive nothing)
>  - All JMX scrapes blocked indefinitely
>  - Management API (removeMessage) frozen
>  - Observed elapsed time before detection: ~314,000 seconds (3.6 days). Only a broker restart clears it.
> Root cause: iterQueue() acquires the synchronized outer monitor unconditionally, then blocks on ReentrantLock.lock() with no timeout. depage() acquires the ReentrantLock then tries to acquire the outer monitor. Lock ordering is not consistent between these two paths.
> Trigger: Requires all three of: active paging on the queue, a concurrent removeMessage management call, and JMX scraping of Paused (which piles up threads on the outer monitor, increasing the window).
> Workaround: Remove the Paused attribute from JMX exporter scrape patterns. This eliminates the prometheus-http pile-up, but does not fix the underlying lock-ordering inversion between the management thread and the paging thread.
> Steps to reproduce:
> 1. Configure a queue with paging enabled and active in-flight paging
> 2. Trigger a concurrent QueueControlImpl.removeMessage() via JMX or web console
> 3. (Optional) Have JMX scraping of Paused attribute active at 60s interval



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@artemis.apache.org
For additional commands, e-mail: issues-help@artemis.apache.org