You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "gogodjzhu (via GitHub)" <gi...@apache.org> on 2026/10/09 03:12:53 UTC

[I] [Bug] PushConsumer shutdown can race with asynchronous rebalance and leave orphan ProcessQueue [rocketmq]

gogodjzhu opened a new issue, #11309:
URL: https://github.com/apache/rocketmq/issues/11309

   ### Version
   
   Current `develop` branch.
   
   ### Description
   
   `DefaultMQPushConsumer.shutdown()` can race with asynchronous rebalance work
   performed by the shared `MQClientInstance`.
   
   The asynchronous rebalance path is:
   
   ```text
   RebalanceService
     → MQClientInstance.doRebalance()
     → MQConsumerInner.tryRebalance()
     → DefaultMQPushConsumerImpl.tryRebalance()
     → RebalanceImpl.doRebalance()
   ```
   
   Before this fix, `shutdown()` and `doRebalance()` were serialized by the
   consumer instance lock, but the actual asynchronous entry point,
   `tryRebalance()`, was neither synchronized nor guarded by the consumer
   lifecycle state.
   
   As a result, an asynchronous rebalance can still be in progress while the
   consumer shuts down, or can run after shutdown releases the consumer lock:
   
   1. `shutdown()` destroys `RebalanceImpl`, marks existing `ProcessQueue`
      instances as dropped, and clears `processQueueTable`;
   2. a concurrent or late `tryRebalance()` adds a new `ProcessQueue` to the
      table;
   3. rebalance dispatches a `PullRequest` referencing that `ProcessQueue` to
      `PullMessageService`;
   4. the consumer lifecycle has already ended, so the newly created
      `ProcessQueue` no longer has a normal rebalance-management or cleanup path.
   
   This leaves an orphan `ProcessQueue`: it can still be referenced by a queued
   `PullRequest`, but is no longer managed by the consumer's current
   `processQueueTable`.
   
   ### Impact
   
   An orphan `ProcessQueue` can cause:
   
   - **Unnecessary pulls and resource leakage**: a `PullRequest` referencing the
     orphan queue can remain in the shared `PullMessageService` queue. Since the
     queue was not marked as dropped, the pull path cannot discard it immediately.
   - **Missing lifecycle cleanup**: because the queue is not reliably managed by
     `processQueueTable`, subsequent rebalance and normal shutdown cleanup cannot
     reliably find and mark it as dropped.
   - **Inconsistent message-processing state**: stale pull and consume work can
     become detached from the consumer lifecycle, potentially causing useless
     consume scheduling, offset-handling anomalies, or duplicate-processing risk.
   - **Higher risk during frequent restart**: when a shared `MQClientInstance`
     remains alive for other producers or consumers, `RebalanceService` continues
     running and rapid consumer restarts make this race more likely.
   
   ### Reproduction
   
   1. Create a `DefaultMQPushConsumer` registered with a shared
      `MQClientInstance`.
   2. Let `RebalanceImpl.doRebalance()` enter the path that creates a
      `ProcessQueue` and dispatches a `PullRequest`.
   3. Call `DefaultMQPushConsumer.shutdown()` concurrently while the asynchronous
      rebalance is in progress.
   4. After shutdown, inspect `PullMessageService`:
      - the `PullRequest` references a `ProcessQueue` that is absent from the
        consumer's current `processQueueTable`; and
      - the referenced `ProcessQueue` is not marked as dropped.
   
   ### Proposed Fix
   
   Synchronize `DefaultMQPushConsumerImpl.tryRebalance()` with the same consumer
   instance lock used by `shutdown()`, and check that the consumer is in the
   `RUNNING` state before performing rebalance.
   
   This ensures that:
   
   - shutdown waits for a rebalance already in progress;
   - rebalance arriving after shutdown is skipped because the consumer is no
     longer `RUNNING`; and
   - no new `ProcessQueue` or `PullRequest` is created after
     `RebalanceImpl.destroy()`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] PushConsumer shutdown can race with asynchronous rebalance and leave orphan ProcessQueue [rocketmq]

Posted by "RockteMQ-AI (via GitHub)" <gi...@apache.org>.
RockteMQ-AI commented on issue #11309:
URL: https://github.com/apache/rocketmq/issues/11309#issuecomment-6101106991

   **Issue Evaluation**
   
   Category: `type/bug` (concurrency) | Status: **Confirmed**
   
   The reported race condition is valid. Code analysis confirms:
   
   **Root Cause:**
   - `DefaultMQPushConsumerImpl.shutdown()` (line 903) is `synchronized` and calls `rebalanceImpl.destroy()` at line 913
   - `DefaultMQPushConsumerImpl.tryRebalance()` (line 1397) is **not synchronized** and only checks `this.pause`, without verifying `serviceState`
   - The asynchronous rebalance path (`RebalanceService` → `MQClientInstance.doRebalance()` → `tryRebalance()`) can execute concurrently with or after `shutdown()`
   
   **Impact:** As described — orphan `ProcessQueue` instances that are no longer managed by `processQueueTable` but still referenced by queued `PullRequest` objects. This causes resource leakage, missing cleanup, and inconsistent state, especially during frequent consumer restarts with shared `MQClientInstance`.
   
   **Severity:** Medium — affects consumer lifecycle correctness, more likely under high-frequency restart scenarios.
   
   **Suggested Fix Direction:**
   Add a `serviceState` check in `tryRebalance()` before delegating to `rebalanceImpl.doRebalance()`:
   ```java
   public boolean tryRebalance() {
       if (this.serviceState != ServiceState.RUNNING) {
           return false;
       }
       if (!this.pause) {
           return this.rebalanceImpl.doRebalance(this.isConsumeOrderly());
       }
       return false;
   }
   ```
   
   An automated fix proposal will be generated. Reply `/approve` to proceed with PR generation.
   
   ---
   *Automated evaluation by github-manager-bot*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org