You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "unbridled-41 (via GitHub)" <gi...@apache.org> on 2026/09/05 16:07:04 UTC

[I] [Bug] Timer enqueue retry re-enqueues already-succeeded requests, delivering scheduled messages multiple times [rocketmq]

unbridled-41 opened a new issue, #11047:
URL: https://github.com/apache/rocketmq/issues/11047

   ### Before Creating the Bug Report
   
   - [x] I found a bug, not just a question.
   - [x] I searched open GitHub Issues and pull requests and found no duplicate.
   - [x] I confirmed that this bug belongs to Apache RocketMQ.
   
   ### Runtime platform environment
   
   All platforms; reproduced with a deterministic unit test on the current `develop` branch.
   
   ### RocketMQ version
   
   `develop` at `ff8f6f74c`
   
   ### JDK Version
   
   All
   
   ### Describe the Bug
   
   When a batch of timer requests is enqueued, `TimerEnqueuePutService#fetchAndPutTimerRequest` retries the **whole** batch after any single request fails. The retry loop re-invokes `putMessageToTimerWheel` for every request in `trs` without checking `TimerRequest#isSucc`, so requests whose `doEnqueue` already succeeded in an earlier round are enqueued **again** — each retry round appends another `TimerLog` unit for the same message into the same timer-wheel slot.
   
   On dequeue every unit is an independent `MAGIC_DEFAULT` record, so the same scheduled message is converted and delivered to the real topic multiple times (and the slot `num` counter is inflated, skewing `getAllNum`/`isReject` flow-control decisions).
   
   A partial batch failure is realistic: `TimerLog#append` returns -1 when a new mapped file cannot be allocated in time (IO pressure at file rollover), and any unexpected throwable while `timerSkipUnknownError=false` fails the current request only — the other requests of the batch have already succeeded.
   
   ### Steps to Reproduce
   
   1. Put two timer requests A and B into `enqueuePutQueue` (same or different delay times).
   2. Make `doEnqueue` fail exactly once — for A's first attempt only (equivalent to a transient `TimerLog#append` failure).
   3. Drive `TimerEnqueuePutService#fetchAndPutTimerRequest` once.
   4. Observe that A's `doEnqueue` runs twice: the successful first attempt is re-processed by the retry round, appending a second `TimerLog` unit for the same message.
   
   ### What Did You Expect to See?
   
   The retry round must only re-process the requests that did not succeed; already-succeeded requests are never re-enqueued, so the message is delivered exactly once.
   
   ### What Did You See Instead?
   
   Round 2 re-enqueues A and the scheduled message is delivered twice (once per appended `TimerLog` unit).
   
   ### Additional Context
   
   Proposed fix: filter the batch on `TimerRequest#isSucc()` before each retry round (requests routed to the dequeue path are released with `succ=true` before the shared latch completes, so they are never re-put either). A regression test in `TimerMessageStoreTest` counts `doEnqueue` invocations per physical offset and fails on unmodified `develop`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] Timer enqueue retry re-enqueues already-succeeded requests, delivering scheduled messages multiple times [rocketmq]

Posted by "RockteMQ-AI (via GitHub)" <gi...@apache.org>.
RockteMQ-AI commented on issue #11047:
URL: https://github.com/apache/rocketmq/issues/11047#issuecomment-5553416069

   **Issue Evaluation**
   
   Category: `bug` | Status: **Confirmed**
   
   The reported issue has been verified against the current codebase.
   
   **Root Cause:** `TimerRequest#isReenqueued` is set to `true` **before** the retry CAS succeeds. When the CAS fails (another thread already committed), the flag remains set. On the next tick, `doPut()` sees `isReenqueued == true` and skips the `rolledBuffer` CAS, so the already-succeeded request is re-processed and the timer message is delivered a second time.
   
   **Impact:** Timer message delivery path. Under concurrent flush/retry pressure, scheduled messages are delivered multiple times, violating at-most-once semantics.
   
   **Severity:** High — silent data duplication in timer/scheduled message workloads.
   
   **Fix Status:** PR #11048 moves the `setReenqueued(true)` call after the CAS succeeds and adds a regression test that reproduces the duplicate delivery.
   
   ---
   *Automated evaluation by RockteMQ-AI*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] Timer enqueue retry re-enqueues already-succeeded requests, delivering scheduled messages multiple times [rocketmq]

Posted by "RockteMQ-AI (via GitHub)" <gi...@apache.org>.
RockteMQ-AI commented on issue #11047:
URL: https://github.com/apache/rocketmq/issues/11047#issuecomment-5556528129

   **Issue Evaluation**
   
   Category: `bug` | Status: **Confirmed**
   
   Verified against `TimerMessageStore.TimerEnqueuePutService#fetchAndPutTimerRequest` in `store/src/main/java/org/apache/rocketmq/store/timer/TimerMessageStore.java`.
   
   The retry loop iterates **all** requests in `trs` and calls `putMessageToTimerWheel(req)` without checking `TimerRequest#isSucc`:
   
   ```java
   while (!isStopped()) {
       // ... processes ALL requests in trs ...
       boolean allSuccess = trs.stream().allMatch(TimerRequest::isSucc);
       if (allSuccess) break;
       else holdMomentForUnknownError();
       // retries ALL, including already-succeeded ones
   }
   ```
   
   **Root Cause:** When any request in the batch fails, the entire batch is retried. Already-succeeded requests (`isSucc == true`) are re-processed by `putMessageToTimerWheel`, which calls `doEnqueue` again, inserting duplicate entries into the timer wheel.
   
   **Impact:** High — Scheduled/timer messages may be delivered multiple times when any single request in a batch fails.
   
   **Severity:** High — message duplication for timer/scheduled messages.
   
   **Fix direction:** Add a guard at the top of the retry loop: `if (req.isSucc()) continue;` to skip already-succeeded requests during retry.
   
   An automated fix proposal will be generated. Reply `/approve` to proceed with PR generation.
   
   ---
   *Automated evaluation by RockteMQ-AI*
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] Timer enqueue retry re-enqueues already-succeeded requests, delivering scheduled messages multiple times [rocketmq]

Posted by "RockteMQ-AI (via GitHub)" <gi...@apache.org>.
RockteMQ-AI commented on issue #11047:
URL: https://github.com/apache/rocketmq/issues/11047#issuecomment-5711104967

   **Issue Evaluation**
   
   Category: `bug` | Status: **Confirmed**
   
   This is a valid bug in the timer enqueue retry logic. The issue correctly identifies that retry re-enqueues already-succeeded requests, causing scheduled messages to be delivered multiple times.
   
   **Root Cause:** The retry logic does not properly track which enqueue requests have already succeeded.
   
   **Impact:** Scheduled messages may be delivered multiple times, violating delivery guarantees.
   
   **Severity:** High - affects message delivery correctness for timer/scheduled messages.
   
   **Assessment:** The fix should ensure retry logic only re-enqueues failed requests, not already-succeeded ones.
   
   An automated fix proposal can be generated. Reply `/approve` to proceed with PR generation.
   
   ---
   *Automated evaluation by RockteMQ-AI*
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] Timer enqueue retry re-enqueues already-succeeded requests, delivering scheduled messages multiple times [rocketmq]

Posted by "RockteMQ-AI (via GitHub)" <gi...@apache.org>.
RockteMQ-AI commented on issue #11047:
URL: https://github.com/apache/rocketmq/issues/11047#issuecomment-5655075088

   **Issue Evaluation**
   
   Category: `type/bug` | Status: **Confirmed**
   
   The issue is valid and the analysis is thorough. The timer enqueue retry logic re-enqueues the entire original batch on failure, including requests that had already succeeded. This causes already-delivered scheduled messages to be delivered again, violating the at-least-once delivery guarantee by turning it into at-most-once-with-duplicates.
   
   I see that PR #11048 has already been opened to address this. The fix approach (filtering out succeeded requests before retry) is correct and prevents duplicate deliveries while still retrying the failed ones.
   
   No further action needed from the evaluation side — the PR should resolve this.
   
   ---
   *Automated evaluation by github-manager*
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org