You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "unbridled-41 (via GitHub)" <gi...@apache.org> on 2026/08/29 17:37:43 UTC

[PR] [ISSUE #10988] Fix concurrent commitOffset race losing queue offsets for a new topic@group [rocketmq]

unbridled-41 opened a new pull request, #10992:
URL: https://github.com/apache/rocketmq/pull/10992

   <!-- Please make sure the target branch is right. In most case, the target branch should be `develop`. -->
   
   ### Which Issue(s) This PR Fixes
   
   - Fixes #10988
   
   ### Brief Description
   
   `ConsumerOffsetManager#commitOffset` used a non-atomic check-then-act to create the per `topic@group` map:
   
   ```java
   ConcurrentMap<Integer, Long> map = this.offsetTable.get(key);
   if (null == map) {
       map = new ConcurrentHashMap<>(2);
       map.put(queueId, offset);
       this.offsetTable.put(key, map);      // last put wins, the other thread's map is dropped
   } else { ... }
   ```
   
   When two remoting threads commit offsets for different queues of the same brand-new `topic@group` concurrently (the normal situation right after a consumer group starts on a multi-queue topic, or while a pop consumer acks several queues), both see `null`, both install their own map, and the second `put` silently drops the first queue's offset. `queryOffset` then returns `-1` for that queue until its next commit, so a consumer reconnect/restart in the window re-initializes per `consumeFromWhere` (duplicate consumption or skipping to max). The same race re-arms after `removeOffset(group)` / `cleanOffsetByTopic` while the group is still consuming.
   
   The fix replaces the get/put pair with `offsetTable.computeIfAbsent(key, k -> new ConcurrentHashMap<>(2))` — the same idiom the class already uses in `commitPullOffset` — so concurrent commits always land in the same map. The less-than-store warn behavior is unchanged.
   
   ### How Did You Test This Change?
   
   Added `ConsumerOffsetManagerTest#testConcurrentCommitOffsetDoesNotLoseQueues`: 8 threads commit a distinct queue for 500 brand-new `topic@group` keys each, then the test asserts every key retains all 8 queue offsets. It fails reproducibly on the unfixed code and passes with this change.
   
   `mvn -pl broker test -Dtest=ConsumerOffsetManagerTest` passes (6/6).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [ISSUE #10988] Fix concurrent commitOffset race losing queue offsets for a new topic@group [rocketmq]

Posted by "unbridled-41 (via GitHub)" <gi...@apache.org>.
unbridled-41 closed pull request #10992: [ISSUE #10988] Fix concurrent commitOffset race losing queue offsets for a new topic@group
URL: https://github.com/apache/rocketmq/pull/10992


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [ISSUE #10988] Fix concurrent commitOffset race losing queue offsets for a new topic@group [rocketmq]

Posted by "unbridled-41 (via GitHub)" <gi...@apache.org>.
unbridled-41 commented on PR #10992:
URL: https://github.com/apache/rocketmq/pull/10992#issuecomment-5468005513

   ### Test evidence (before → after)
   
   `ConsumerOffsetManagerTest#testConcurrentCommitOffsetDoesNotLoseQueues` was run repeatedly on JDK 8 with the fix reverted and with the fix applied (5 runs each, `mvn -pl broker test -Dtest='ConsumerOffsetManagerTest#testConcurrentCommitOffsetDoesNotLoseQueues'`):
   
   **On the unfixed code: 5/5 runs fail.** Example failure (queues 0 and 7 of `topic1` lost — the loser's map was overwritten):
   
   ```
   java.lang.AssertionError:
   [offsets of topic1]
   Expected size: 8 but was: 6 in:
   {1=1L, 2=1L, 3=1L, 4=1L, 5=1L, 6=1L}
   ```
   
   The race window is the very first `offsetTable.get(key)`/`offsetTable.put(key, map)` pair per fresh `topic@group` key, so it reproduces reliably, not stochastically.
   
   **With this PR: 5/5 runs pass** (`Tests run: 1, Failures: 0, Errors: 0, Skipped: 0` each run; the whole `ConsumerOffsetManagerTest` is 6/6 green).
   
   The loss is not just a test artifact: every commit the losing thread made for that queue is gone until that queue's next commit (5s flush cadence, but `queryOffset` returns `-1` immediately), so a consumer reconnect in that window re-initializes the queue per `consumeFromWhere`.
   
   Side note on CI: the workflow runs for this PR are in `action_required` state (first-time contributor) and will start once a maintainer approves them.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [ISSUE #10988] Fix concurrent commitOffset race losing queue offsets for a new topic@group [rocketmq]

Posted by "unbridled-41 (via GitHub)" <gi...@apache.org>.
unbridled-41 commented on PR #10992:
URL: https://github.com/apache/rocketmq/pull/10992#issuecomment-5547356168

   Evidence chain re-verified on 2026-09-05 (JDK 21, Maven 3.8.7), anchored to commits:
   
   - **After fix** — fix commit 549aa9acb (the develop-merge on the branch is from `update-branch`, no code change):
     `mvn -pl broker test -Dtest=ConsumerOffsetManagerTest` → `Tests run: 6, Failures: 0, Errors: 0, Skipped: 0`.
   - **Before fix** — base commit e348efa66 with the same regression test file:
     `mvn -pl broker test -Dtest=ConsumerOffsetManagerTest#testConcurrentCommitOffsetDoesNotLoseQueues` →
     `Tests run: 1, Failures: 1` — `java.lang.AssertionError: [offsets of topic1] Expected size: 8 but was: 6 in: {0=1L, 1=1L, 3=1L, 4=1L, 5=1L, 7=1L}` (ConsumerOffsetManagerTest.java:157): two of the eight concurrently committed queues lost their offsets through the non-atomic `get`/`put` race.
   - **CI**: all 10 checks green on this PR after re-triggering the workflows via branch update (maven-compile linux/macos/windows, bazel, CodeQL, coverage, license, misspell).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [ISSUE #10988] Fix concurrent commitOffset race losing queue offsets for a new topic@group [rocketmq]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #10992:
URL: https://github.com/apache/rocketmq/pull/10992#issuecomment-5544150064

   ## [Codecov](https://app.codecov.io/gh/apache/rocketmq/pull/10992?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `60.00000%` with `2 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 48.73%. Comparing base ([`ff8f6f7`](https://app.codecov.io/gh/apache/rocketmq/commit/ff8f6f74c560e391261ccd716707c6d20422e253?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`95b1c79`](https://app.codecov.io/gh/apache/rocketmq/commit/95b1c79b86c9babd4b90667204b3a01c58cd3cac?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/rocketmq/pull/10992?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [.../rocketmq/broker/offset/ConsumerOffsetManager.java](https://app.codecov.io/gh/apache/rocketmq/pull/10992?src=pr&el=tree&filepath=broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fbroker%2Foffset%2FConsumerOffsetManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvb2Zmc2V0L0NvbnN1bWVyT2Zmc2V0TWFuYWdlci5qYXZh) | 60.00% | [1 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/10992?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@              Coverage Diff              @@
   ##             develop   #10992      +/-   ##
   =============================================
   - Coverage      48.86%   48.73%   -0.14%     
   + Complexity     13796    13757      -39     
   =============================================
     Files           1381     1381              
     Lines         101574   101571       -3     
     Branches       13213    13212       -1     
   =============================================
   - Hits           49636    49500     -136     
   - Misses         45925    46036     +111     
   - Partials        6013     6035      +22     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/rocketmq/pull/10992?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [ISSUE #10988] Fix concurrent commitOffset race losing queue offsets for a new topic@group [rocketmq]

Posted by "ai-yang (via GitHub)" <gi...@apache.org>.
ai-yang commented on PR #10992:
URL: https://github.com/apache/rocketmq/pull/10992#issuecomment-6356276029

   Coordination with #10700/#10701: #10701 currently has maintainer approval, all 10 reported checks passing, and no merge conflict. It includes the classic-manager initialization fix as well as the RocksDB v1/WAL-ordering part of #10700.
   
   Your earlier offer to consolidate this classic-manager-only change and its deterministic regression into #10701 would avoid overlapping implementations. @unbridled-41 @RongtongJin, could you confirm which PR should carry the classic-manager portion? I am happy to keep the regression coverage in whichever PR the maintainers select.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org