You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "ai-yang (via GitHub)" <gi...@apache.org> on 2026/07/30 16:30:38 UTC
[PR] [ISSUE #10700] Make consumer offset commits concurrency-safe [rocketmq]
ai-yang opened a new pull request, #10701:
URL: https://github.com/apache/rocketmq/pull/10701
### Which Issue(s) This PR Fixes
- Fixes #10700
### Brief Description
Make first offset-map initialization atomic in the classic and RocksDB v1 consumer offset managers, and serialize incremental whole-map persistence for the same `topic@group`.
- Publish one shared inner queue-offset map with outer `putIfAbsent`, then update the selected winner map.
- Preserve the RocksDB v1 LMQ-specific map construction.
- In incremental mode, order the queue update, version update, whole-map serialization, and WAL batch write under the inner map monitor.
- Keep non-incremental commits lock-free across the existing concurrent map and allow different `topic@group` keys to proceed independently.
- Leave RocksDB v2 unchanged because it already initializes atomically and persists each queue under an independent key.
### Root Cause
Classic and RocksDB v1 used check-then-act initialization followed by unconditional outer `put`, so two first commits could publish different inner maps and overwrite one queue. RocksDB v1 incremental mode additionally allowed an older serialized whole-map batch to be written after a newer batch for the same RocksDB key.
Historical unmerged PR #1427 identified the classic initialization race. This PR preserves that analysis while extending coverage to RocksDB v1, incremental WAL ordering, and deterministic persistence/reload tests. Open PRs #10625, #9602, and #9877 touch related files but do not change these commit paths.
### Impact
Concurrent first commits retain every queue in memory and after persistence. Incremental restart recovery cannot regress to an older same-key snapshot. Existing rollback warnings, version semantics, LMQ allocation, and v2 behavior are preserved.
### How Did You Test This Change?
- Deterministic red tests on the previous implementation:
- classic concurrent first commits: expected 2 queues, reloaded 1;
- RocksDB v1 periodic LMQ commits: expected 2 queues, retained 1;
- RocksDB v1 incremental delayed old batch: expected 2 queues after restart, reloaded 1.
- `ConsumerOffsetManagerTest` + `RocksDBConsumerOffsetManagerTest`: 21 tests passed.
- Full `broker` test suite: 755 tests passed, 0 failures, 0 errors, 4 skipped.
- Maven Checkstyle: 0 violations.
- SpotBugs: 0 findings.
- `git diff --check`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #10700] Make consumer offset commits concurrency-safe [rocketmq]
Posted by "ai-yang (via GitHub)" <gi...@apache.org>.
ai-yang commented on PR #10701:
URL: https://github.com/apache/rocketmq/pull/10701#issuecomment-6119984513
Following up on the approval: head `03695c8703aa26d52f7d2b999935a5a8513c8224` still has all 10 checks passing and no merge conflict. Could a maintainer confirm whether anything else is needed before merging? The existing description covers coordination with #10992 and the additional RocksDB v1/WAL-ordering scope.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #10700] Make consumer offset commits concurrency-safe [rocketmq]
Posted by "ai-yang (via GitHub)" <gi...@apache.org>.
ai-yang commented on PR #10701:
URL: https://github.com/apache/rocketmq/pull/10701#issuecomment-5463252649
Refreshed this PR onto the current `develop` (`e348efa66`) and force-pushed one signed-off commit (`03695c870`). I also added an inline explanation for the per-key `synchronized(map)` block: it orders the in-memory update, whole-map snapshot, and WAL write so an older snapshot cannot overwrite newer offsets.
- `ConsumerOffsetManagerTest` + `RocksDBConsumerOffsetManagerTest`: 21/21 passed;
- all 10 modules in the targeted `broker -am` reactor succeeded;
- Checkstyle and SpotBugs reported no findings;
- `git diff --check` passed.
The production change is +37/-20 lines; +261/-6 lines are deterministic concurrency/persistence tests. Full CI has been retriggered.
@RongtongJin @lizhimins, could you please take a human review when convenient, especially of same-key WAL ordering and the classic/RocksDB v1 initialization paths?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #10700] Make consumer offset commits concurrency-safe [rocketmq]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #10701:
URL: https://github.com/apache/rocketmq/pull/10701#issuecomment-5463389819
## [Codecov](https://app.codecov.io/gh/apache/rocketmq/pull/10701?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `79.31034%` with `6 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 48.47%. Comparing base ([`e348efa`](https://app.codecov.io/gh/apache/rocketmq/commit/e348efa66b08eb645ee123706ea6492fa9a3ad35?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`03695c8`](https://app.codecov.io/gh/apache/rocketmq/commit/03695c8703aa26d52f7d2b999935a5a8513c8224?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/rocketmq/pull/10701?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...broker/config/v1/RocksDBConsumerOffsetManager.java](https://app.codecov.io/gh/apache/rocketmq/pull/10701?src=pr&el=tree&filepath=broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fbroker%2Fconfig%2Fv1%2FRocksDBConsumerOffsetManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvY29uZmlnL3YxL1JvY2tzREJDb25zdW1lck9mZnNldE1hbmFnZXIuamF2YQ==) | 82.60% | [3 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/10701?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../rocketmq/broker/offset/ConsumerOffsetManager.java](https://app.codecov.io/gh/apache/rocketmq/pull/10701?src=pr&el=tree&filepath=broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fbroker%2Foffset%2FConsumerOffsetManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvb2Zmc2V0L0NvbnN1bWVyT2Zmc2V0TWFuYWdlci5qYXZh) | 66.66% | [1 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/10701?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## develop #10701 +/- ##
=============================================
- Coverage 48.58% 48.47% -0.11%
+ Complexity 13678 13647 -31
=============================================
Files 1381 1381
Lines 101475 101486 +11
Branches 13190 13192 +2
=============================================
- Hits 49304 49198 -106
- Misses 46170 46266 +96
- Partials 6001 6022 +21
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/rocketmq/pull/10701?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org