You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "Crazylychee (via GitHub)" <gi...@apache.org> on 2025/08/17 17:42:37 UTC
[PR] [RIP-80] Master-slave metadata incremental synchronization [rocketmq]
Crazylychee opened a new pull request, #9617:
URL: https://github.com/apache/rocketmq/pull/9617
<!-- Please make sure the target branch is right. In most case, the target branch should be `develop`. -->
### Which Issue(s) This PR Fixes
<!-- Please ensure that the related issue has already been created, and [link this pull request to that issue using keywords](<https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword>) to ensure automatic closure. -->
Fixes #9616
### Brief Description
Master Node:
- MetadataChangeObserver serves as a monitoring interface, extending two subclasses, NoopMetadataChangeObserver and SyncMetadataChangeObserver, for implementing no-op and synchronous message sending, respectively. Configuration map operations in configuration classes such as TimerMetrics, MessageRequestModeManager, ScheduleMessageService, ConsumerOffsetManager, SubscriptionGroupManager, and TopicConfigManager have been modified: Compute methods are used to ensure the atomicity of message sending and in-memory configuration updates, ensuring simultaneous success and failure to prevent data inconsistencies.
- For the frequently updated consumerOffset, a dual buffering approach is implemented, with scheduled and buffer-full synchronous message sending strategies. This reduces the synchronous message sending burden caused by frequent consumerOffset updates. The scheduled strategy also mitigates the long inconsistency window caused by the buffer-full strategy in low-frequency updates.
- A scheduled task is used to store the latest metadata snapshot, which contains the current metadata and the maximum offset of the corresponding system topic.
- Methods for obtaining snapshots from the master node are defined in the AdminBrokerProcessor and BrokerOuterAPIs.
Slave nodes:
- Use DefaultLitePullConsumer to set the starting consumption point for consuming synchronous messages
<!-- Write a brief description for your pull request to help the maintainer understand the reasons behind your changes. -->
### How Did You Test This Change?
broker config
```conf
allowMetadataIncrementalSync = true
```
Relevant work is already in progress. Proposal documentation is available here:
https://m0sgzrp00oj.feishu.cn/docx/CvbedEJXto7A2BxfYVXcLXATnbd?from=from_copylink
<!-- In order to ensure the code quality of Apache RocketMQ, we expect every pull request to have undergone thorough testing. -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [RIP-81] Master-slave metadata incremental synchronization [rocketmq]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #9617:
URL: https://github.com/apache/rocketmq/pull/9617#issuecomment-3213111427
## [Codecov](https://app.codecov.io/gh/apache/rocketmq/pull/9617?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `20.24336%` with `721 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 47.65%. Comparing base ([`f40a69f`](https://app.codecov.io/gh/apache/rocketmq/commit/f40a69f9f14d5b2c4144e5288734fd0ac8dc5463?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`4d3a0ed`](https://app.codecov.io/gh/apache/rocketmq/commit/4d3a0ed5511c83efd9951096d0545067847082c5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/rocketmq/pull/9617?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...apache/rocketmq/broker/slave/SlaveSynchronize.java](https://app.codecov.io/gh/apache/rocketmq/pull/9617?src=pr&el=tree&filepath=broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fbroker%2Fslave%2FSlaveSynchronize.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvc2xhdmUvU2xhdmVTeW5jaHJvbml6ZS5qYXZh) | 9.22% | [187 Missing :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/9617?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...a/org/apache/rocketmq/broker/BrokerController.java](https://app.codecov.io/gh/apache/rocketmq/pull/9617?src=pr&el=tree&filepath=broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fbroker%2FBrokerController.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvQnJva2VyQ29udHJvbGxlci5qYXZh) | 3.10% | [122 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/9617?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/rocketmq/broker/out/BrokerOuterAPI.java](https://app.codecov.io/gh/apache/rocketmq/pull/9617?src=pr&el=tree&filepath=broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fbroker%2Fout%2FBrokerOuterAPI.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvb3V0L0Jyb2tlck91dGVyQVBJLmphdmE=) | 0.00% | [110 Missing :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/9617?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ocketmq/broker/processor/AdminBrokerProcessor.java](https://app.codecov.io/gh/apache/rocketmq/pull/9617?src=pr&el=tree&filepath=broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fbroker%2Fprocessor%2FAdminBrokerProcessor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvcHJvY2Vzc29yL0FkbWluQnJva2VyUHJvY2Vzc29yLmphdmE=) | 0.00% | [97 Missing :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/9617?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...pache/rocketmq/common/sync/MetadataChangeInfo.java](https://app.codecov.io/gh/apache/rocketmq/pull/9617?src=pr&el=tree&filepath=common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fcommon%2Fsync%2FMetadataChangeInfo.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jb21tb24vc3luYy9NZXRhZGF0YUNoYW5nZUluZm8uamF2YQ==) | 0.00% | [40 Missing :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/9617?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../broker/loadbalance/MessageRequestModeManager.java](https://app.codecov.io/gh/apache/rocketmq/pull/9617?src=pr&el=tree&filepath=broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fbroker%2Floadbalance%2FMessageRequestModeManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvbG9hZGJhbGFuY2UvTWVzc2FnZVJlcXVlc3RNb2RlTWFuYWdlci5qYXZh) | 22.85% | [25 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/9617?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/rocketmq/broker/sync/SyncMessageProducer.java](https://app.codecov.io/gh/apache/rocketmq/pull/9617?src=pr&el=tree&filepath=broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fbroker%2Fsync%2FSyncMessageProducer.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvc3luYy9TeW5jTWVzc2FnZVByb2R1Y2VyLmphdmE=) | 0.00% | [25 Missing :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/9617?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../rocketmq/broker/offset/ConsumerOffsetManager.java](https://app.codecov.io/gh/apache/rocketmq/pull/9617?src=pr&el=tree&filepath=broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fbroker%2Foffset%2FConsumerOffsetManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvb2Zmc2V0L0NvbnN1bWVyT2Zmc2V0TWFuYWdlci5qYXZh) | 48.93% | [22 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/9617?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../broker/subscription/SubscriptionGroupManager.java](https://app.codecov.io/gh/apache/rocketmq/pull/9617?src=pr&el=tree&filepath=broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fbroker%2Fsubscription%2FSubscriptionGroupManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvc3Vic2NyaXB0aW9uL1N1YnNjcmlwdGlvbkdyb3VwTWFuYWdlci5qYXZh) | 64.58% | [16 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/9617?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../org/apache/rocketmq/store/timer/TimerMetrics.java](https://app.codecov.io/gh/apache/rocketmq/pull/9617?src=pr&el=tree&filepath=store%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fstore%2Ftimer%2FTimerMetrics.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL3RpbWVyL1RpbWVyTWV0cmljcy5qYXZh) | 37.03% | [15 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/9617?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| ... and [10 more](https://app.codecov.io/gh/apache/rocketmq/pull/9617?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## develop #9617 +/- ##
=============================================
- Coverage 48.15% 47.65% -0.51%
+ Complexity 12083 12049 -34
=============================================
Files 1313 1317 +4
Lines 92839 93687 +848
Branches 11870 11946 +76
=============================================
- Hits 44710 44643 -67
- Misses 42636 43512 +876
- Partials 5493 5532 +39
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/rocketmq/pull/9617?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [RIP-81] Master-slave metadata incremental synchronization [rocketmq]
Posted by "RongtongJin (via GitHub)" <gi...@apache.org>.
RongtongJin merged PR #9617:
URL: https://github.com/apache/rocketmq/pull/9617
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org