You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2022/06/11 11:30:53 UTC

[GitHub] [rocketmq] hzh0425 opened a new pull request, #4449: [Summer of code] Use confirm offset in ReputMessageService

hzh0425 opened a new pull request, #4449:
URL: https://github.com/apache/rocketmq/pull/4449

   **Make sure set the target branch to `develop`**
   
   ## What is the purpose of the change
   
   tracking issue: https://github.com/apache/rocketmq/issues/4330
   
   ## Brief changelog
   
   Use confirm offset in ReputMessageService
   
   ## Verifying this change
   
   XXXX
   
   Follow this checklist to help us incorporate your contribution quickly and easily. Notice, `it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR`.
   
   - [x] Make sure there is a [Github issue](https://github.com/apache/rocketmq/issues) filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue. 
   - [x] Format the pull request title like `[ISSUE #123] Fix UnknownException when host config not exist`. Each commit in the pull request should have a meaningful subject line and body.
   - [x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
   - [x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in [test module](https://github.com/apache/rocketmq/tree/master/test).
   - [x] Run `mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle` to make sure basic checks pass. Run `mvn clean install -DskipITs` to make sure unit-test pass. Run `mvn clean test-compile failsafe:integration-test`  to make sure integration-test pass.
   - [ ] If this contribution is large, please file an [Apache Individual Contributor License Agreement](http://www.apache.org/licenses/#clas).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] coveralls commented on pull request #4449: [Summer of code] Use confirm offset in ReputMessageService

Posted by GitBox <gi...@apache.org>.
coveralls commented on PR #4449:
URL: https://github.com/apache/rocketmq/pull/4449#issuecomment-1152918508

   
   [![Coverage Status](https://coveralls.io/builds/49942330/badge)](https://coveralls.io/builds/49942330)
   
   Coverage increased (+0.02%) to 47.298% when pulling **5882be707423f246c5d418b8c81a00a79c630b17 on hzh0425:feature/use-confirmOffset** into **6645d0a42b743fc003ed54658cd24e99c375ac8b on apache:5.0.0-beta-dledger-controller**.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] RongtongJin commented on a diff in pull request #4449: [Summer of code] Use confirm offset in ReputMessageService

Posted by GitBox <gi...@apache.org>.
RongtongJin commented on code in PR #4449:
URL: https://github.com/apache/rocketmq/pull/4449#discussion_r895015426


##########
store/src/main/java/org/apache/rocketmq/store/ha/autoswitch/AutoSwitchHAService.java:
##########
@@ -220,14 +220,21 @@ public void updateConnectionLastCaughtUpTime(final String slaveAddress, final lo
      * Get confirm offset (min slaveAckOffset of all syncStateSet members)
      */
     public long getConfirmOffset() {
-        final Set<String> currentSyncStateSet = getSyncStateSet();
-        long confirmOffset = this.defaultMessageStore.getMaxPhyOffset();
-        for (HAConnection connection : this.connectionList) {
-            if (currentSyncStateSet.contains(connection.getClientAddress())) {
-                confirmOffset = Math.min(confirmOffset, connection.getSlaveAckOffset());
+        if (this.defaultMessageStore.getMessageStoreConfig().getBrokerRole() == BrokerRole.SYNC_MASTER) {
+            final Set<String> currentSyncStateSet = getSyncStateSet();
+            long confirmOffset = this.defaultMessageStore.getMaxPhyOffset();
+            for (HAConnection connection : this.connectionList) {
+                if (currentSyncStateSet.contains(connection.getClientAddress())) {
+                    confirmOffset = Math.min(confirmOffset, connection.getSlaveAckOffset());
+                }
+            }
+            return confirmOffset;
+        } else {
+            if (this.haClient != null) {
+                return this.haClient.getConfirmOffset();

Review Comment:
   这里可以做一个优化。。把confirmOffset记录下来，利用空间换时间，再每次slave ack的时候更新。。不然每次reput一条消息都会调用一下，比较消耗资源。



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] hzh0425 commented on pull request #4449: [Summer of code] Use confirm offset in ReputMessageService

Posted by GitBox <gi...@apache.org>.
hzh0425 commented on PR #4449:
URL: https://github.com/apache/rocketmq/pull/4449#issuecomment-1152917799

   问题场景: 网络分区下, master 与 slave, master 与 controller 都分区了, 但是没有和 client 分区. client 继续往 master 发送消息. 虽然在 allAckSyncStateSet = true 下, 不会发送成功, 但是这些消息仍然会被 reput by reputMessageService. 
   后续切换 master 的时候, 就会出现消息丢失的现象.
   解决办法就是让 reputMessageService 利用 autoSwitchHaSerivce 中的 ConfirmOffset 作为 reput 边界, 这样当消息发送失败时 (allAckSyncStateSet = true), confirmOffset 不会增大, reputMessageService 就不会 reput 这些失败的消息.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] codecov-commenter commented on pull request #4449: [Summer of code] Use confirm offset in ReputMessageService

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #4449:
URL: https://github.com/apache/rocketmq/pull/4449#issuecomment-1152918009

   # [Codecov](https://codecov.io/gh/apache/rocketmq/pull/4449?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#4449](https://codecov.io/gh/apache/rocketmq/pull/4449?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ba7d4f5) into [5.0.0-beta-dledger-controller](https://codecov.io/gh/apache/rocketmq/commit/81dc9386a1a5f932896bea3ddfa9452cddea1786?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (81dc938) will **decrease** coverage by `0.08%`.
   > The diff coverage is `75.00%`.
   
   > :exclamation: Current head ba7d4f5 differs from pull request most recent head 5882be7. Consider uploading reports for the commit 5882be7 to get more accurate results
   
   ```diff
   @@                         Coverage Diff                         @@
   ##             5.0.0-beta-dledger-controller    #4449      +/-   ##
   ===================================================================
   - Coverage                            43.17%   43.09%   -0.09%     
   + Complexity                            6343     6335       -8     
   ===================================================================
     Files                                  841      841              
     Lines                                59824    59833       +9     
     Branches                              8152     8154       +2     
   ===================================================================
   - Hits                                 25830    25785      -45     
   - Misses                               30643    30689      +46     
   - Partials                              3351     3359       +8     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/rocketmq/pull/4449?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...ketmq/store/ha/autoswitch/AutoSwitchHAService.java](https://codecov.io/gh/apache/rocketmq/pull/4449/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL2hhL2F1dG9zd2l0Y2gvQXV0b1N3aXRjaEhBU2VydmljZS5qYXZh) | `57.05% <63.63%> (-0.19%)` | :arrow_down: |
   | [...org/apache/rocketmq/store/DefaultMessageStore.java](https://codecov.io/gh/apache/rocketmq/pull/4449/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL0RlZmF1bHRNZXNzYWdlU3RvcmUuamF2YQ==) | `54.03% <100.00%> (+0.14%)` | :arrow_up: |
   | [...cketmq/store/ha/autoswitch/AutoSwitchHAClient.java](https://codecov.io/gh/apache/rocketmq/pull/4449/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL2hhL2F1dG9zd2l0Y2gvQXV0b1N3aXRjaEhBQ2xpZW50LmphdmE=) | `75.56% <100.00%> (+0.09%)` | :arrow_up: |
   | [...org/apache/rocketmq/common/stats/StatsItemSet.java](https://codecov.io/gh/apache/rocketmq/pull/4449/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jb21tb24vc3RhdHMvU3RhdHNJdGVtU2V0LmphdmE=) | `41.79% <0.00%> (-8.96%)` | :arrow_down: |
   | [...etmq/namesrv/routeinfo/BatchUnRegisterService.java](https://codecov.io/gh/apache/rocketmq/pull/4449/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bmFtZXNydi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcm9ja2V0bXEvbmFtZXNydi9yb3V0ZWluZm8vQmF0Y2hVblJlZ2lzdGVyU2VydmljZS5qYXZh) | `94.73% <0.00%> (-5.27%)` | :arrow_down: |
   | [...ache/rocketmq/common/stats/MomentStatsItemSet.java](https://codecov.io/gh/apache/rocketmq/pull/4449/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jb21tb24vc3RhdHMvTW9tZW50U3RhdHNJdGVtU2V0LmphdmE=) | `39.13% <0.00%> (-4.35%)` | :arrow_down: |
   | [...mq/broker/topic/TopicQueueMappingCleanService.java](https://codecov.io/gh/apache/rocketmq/pull/4449/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvdG9waWMvVG9waWNRdWV1ZU1hcHBpbmdDbGVhblNlcnZpY2UuamF2YQ==) | `12.43% <0.00%> (-3.63%)` | :arrow_down: |
   | [...a/org/apache/rocketmq/broker/BrokerController.java](https://codecov.io/gh/apache/rocketmq/pull/4449/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvQnJva2VyQ29udHJvbGxlci5qYXZh) | `44.04% <0.00%> (-3.31%)` | :arrow_down: |
   | [...a/org/apache/rocketmq/filter/util/BloomFilter.java](https://codecov.io/gh/apache/rocketmq/pull/4449/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZmlsdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9maWx0ZXIvdXRpbC9CbG9vbUZpbHRlci5qYXZh) | `59.13% <0.00%> (-2.16%)` | :arrow_down: |
   | [...he/rocketmq/broker/processor/PopReviveService.java](https://codecov.io/gh/apache/rocketmq/pull/4449/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvcHJvY2Vzc29yL1BvcFJldml2ZVNlcnZpY2UuamF2YQ==) | `24.90% <0.00%> (-1.81%)` | :arrow_down: |
   | ... and [13 more](https://codecov.io/gh/apache/rocketmq/pull/4449/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/rocketmq/pull/4449?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/rocketmq/pull/4449?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [81dc938...5882be7](https://codecov.io/gh/apache/rocketmq/pull/4449?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] RongtongJin merged pull request #4449: [Summer of code] Use confirm offset in ReputMessageService

Posted by GitBox <gi...@apache.org>.
RongtongJin merged PR #4449:
URL: https://github.com/apache/rocketmq/pull/4449


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] coveralls commented on pull request #4449: [Summer of code] Use confirm offset in ReputMessageService

Posted by GitBox <gi...@apache.org>.
coveralls commented on PR #4449:
URL: https://github.com/apache/rocketmq/pull/4449#issuecomment-1152918506

   
   [![Coverage Status](https://coveralls.io/builds/49942330/badge)](https://coveralls.io/builds/49942330)
   
   Coverage increased (+0.02%) to 47.298% when pulling **5882be707423f246c5d418b8c81a00a79c630b17 on hzh0425:feature/use-confirmOffset** into **6645d0a42b743fc003ed54658cd24e99c375ac8b on apache:5.0.0-beta-dledger-controller**.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org