You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2022/04/06 08:28:49 UTC

[GitHub] [rocketmq] cserwen opened a new pull request, #4122: [ISSUE #3914] Support multi dirs storage in DLedger

cserwen opened a new pull request, #4122:
URL: https://github.com/apache/rocketmq/pull/4122

**Make sure set the target branch to `develop`**

## What is the purpose of the change

Fix #3914

## Brief changelog

XX

## Verifying this change

XXXX

Follow this checklist to help us incorporate your contribution quickly and easily. Notice, `it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR`.

- [x] Make sure there is a [Github issue](https://github.com/apache/rocketmq/issues) filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
- [x] Format the pull request title like `[ISSUE #123] Fix UnknownException when host config not exist`. Each commit in the pull request should have a meaningful subject line and body.
- [x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
- [x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in [test module](https://github.com/apache/rocketmq/tree/master/test).
- [x] Run `mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle` to make sure basic checks pass. Run `mvn clean install -DskipITs` to make sure unit-test pass. Run `mvn clean test-compile failsafe:integration-test` to make sure integration-test pass.
- [ ] If this contribution is large, please file an [Apache Individual Contributor License Agreement](http://www.apache.org/licenses/#clas).

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] codecov-commenter commented on pull request #4122: [ISSUE #3914] Support multi dirs storage in DLedger

Posted by GitBox <gi...@apache.org>.

codecov-commenter commented on PR #4122:
URL: https://github.com/apache/rocketmq/pull/4122#issuecomment-1091056081

   # [Codecov](https://codecov.io/gh/apache/rocketmq/pull/4122?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#4122](https://codecov.io/gh/apache/rocketmq/pull/4122?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0c23af7) into [develop](https://codecov.io/gh/apache/rocketmq/commit/e7c87fa1cee96b48993fae0eb2de3974106e9e92?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e7c87fa) will **decrease** coverage by `0.06%`.
   > The diff coverage is `100.00%`.
   
   ```diff
   @@              Coverage Diff              @@
   ##             develop    #4122      +/-   ##
   =============================================
   - Coverage      47.92%   47.85%   -0.07%     
     Complexity      4999     4999              
   =============================================
     Files            634      635       +1     
     Lines          42529    42512      -17     
     Branches        5573     5567       -6     
   =============================================
   - Hits           20380    20343      -37     
   - Misses         19651    19677      +26     
   + Partials        2498     2492       -6     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/rocketmq/pull/4122?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...ache/rocketmq/store/config/MessageStoreConfig.java](https://codecov.io/gh/apache/rocketmq/pull/4122/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL2NvbmZpZy9NZXNzYWdlU3RvcmVDb25maWcuamF2YQ==) | `61.96% <100.00%> (+0.71%)` | :arrow_up: |
   | [...pache/rocketmq/store/dledger/DLedgerCommitLog.java](https://codecov.io/gh/apache/rocketmq/pull/4122/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL2RsZWRnZXIvRExlZGdlckNvbW1pdExvZy5qYXZh) | `76.07% <100.00%> (+0.09%)` | :arrow_up: |
   | [.../broker/subscription/SubscriptionGroupManager.java](https://codecov.io/gh/apache/rocketmq/pull/4122/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvc3Vic2NyaXB0aW9uL1N1YnNjcmlwdGlvbkdyb3VwTWFuYWdlci5qYXZh) | `64.70% <0.00%> (-16.48%)` | :arrow_down: |
   | [...org/apache/rocketmq/store/ha/WaitNotifyObject.java](https://codecov.io/gh/apache/rocketmq/pull/4122/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL2hhL1dhaXROb3RpZnlPYmplY3QuamF2YQ==) | `66.07% <0.00%> (-5.36%)` | :arrow_down: |
   | [...ketmq/common/protocol/body/ConsumerConnection.java](https://codecov.io/gh/apache/rocketmq/pull/4122/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jb21tb24vcHJvdG9jb2wvYm9keS9Db25zdW1lckNvbm5lY3Rpb24uamF2YQ==) | `95.83% <0.00%> (-4.17%)` | :arrow_down: |
   | [...lient/impl/consumer/DefaultMQPushConsumerImpl.java](https://codecov.io/gh/apache/rocketmq/pull/4122/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y2xpZW50L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jbGllbnQvaW1wbC9jb25zdW1lci9EZWZhdWx0TVFQdXNoQ29uc3VtZXJJbXBsLmphdmE=) | `40.17% <0.00%> (-3.86%)` | :arrow_down: |
   | [...he/rocketmq/client/trace/AsyncTraceDispatcher.java](https://codecov.io/gh/apache/rocketmq/pull/4122/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y2xpZW50L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jbGllbnQvdHJhY2UvQXN5bmNUcmFjZURpc3BhdGNoZXIuamF2YQ==) | `77.22% <0.00%> (-2.48%)` | :arrow_down: |
   | [...ache/rocketmq/store/AllocateMappedFileService.java](https://codecov.io/gh/apache/rocketmq/pull/4122/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL0FsbG9jYXRlTWFwcGVkRmlsZVNlcnZpY2UuamF2YQ==) | `39.02% <0.00%> (-2.44%)` | :arrow_down: |
   | [...a/org/apache/rocketmq/filter/util/BloomFilter.java](https://codecov.io/gh/apache/rocketmq/pull/4122/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZmlsdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9maWx0ZXIvdXRpbC9CbG9vbUZpbHRlci5qYXZh) | `60.43% <0.00%> (-2.20%)` | :arrow_down: |
   | [...e/rocketmq/remoting/netty/NettyRemotingServer.java](https://codecov.io/gh/apache/rocketmq/pull/4122/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cmVtb3Rpbmcvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3JlbW90aW5nL25ldHR5L05ldHR5UmVtb3RpbmdTZXJ2ZXIuamF2YQ==) | `57.34% <0.00%> (-1.90%)` | :arrow_down: |
   | ... and [22 more](https://codecov.io/gh/apache/rocketmq/pull/4122/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/rocketmq/pull/4122?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/rocketmq/pull/4122?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e7c87fa...0c23af7](https://codecov.io/gh/apache/rocketmq/pull/4122?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] coveralls commented on pull request #4122: [ISSUE #3914] Support multi dirs storage in DLedger

Posted by GitBox <gi...@apache.org>.

coveralls commented on PR #4122:
URL: https://github.com/apache/rocketmq/pull/4122#issuecomment-1091062865

   
   [![Coverage Status](https://coveralls.io/builds/48070672/badge)](https://coveralls.io/builds/48070672)
   
   Coverage increased (+0.1%) to 51.887% when pulling **0c23af7b6bb28a383c672c8ed275932f01a4fafd on cserwen:multi_dir** into **8745d4c538d5c96e65a56f6278029c5765e7d922 on apache:develop**.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] cserwen commented on pull request #4122: [ISSUE #3914] Support multi dirs storage in DLedger

Posted by GitBox <gi...@apache.org>.

cserwen commented on PR #4122:
URL: https://github.com/apache/rocketmq/pull/4122#issuecomment-1091323626

   @RongtongJin @Jason918 Hello, can you help me to review this pr?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] cserwen commented on a diff in pull request #4122: [ISSUE #3914] Support multi dirs storage in DLedger

Posted by GitBox <gi...@apache.org>.

cserwen commented on code in PR #4122:
URL: https://github.com/apache/rocketmq/pull/4122#discussion_r846894151


##########
store/src/main/java/org/apache/rocketmq/store/config/MessageStoreConfig.java:
##########
@@ -25,6 +25,10 @@
 
     public static final String MULTI_PATH_SPLITTER = System.getProperty("rocketmq.broker.multiPathSplitter", ",");
 
+    static {
+        System.setProperty("dLedger.multiPath.Splitter", MULTI_PATH_SPLITTER);
+    }

Review Comment:
   @RongtongJin  Good suggestion!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] cserwen commented on a diff in pull request #4122: [ISSUE #3914] Support multi dirs storage in DLedger

Posted by GitBox <gi...@apache.org>.

cserwen commented on code in PR #4122:
URL: https://github.com/apache/rocketmq/pull/4122#discussion_r852047716


##########
store/src/main/java/org/apache/rocketmq/store/config/MessageStoreConfig.java:
##########
@@ -33,6 +33,9 @@ public class MessageStoreConfig {
     @ImportantField
     private String storePathCommitLog = null;
 
+    @ImportantField
+    private String storePathDLedgerCommitLog = null;

Review Comment:
   @Jason918  I found this case by the unit test.
   https://github.com/apache/rocketmq/blob/50e314eb8ada23283dcdbb261766183c90fd435c/store/src/test/java/org/apache/rocketmq/store/dledger/MixCommitlogTest.java#L31
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] RongtongJin commented on a diff in pull request #4122: [ISSUE #3914] Support multi dirs storage in DLedger

Posted by GitBox <gi...@apache.org>.

RongtongJin commented on code in PR #4122:
URL: https://github.com/apache/rocketmq/pull/4122#discussion_r846599000


##########
store/src/main/java/org/apache/rocketmq/store/config/MessageStoreConfig.java:
##########
@@ -25,6 +25,10 @@
 
     public static final String MULTI_PATH_SPLITTER = System.getProperty("rocketmq.broker.multiPathSplitter", ",");
 
+    static {
+        System.setProperty("dLedger.multiPath.Splitter", MULTI_PATH_SPLITTER);
+    }

Review Comment:
   Should we set this property when enbaleDLedgerCommitlog is true？



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] Jason918 commented on a diff in pull request #4122: [ISSUE #3914] Support multi dirs storage in DLedger

Posted by GitBox <gi...@apache.org>.

Jason918 commented on code in PR #4122:
URL: https://github.com/apache/rocketmq/pull/4122#discussion_r852043279


##########
store/src/main/java/org/apache/rocketmq/store/config/MessageStoreConfig.java:
##########
@@ -33,6 +33,9 @@ public class MessageStoreConfig {
     @ImportantField
     private String storePathCommitLog = null;
 
+    @ImportantField
+    private String storePathDLedgerCommitLog = null;

Review Comment:
   Why not reuse storePathCommitLog? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] coveralls commented on pull request #4122: [ISSUE #3914] Support multi dirs storage in DLedger

Posted by GitBox <gi...@apache.org>.

coveralls commented on PR #4122:
URL: https://github.com/apache/rocketmq/pull/4122#issuecomment-1091062862

   
   [![Coverage Status](https://coveralls.io/builds/48070672/badge)](https://coveralls.io/builds/48070672)
   
   Coverage increased (+0.1%) to 51.887% when pulling **0c23af7b6bb28a383c672c8ed275932f01a4fafd on cserwen:multi_dir** into **8745d4c538d5c96e65a56f6278029c5765e7d922 on apache:develop**.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] cserwen commented on pull request #4122: [ISSUE #3914] Support multi dirs storage in DLedger

Posted by GitBox <gi...@apache.org>.

cserwen commented on PR #4122:
URL: https://github.com/apache/rocketmq/pull/4122#issuecomment-1096623242

   @RongtongJin ping~


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] RongtongJin merged pull request #4122: [ISSUE #3914] Support multi dirs storage in DLedger

Posted by GitBox <gi...@apache.org>.

RongtongJin merged PR #4122:
URL: https://github.com/apache/rocketmq/pull/4122


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] cserwen commented on pull request #4122: [ISSUE #3914] Support multi dirs storage in DLedger

Posted by GitBox <gi...@apache.org>.

cserwen commented on PR #4122:
URL: https://github.com/apache/rocketmq/pull/4122#issuecomment-1101153838

   @Jason918 ping


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] cserwen commented on a diff in pull request #4122: [ISSUE #3914] Support multi dirs storage in DLedger

Posted by GitBox <gi...@apache.org>.

cserwen commented on code in PR #4122:
URL: https://github.com/apache/rocketmq/pull/4122#discussion_r852046848


##########
store/src/main/java/org/apache/rocketmq/store/config/MessageStoreConfig.java:
##########
@@ -33,6 +33,9 @@ public class MessageStoreConfig {
     @ImportantField
     private String storePathCommitLog = null;
 
+    @ImportantField
+    private String storePathDLedgerCommitLog = null;

Review Comment:
   @Jason918  If we use the path of commitLog directly, there will be incompatibility in the mixed commitLog. That is, the commitLog of Master-slave will be read by DLedger. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] cserwen commented on a diff in pull request #4122: [ISSUE #3914] Support multi dirs storage in DLedger

Posted by GitBox <gi...@apache.org>.

cserwen commented on code in PR #4122:
URL: https://github.com/apache/rocketmq/pull/4122#discussion_r852046848


##########
store/src/main/java/org/apache/rocketmq/store/config/MessageStoreConfig.java:
##########
@@ -33,6 +33,9 @@ public class MessageStoreConfig {
     @ImportantField
     private String storePathCommitLog = null;
 
+    @ImportantField
+    private String storePathDLedgerCommitLog = null;

Review Comment:
   @Jason918  If we use the path of commitLog directly, there will be incompatibility in the mixed commitLog. That is, the commitLog of Master-slave will be read load DLedgerCommitLog. 



##########
store/src/main/java/org/apache/rocketmq/store/config/MessageStoreConfig.java:
##########
@@ -33,6 +33,9 @@ public class MessageStoreConfig {
     @ImportantField
     private String storePathCommitLog = null;
 
+    @ImportantField
+    private String storePathDLedgerCommitLog = null;

Review Comment:
   @Jason918  If we use the path of commitLog directly, there will be incompatibility in the mixed commitLog. That is, the commitLog of Master-slave will be load by DLedgerCommitLog. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org