You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2022/08/15 08:34:53 UTC

[GitHub] [rocketmq] shengminw opened a new pull request, #4825: [ISSUE#4824] Support consumequeue, indexfile multiple directories storage mode

shengminw opened a new pull request, #4825:
URL: https://github.com/apache/rocketmq/pull/4825

   [ISSUE#4824](https://github.com/apache/rocketmq/issues/4824)
   
   In RIP-7 https://github.com/apache/rocketmq/wiki/RIP-7-Multiple-Directories-Storage-Support, CommitLog File has supported multi-directories storage mode.
   In this case, the commitlog can be stored on multiple disks, but consumequeue/index is still stored on one disk. However, the size of consumequeue/index can reach tens of Gbs, or even 100 Gbs, and the space used can't be easily ignored.
   The current commitLog multipath needs to be configured independently, as follows:
   ```java
   storePathRootDir=/rmqstore1
   storePathCommitLog=/rmqstore1/commitlog:/rmqstore2/commitlog
   ```
   Therefore, I optimized the multi-path mode, and the improvements are as follows:
   ● Add multi-path mode for consumequeue.  
   ● Add multi-path mode for indexfile.
   ● Optimize the method to get the commitlog path. It is not necessary to specify storePathCommitLog separately, and generate the storePathCommitLog from storePathRootDir.
   ● Add a Timed thread to check path capacity. This will  try to avoid writing to a nearly full disk.
   ● Loading  metadata in multi-path. By default, the config file is stored in the first path, and the config bak file is stored in the second path. For example: storePathRootDir = /rmqstore0:/rmqstore1:/rmqstore2, config dir = /rmqstore0/config, config bak dir = /rmqstore1/config. At startup, scan all paths and load the config file from the latest path of dataversion.（This ensures that when the path is changed, the latest version can be loaded）
   ● Loading checkpoint file and abort file in multi-path. The loading path of the checkpoint file and the abort file is the same as that of the config file.
   
   ## Chinese Version / 中文版本
   在RIP-7 https://github.com/apache/rocketmq/wiki/RIP-7-Multiple-Directories-Storage-Support 当中，CommitLog文件已经支持了多文件路径的存储模式。
   在这个情况下，commitLog可以被存储至多块磁盘当中，但是consumequeue和index依然存储于单盘上。然而consumequeue 和 index的文件大小也可能达到数十G或者上百G的大小。这样的存储空间并不能被忽略。
   当前的commitLog的多路径需要被额外配置，如下所示：
   ```java
   storePathRootDir=/rmqstore1
   storePathCommitLog=/rmqstore1/commitlog:/rmqstore2/commitlog
   ```
   因此，我对多路径模式做了一定的优化，改进主要如下：
   ● consumequeue的存储增加多路径模式。
   ● indexfile的存储增加多路径模式。
   ● 优化commitlog获取文件路径的方式。不必要单独设置storePathCommitLog，storePathCommitLog可以从storePathRootDir根目录下直接生成。
   ● 增加一个定时线程检查路径的容量。这样能尽量避免写入至快满的磁盘中。
   ● 多路径下的元数据加载。默认将config文件夹写入至第1个路径中，config文件夹的bak文件写入至第2个路径。例如：storePathRootDir = /rmqstore0:/rmqstore1:/rmqstore2, config dir = /rmqstore0/config, config bak dir = /rmqstore1/config。在启动的时候，扫描所有路径，选择dataversion最新的所在路径作为初始加载路径。（这样能确保路径被更改时，能够从最新的路径下加载）
   ● 加载checkpoint文件和abort文件。checkpoint文件和abort文件的加载路径与config路径保持一致。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] shengminw commented on pull request #4825: [ISSUE#4824] Support consumequeue, indexfile multiple directories storage mode

Posted by GitBox <gi...@apache.org>.
shengminw commented on PR #4825:
URL: https://github.com/apache/rocketmq/pull/4825#issuecomment-1228139819

   @aaron-ai thanks for your suggestion, I will check the compatible with windows platform.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] github-actions[bot] commented on pull request #4825: [ISSUE#4824] Support consumequeue, indexfile multiple directories storage mode

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #4825:
URL: https://github.com/apache/rocketmq/pull/4825#issuecomment-1712666895

   This PR is stale because it has been open for 365 days with no activity. It will be closed in 3 days if no further activity occurs. If you wish not to mark it as stale, please leave a comment in this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] zongtanghu commented on pull request #4825: [ISSUE#4824] Support consumequeue, indexfile multiple directories storage mode

Posted by GitBox <gi...@apache.org>.
zongtanghu commented on PR #4825:
URL: https://github.com/apache/rocketmq/pull/4825#issuecomment-1226729935

   Can you fix the codes' conflicts firstly! @shengminw 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] coveralls commented on pull request #4825: [ISSUE#4824] Support consumequeue, indexfile multiple directories storage mode

Posted by GitBox <gi...@apache.org>.
coveralls commented on PR #4825:
URL: https://github.com/apache/rocketmq/pull/4825#issuecomment-1216097710

   
   [![Coverage Status](https://coveralls.io/builds/51682798/badge)](https://coveralls.io/builds/51682798)
   
   Coverage decreased (-0.09%) to 49.139% when pulling **f52caeee88b8644cc3eba9f2062e210c5980fd7c on shengminw:multi-route** into **8dd07c84697d3709f1f4bef5d78089b7d171b6cd on apache:develop**.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] github-actions[bot] closed pull request #4825: [ISSUE#4824] Support consumequeue, indexfile multiple directories storage mode

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #4825: [ISSUE#4824] Support consumequeue, indexfile multiple directories storage  mode
URL: https://github.com/apache/rocketmq/pull/4825


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] coveralls commented on pull request #4825: [ISSUE#4824] Support consumequeue, indexfile multiple directories storage mode

Posted by GitBox <gi...@apache.org>.
coveralls commented on PR #4825:
URL: https://github.com/apache/rocketmq/pull/4825#issuecomment-1216097709

   
   [![Coverage Status](https://coveralls.io/builds/51682798/badge)](https://coveralls.io/builds/51682798)
   
   Coverage decreased (-0.09%) to 49.139% when pulling **f52caeee88b8644cc3eba9f2062e210c5980fd7c on shengminw:multi-route** into **8dd07c84697d3709f1f4bef5d78089b7d171b6cd on apache:develop**.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] github-actions[bot] commented on pull request #4825: [ISSUE#4824] Support consumequeue, indexfile multiple directories storage mode

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #4825:
URL: https://github.com/apache/rocketmq/pull/4825#issuecomment-1718466250

   This PR was closed because it has been inactive for 3 days since being marked as stale.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] codecov-commenter commented on pull request #4825: [ISSUE#4824] Support consumequeue, indexfile multiple directories storage mode

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #4825:
URL: https://github.com/apache/rocketmq/pull/4825#issuecomment-1216097225

   # [Codecov](https://codecov.io/gh/apache/rocketmq/pull/4825?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#4825](https://codecov.io/gh/apache/rocketmq/pull/4825?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f52caee) into [develop](https://codecov.io/gh/apache/rocketmq/commit/8dd07c84697d3709f1f4bef5d78089b7d171b6cd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8dd07c8) will **decrease** coverage by `0.14%`.
   > The diff coverage is `47.89%`.
   
   ```diff
   @@              Coverage Diff              @@
   ##             develop    #4825      +/-   ##
   =============================================
   - Coverage      44.94%   44.80%   -0.15%     
   - Complexity      7640     7687      +47     
   =============================================
     Files            980      982       +2     
     Lines          68135    68563     +428     
     Branches        9014     9075      +61     
   =============================================
   + Hits           30622    30717      +95     
   - Misses         33729    34006     +277     
   - Partials        3784     3840      +56     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/rocketmq/pull/4825?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...cketmq/broker/offset/LmqConsumerOffsetManager.java](https://codecov.io/gh/apache/rocketmq/pull/4825/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvb2Zmc2V0L0xtcUNvbnN1bWVyT2Zmc2V0TWFuYWdlci5qYXZh) | `74.35% <0.00%> (-1.96%)` | :arrow_down: |
   | [...apache/rocketmq/store/timer/TimerMessageStore.java](https://codecov.io/gh/apache/rocketmq/pull/4825/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL3RpbWVyL1RpbWVyTWVzc2FnZVN0b3JlLmphdmE=) | `53.96% <0.00%> (-0.22%)` | :arrow_down: |
   | [...a/org/apache/rocketmq/broker/BrokerController.java](https://codecov.io/gh/apache/rocketmq/pull/4825/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvQnJva2VyQ29udHJvbGxlci5qYXZh) | `42.74% <8.10%> (-1.15%)` | :arrow_down: |
   | [...apache/rocketmq/store/queue/BatchConsumeQueue.java](https://codecov.io/gh/apache/rocketmq/pull/4825/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL3F1ZXVlL0JhdGNoQ29uc3VtZVF1ZXVlLmphdmE=) | `66.73% <9.09%> (-2.77%)` | :arrow_down: |
   | [...apache/rocketmq/broker/BrokerPathConfigHelper.java](https://codecov.io/gh/apache/rocketmq/pull/4825/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvQnJva2VyUGF0aENvbmZpZ0hlbHBlci5qYXZh) | `32.20% <18.18%> (-47.80%)` | :arrow_down: |
   | [...cketmq/broker/offset/ConsumerOrderInfoManager.java](https://codecov.io/gh/apache/rocketmq/pull/4825/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvb2Zmc2V0L0NvbnN1bWVyT3JkZXJJbmZvTWFuYWdlci5qYXZh) | `8.49% <33.33%> (+0.35%)` | :arrow_up: |
   | [...c/main/java/org/apache/rocketmq/common/MixAll.java](https://codecov.io/gh/apache/rocketmq/pull/4825/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jb21tb24vTWl4QWxsLmphdmE=) | `43.90% <37.50%> (-0.70%)` | :arrow_down: |
   | [...java/org/apache/rocketmq/common/ConfigManager.java](https://codecov.io/gh/apache/rocketmq/pull/4825/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jb21tb24vQ29uZmlnTWFuYWdlci5qYXZh) | `40.35% <40.90%> (-19.11%)` | :arrow_down: |
   | [...e/rocketmq/store/config/StorePathConfigHelper.java](https://codecov.io/gh/apache/rocketmq/pull/4825/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL2NvbmZpZy9TdG9yZVBhdGhDb25maWdIZWxwZXIuamF2YQ==) | `55.88% <52.17%> (-7.76%)` | :arrow_down: |
   | [...org/apache/rocketmq/store/DefaultMessageStore.java](https://codecov.io/gh/apache/rocketmq/pull/4825/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL0RlZmF1bHRNZXNzYWdlU3RvcmUuamF2YQ==) | `55.23% <54.43%> (-0.30%)` | :arrow_down: |
   | ... and [62 more](https://codecov.io/gh/apache/rocketmq/pull/4825/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] aaron-ai commented on pull request #4825: [ISSUE#4824] Support consumequeue, indexfile multiple directories storage mode

Posted by GitBox <gi...@apache.org>.
aaron-ai commented on PR #4825:
URL: https://github.com/apache/rocketmq/pull/4825#issuecomment-1228131238

   We intend to add CI pipelines for more platforms recently besides Linux, there are too many unit test could not pass in other platforms **especially Windows**.
   
   It would be awesome if you could concern more about cross-platform issues.
   
   By the way, if you don't have windows device for testing now, Github Action is a good choice. You can refer to https://github.com/aaron-ai/rocketmq/pull/1. I am pleased to provide support.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] shengminw commented on pull request #4825: [ISSUE#4824] Support consumequeue, indexfile multiple directories storage mode

Posted by GitBox <gi...@apache.org>.
shengminw commented on PR #4825:
URL: https://github.com/apache/rocketmq/pull/4825#issuecomment-1226828298

   > Can you fix the codes' conflicts firstly! @shengminw
   
   Sorry, this conflict was newly created when the develop branch was merged yesterday. I have fixed this conflict.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] aaron-ai commented on a diff in pull request #4825: [ISSUE#4824] Support consumequeue, indexfile multiple directories storage mode

Posted by GitBox <gi...@apache.org>.
aaron-ai commented on code in PR #4825:
URL: https://github.com/apache/rocketmq/pull/4825#discussion_r955713918


##########
broker/src/main/java/org/apache/rocketmq/broker/offset/ConsumerOrderInfoManager.java:
##########
@@ -266,6 +266,15 @@ public String configFilePath() {
         }
     }
 
+    @Override
+    public String configFileBakPath() {
+        if (brokerController != null) {
+            return BrokerPathConfigHelper.getConsumerOrderInfoBakPath(this.brokerController.getMessageStoreConfig().getStorePathRootDir());
+        } else {
+            return BrokerPathConfigHelper.getConsumerOrderInfoBakPath("~");

Review Comment:
   ditto



##########
broker/src/main/java/org/apache/rocketmq/broker/filter/ConsumerFilterManager.java:
##########
@@ -226,6 +226,16 @@ public String configFilePath() {
         return BrokerPathConfigHelper.getConsumerFilterPath("./unit_test");
     }
 
+    @Override
+    public String configFileBakPath() {
+        if (this.brokerController != null) {
+            return BrokerPathConfigHelper.getConsumerFilterBakPath(
+                    this.brokerController.getMessageStoreConfig().getStorePathRootDir()
+            );
+        }
+        return BrokerPathConfigHelper.getConsumerFilterBakPath("./unit_test");

Review Comment:
   is "./unit_test" here compatible with windows platform?



##########
broker/src/test/java/org/apache/rocketmq/broker/BrokerControllerTest.java:
##########
@@ -67,6 +71,65 @@ public void testBrokerRestart() throws Exception {
         brokerController.shutdown();
     }
 
+    @Test
+    public void testMultiPathStart() throws Exception {
+        MessageStoreConfig messageStoreConfig = new MessageStoreConfig();
+        messageStoreConfig.setStorePathRootDir(
+                "target/unit_test_store/a/store" + MixAll.MULTI_PATH_SPLITTER

Review Comment:
   ditto



##########
broker/src/test/java/org/apache/rocketmq/broker/BrokerControllerTest.java:
##########
@@ -67,6 +71,65 @@ public void testBrokerRestart() throws Exception {
         brokerController.shutdown();
     }
 
+    @Test
+    public void testMultiPathStart() throws Exception {
+        MessageStoreConfig messageStoreConfig = new MessageStoreConfig();
+        messageStoreConfig.setStorePathRootDir(
+                "target/unit_test_store/a/store" + MixAll.MULTI_PATH_SPLITTER
+                + "target/unit_test_store/b/store" + MixAll.MULTI_PATH_SPLITTER
+                + "target/unit_test_store/c/store"
+        );
+        BrokerController brokerController = new BrokerController(
+                new BrokerConfig(),
+                new NettyServerConfig(),
+                new NettyClientConfig(),
+                messageStoreConfig);
+        assertThat(brokerController.initialize()).isTrue();
+        brokerController.start();
+        brokerController.shutdown();
+        UtilAll.deleteFile(new File(new MessageStoreConfig().getStorePathRootDir()));
+    }
+
+    @Test
+    public void testMultiPathLoadConfig() {
+        MessageStoreConfig messageStoreConfig = new MessageStoreConfig();
+        messageStoreConfig.setStorePathRootDir(
+                "target/unit_test_store/a/store" + MixAll.MULTI_PATH_SPLITTER
+                        + "target/unit_test_store/b/store" + MixAll.MULTI_PATH_SPLITTER

Review Comment:
   ditto



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org