You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2022/08/15 03:55:32 UTC
[GitHub] [rocketmq] shengminw opened a new issue, #4824: Support consumequeue, indexfile multiple directories storage mode
shengminw opened a new issue, #4824:
URL: https://github.com/apache/rocketmq/issues/4824
In RIP-7 https://github.com/apache/rocketmq/wiki/RIP-7-Multiple-Directories-Storage-Support, CommitLog File has supported multi-directories storage mode.
In this case, the commitlog can be stored on multiple disks, but consumequeue/index is still stored on one disk. However, the size of consumequeue/index can reach tens of Gbs, or even 100 Gbs, and the space used can't be easily ignored.
The current commitLog multipath needs to be configured independently, as follows:
```java
storePathRootDir=/rmqstore1
storePathCommitLog=/rmqstore1/commitlog:/rmqstore2/commitlog
```
Therefore, I optimized the multi-path mode, and the improvements are as follows:
● Add multi-path mode for consumequeue.
● Add multi-path mode for indexfile.
● Optimize the method to get the commitlog path. It is not necessary to specify storePathCommitLog separately, and generate the storePathCommitLog from storePathRootDir.
● Add a Timed thread to check path capacity. This will try to avoid writing to a nearly full disk.
● Loading metadata in multi-path. By default, the config file is stored in the first path, and the config bak file is stored in the second path. For example: storePathRootDir = /rmqstore0:/rmqstore1:/rmqstore2, config dir = /rmqstore0/config, config bak dir = /rmqstore1/config. At startup, scan all paths and load the config file from the latest path of dataversion.（This ensures that when the path is changed, the latest version can be loaded）
● Loading checkpoint file and abort file in multi-path. The loading path of the checkpoint file and the abort file is the same as that of the config file.
## Chinese Version / 中文版本
在RIP-7 https://github.com/apache/rocketmq/wiki/RIP-7-Multiple-Directories-Storage-Support 当中，CommitLog文件已经支持了多文件路径的存储模式。
在这个情况下，commitLog可以被存储至多块磁盘当中，但是consumequeue和index依然存储于单盘上。然而consumequeue 和 index的文件大小也可能达到数十G或者上百G的大小。这样的存储空间并不能被忽略。
当前的commitLog的多路径需要被额外配置，如下所示：
```java
storePathRootDir=/rmqstore1
storePathCommitLog=/rmqstore1/commitlog:/rmqstore2/commitlog
```
因此，我对多路径模式做了一定的优化，改进主要如下：
● consumequeue的存储增加多路径模式。
● indexfile的存储增加多路径模式。
● 优化commitlog获取文件路径的方式。不必要单独设置storePathCommitLog，storePathCommitLog可以从storePathRootDir根目录下直接生成。
● 增加一个定时线程检查路径的容量。这样能尽量避免写入至快满的磁盘中。
● 多路径下的元数据加载。默认将config文件夹写入至第1个路径中，config文件夹的bak文件写入至第2个路径。例如：storePathRootDir = /rmqstore0:/rmqstore1:/rmqstore2, config dir = /rmqstore0/config, config bak dir = /rmqstore1/config。在启动的时候，扫描所有路径，选择dataversion最新的所在路径作为初始加载路径。（这样能确保路径被更改时，能够从最新的路径下加载）
● 加载checkpoint文件和abort文件。checkpoint文件和abort文件的加载路径与config路径保持一致。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support consumequeue, indexfile multiple directories storage mode [rocketmq]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #4824: Support consumequeue, indexfile multiple directories storage mode
URL: https://github.com/apache/rocketmq/issues/4824
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] lizhanhui commented on issue #4824: Support consumequeue, indexfile multiple directories storage mode
Posted by GitBox <gi...@apache.org>.
lizhanhui commented on issue #4824:
URL: https://github.com/apache/rocketmq/issues/4824#issuecomment-1214671387
@shengminw
In your RIP:
> So the deletion should be executed when any of the disks reached the threshold (set by diskMaxUsedSpaceRatio)
Is this a good idea? Attempting to delete expiring data is pretty expensive, which may compete IOPS with normal data persistence, causing unexpected latency spikes.
Another issue: similar to https://github.com/apache/rocketmq/issues/4803 Watermarks measured in percent are inferior to those measured in kilobytes(megabytes) as Linux kernel does. The reclamation of RocketMQ should follow the same way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] shengminw commented on issue #4824: Support consumequeue, indexfile multiple directories storage mode
Posted by GitBox <gi...@apache.org>.
shengminw commented on issue #4824:
URL: https://github.com/apache/rocketmq/issues/4824#issuecomment-1214753169
@lizhanhui
Thanks for your suggestion, I think it is a good way that measuring in megabytes is better than the ratio.
But RIP-7 is not my job, I just make further improvements to the issue. This issue hopes to expand the multi-disk paths of consumequeue, indexfile and related files on the basis of RIP-7.
Maybe disk capacity measurement can be resolved in a new issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support consumequeue, indexfile multiple directories storage mode [rocketmq]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4824:
URL: https://github.com/apache/rocketmq/issues/4824#issuecomment-1951495099
This issue was closed because it has been inactive for 3 days since being marked as stale.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] lizhanhui commented on issue #4824: Support consumequeue, indexfile multiple directories storage mode
Posted by GitBox <gi...@apache.org>.
lizhanhui commented on issue #4824:
URL: https://github.com/apache/rocketmq/issues/4824#issuecomment-1214965524
@shengminw Okay, move on with your improvement...I thought you are building RIP-7.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support consumequeue, indexfile multiple directories storage mode [rocketmq]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4824:
URL: https://github.com/apache/rocketmq/issues/4824#issuecomment-1947536142
This issue is stale because it has been open for 365 days with no activity. It will be closed in 3 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org