You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by lizhanhui <gi...@git.apache.org> on 2017/01/05 09:53:19 UTC
[GitHub] incubator-rocketmq pull request #28: [ROCKETMQ-27]Add store commitlog path o...
GitHub user lizhanhui opened a pull request:
https://github.com/apache/incubator-rocketmq/pull/28
[ROCKETMQ-27]Add store commitlog path online
RocketMQ broker stores all messages before they are expired, which requires pretty much storage for a high load system. For now, the message store can be configured to use only one partition. Quite often, we find broker runs out of storage as business expands quickly. Online resizing a partition most of time is not possible(with absence of LVM) whilst attaching a new disk/partition in cloud environment is super easy.
It would be nice for RocketMQ to be capable of adding extra commit log store path online, allocating commit log files onto partitions based on free/available space.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/lizhanhui/incubator-rocketmq add_store_commitlog_path_online
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-rocketmq/pull/28.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #28
----
commit c6d078e01941e2ab6e38c57855403e961359ed82
Author: Zhanhui Li <li...@apache.org>
Date: 2017-01-05T03:54:01Z
not yet persist commit log store path
commit 83c97849f7a06ff3c249824beb06bae543a7d6ec
Author: Zhanhui Li <li...@apache.org>
Date: 2017-01-05T04:57:20Z
Persist commit log store path after changing.
commit 21accafca95e69eba372fc6ed85fd87d583e870f
Author: Zhanhui Li <li...@apache.org>
Date: 2017-01-05T06:32:48Z
Fix code style.
commit c06de73239d377c187bff4cd6f152227c96eb011
Author: Zhanhui Li <li...@apache.org>
Date: 2017-01-05T06:37:32Z
Fix code style.
commit aedfc0d8d62b9b39a49a4e5a14ae6dc77297fc3f
Author: Zhanhui Li <li...@apache.org>
Date: 2017-01-05T07:35:02Z
Add command to mqadmin
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rocketmq issue #28: [ROCKETMQ-27]Add store commitlog path online
Posted by dongeforever <gi...@git.apache.org>.
Github user dongeforever commented on the issue:
https://github.com/apache/incubator-rocketmq/pull/28
@lizhanhui LGTM. Version 4.2.0 is going to be released now. It will be nice to contain this feature. Could you please add one IT Test to maintain the quality?
---
[GitHub] incubator-rocketmq issue #28: [ROCKETMQ-27]Add store commitlog path online
Posted by lizhanhui <gi...@git.apache.org>.
Github user lizhanhui commented on the issue:
https://github.com/apache/incubator-rocketmq/pull/28
@Jaskey @vsair No, Broker only delete expired files.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rocketmq issue #28: [ROCKETMQ-27]Add store commitlog path online
Posted by Jaskey <gi...@git.apache.org>.
Github user Jaskey commented on the issue:
https://github.com/apache/incubator-rocketmq/pull/28
Cool enhancement for this.
But just one thing to confirm, actually rocketmq will delete commit log when the disk space is ready to be full(85% by default) even though the commit log file is not expired, won't it @lizhanhui
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rocketmq issue #28: [ROCKETMQ-27]Add store commitlog path online
Posted by lizhanhui <gi...@git.apache.org>.
Github user lizhanhui commented on the issue:
https://github.com/apache/incubator-rocketmq/pull/28
OK.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rocketmq issue #28: [ROCKETMQ-27]Add store commitlog path online
Posted by vongosling <gi...@git.apache.org>.
Github user vongosling commented on the issue:
https://github.com/apache/incubator-rocketmq/pull/28
Cool, could we discuss this new feature in dev email list ? BTW, This branch has conflicts that must be resolved:-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rocketmq issue #28: [ROCKETMQ-27]Add store commitlog path online
Posted by vsair <gi...@git.apache.org>.
Github user vsair commented on the issue:
https://github.com/apache/incubator-rocketmq/pull/28
@Jaskey yeah, it's true that rocketmq will delete commit logs when disk is full, even though not expired.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rocketmq issue #28: [ROCKETMQ-27]Add store commitlog path online
Posted by zhouxinyu <gi...@git.apache.org>.
Github user zhouxinyu commented on the issue:
https://github.com/apache/incubator-rocketmq/pull/28
Hi, @lizhanhui , Let's handle this PR in RocketMQ-4.1.0 as a new feature.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rocketmq issue #28: [ROCKETMQ-27]Add store commitlog path online
Posted by Jaskey <gi...@git.apache.org>.
Github user Jaskey commented on the issue:
https://github.com/apache/incubator-rocketmq/pull/28
As far as I know, rocketmq will clean the unexpired files as long as space is 85% full and cleanFileForciblyEnable=true(true by default)
Please refer to the code snippet
```
public int deleteExpiredFileByTime(final long expiredTime,
final int deleteFilesInterval,
final long intervalForcibly,
final boolean cleanImmediately) {
Object[] mfs = this.copyMappedFiles(0);
if (null == mfs)
return 0;
int mfsLength = mfs.length - 1;
int deleteCount = 0;
List<MappedFile> files = new ArrayList<MappedFile>();
if (null != mfs) {
for (int i = 0; i < mfsLength; i++) {
MappedFile mappedFile = (MappedFile) mfs[i];
long liveMaxTimestamp = mappedFile.getLastModifiedTimestamp() + expiredTime;
if (System.currentTimeMillis() >= liveMaxTimestamp || cleanImmediately) { // here , it will clean the non expire files as long as cleanImmediately=true
if (mappedFile.destroy(intervalForcibly)) {
files.add(mappedFile);
deleteCount++;
if (files.size() >= DELETE_FILES_BATCH_MAX) {
break;
}
if (deleteFilesInterval > 0 && (i + 1) < mfsLength) {
try {
Thread.sleep(deleteFilesInterval);
} catch (InterruptedException e) {
}
}
} else {
break;
}
}
}
}
deleteExpiredFile(files);
return deleteCount;
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---