You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2022/07/03 03:47:34 UTC
[GitHub] [rocketmq] gentryhuang opened a new issue, #4549: A question about deleteExpiredFile function
gentryhuang opened a new issue, #4549:
URL: https://github.com/apache/rocketmq/issues/4549
**This method of deleteExpiredFile question:**
Messages pile up, and a disk rate threshold (such as 0.85 or 0.90) is immediately executed to delete files. This situation may result in unconsumed messages being deleted and messages being lost.
**Key codes are as follows:**
`private void deleteExpiredFiles() {
int deleteCount = 0;
long fileReservedTime = DefaultMessageStore.this.getMessageStoreConfig().getFileReservedTime();
int deletePhysicFilesInterval = DefaultMessageStore.this.getMessageStoreConfig().getDeleteCommitLogFilesInterval();
int destroyMapedFileIntervalForcibly = DefaultMessageStore.this.getMessageStoreConfig().getDestroyMapedFileIntervalForcibly();
boolean timeup = this.isTimeToDelete();
// core code
boolean spacefull = this.isSpaceToDelete();
boolean manualDelete = this.manualDeleteFileSeveralTimes > 0;
if (timeup || spacefull || manualDelete) {
if (manualDelete)
this.manualDeleteFileSeveralTimes--;
// core code
boolean cleanAtOnce = DefaultMessageStore.this.getMessageStoreConfig().isCleanFileForciblyEnable() && this.cleanImmediately;
log.info("begin to delete before {} hours file. timeup: {} spacefull: {} manualDeleteFileSeveralTimes: {} cleanAtOnce: {}",
fileReservedTime,
timeup,
spacefull,
manualDeleteFileSeveralTimes,
cleanAtOnce);
fileReservedTime *= 60 * 60 * 1000;
deleteCount = DefaultMessageStore.this.commitLog.deleteExpiredFile(
fileReservedTime,
deletePhysicFilesInterval,
destroyMapedFileIntervalForcibly,
cleanAtOnce);
if (deleteCount > 0) {
} else if (spacefull) {
log.warn("disk space will be full soon, but delete file failed.");
}
}
}`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] gentryhuang commented on issue #4549: A question about deleteExpiredFile function
Posted by GitBox <gi...@apache.org>.
gentryhuang commented on issue #4549:
URL: https://github.com/apache/rocketmq/issues/4549#issuecomment-1173383084
> @gentryhuang IMO, users should monitor disk usage before it reaches 0.85. RocketMQ set two level service degradation: when disk usage reaches 0.85, delete oldest files immediately to ease disk pressure for service available(RW); when disk usage reach 0.9, prevent writing for service available(R).
That is, a high disk usage may cause message loss. Do you need to ensure a reasonable disk usage to avoid message loss?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] lizhiboo commented on issue #4549: A question about deleteExpiredFile function
Posted by GitBox <gi...@apache.org>.
lizhiboo commented on issue #4549:
URL: https://github.com/apache/rocketmq/issues/4549#issuecomment-1173213410
@gentryhuang IMO, users should monitor disk usage before it reaches 0.85. RocketMQ set two level service degradation: when disk usage reaches 0.85, delete oldest files immediately to ease disk pressure for service available(RW); when disk usage reach 0.9, prevent writing for service available(R).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] lizhiboo commented on issue #4549: A question about deleteExpiredFile function
Posted by GitBox <gi...@apache.org>.
lizhiboo commented on issue #4549:
URL: https://github.com/apache/rocketmq/issues/4549#issuecomment-1174502208
> > @gentryhuang IMO, users should monitor disk usage before it reaches 0.85. RocketMQ set two level service degradation: when disk usage reaches 0.85, delete oldest files immediately to ease disk pressure for service available(RW); when disk usage reach 0.9, prevent writing for service available(R).
>
> That is, a high disk usage may cause message loss. Do you need to ensure a reasonable disk usage to avoid message loss?
it's hard to find the most suitable threshold, if needed, user can set these two threshold in broker.conf depending on business characteristics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] gentryhuang commented on issue #4549: A question about deleteExpiredFile function
Posted by GitBox <gi...@apache.org>.
gentryhuang commented on issue #4549:
URL: https://github.com/apache/rocketmq/issues/4549#issuecomment-1174554937
> > > @gentryhuang IMO, users should monitor disk usage before it reaches 0.85. RocketMQ set two level service degradation: when disk usage reaches 0.85, delete oldest files immediately to ease disk pressure for service available(RW); when disk usage reach 0.9, prevent writing for service available(R).
> >
> >
> > That is, a high disk usage may cause message loss. Do you need to ensure a reasonable disk usage to avoid message loss?
>
> it's hard to find the most suitable threshold, if needed, user can set these two threshold in broker.conf depending on business characteristics.
okļ¼thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] gentryhuang closed issue #4549: A question about deleteExpiredFile function
Posted by GitBox <gi...@apache.org>.
gentryhuang closed issue #4549: A question about deleteExpiredFile function
URL: https://github.com/apache/rocketmq/issues/4549
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org