You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2021/08/12 15:51:46 UTC
[GitHub] [rocketmq] schnell18 opened a new issue #3257: Ignore irrelevant commit log files on startup
schnell18 opened a new issue #3257:
URL: https://github.com/apache/rocketmq/issues/3257
**BUG REPORT**
1. Please describe the issue you observed:
- What did you do?
Create an empty file under $storePathCommitLog directory. Say, you have commit log store under /data0/rocketmq/commitlogs/,
then you create a file named 'the-bad-empty-file-fails-rocketmq-startup".
Restart broker.
Then the broker will not be able to startup normal.
You may see error message like "the-bad-empty-file-fails-rocketmq-startup 0 length not matched message store value. please check it manually" in the storeerror.log.
- What did you expect to see?
Even in this case, RocketMQ broker should startup normally and log a warning message.
- What did you see instead?
RocketMQ broker does not startup and does not explain the reason clearly.
2. Please tell us about your environment:
It's an ordinary CentOS 7.5 running Intel x86 chip on bare metal.
3. Other information (e.g. detailed explanation, logs, related issues, suggestions how to fix, etc):
This suggestion is to ignore file length is not same as mappedFileSize which is 1G by default. Log a warning and continue to start.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] schnell18 edited a comment on issue #3257: Ignore irrelevant commit log files on startup
Posted by GitBox <gi...@apache.org>.
schnell18 edited a comment on issue #3257:
URL: https://github.com/apache/rocketmq/issues/3257#issuecomment-898884407
Thank you very much for quick reponse. We had this problem on a 4.6.0 production environment and I also reproduced this problem on 4.8.0. It seems to me this issue applies to code on master branch as well.
We lost quite a few data and spent days to recover due to this problem. The failure to start the broker w/ empty files under commit log directory surprised and confused us because other similar brokers started normally. Then our admin created an empty commit log directory and make the broker use that. The broker starts up finally. However, almost all our application topics were lost. It took us a few days to recover messy data caused by this accident.
RocketMQ is quite reliable and it is definitely unlikely to corrupt the commit log by itself.
And commit log file names are 20-digit w/ zero padding. There is no reason to make big fuss about irrelevant files, the 'the-bad-empty-file-fails-rocketmq-startup' in this case, and fails the startup. It makes perfect sense to igore files does not match the commit log name pattern and only fail on real commit log corruption.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] schnell18 commented on issue #3257: Ignore irrelevant commit log files on startup
Posted by GitBox <gi...@apache.org>.
schnell18 commented on issue #3257:
URL: https://github.com/apache/rocketmq/issues/3257#issuecomment-898884407
Thank you very much for quick reponse. We had this problem on a 4.6.0 production environment and I also reproduced this problem on 4.8.0. I also looked at the code on master branch. It has same issue.
This problem caused our application lost data due to failure to start the broker w/ empty files under commit log directory. Then the not-so-well-informed admin created an empty commit log directory and make the broker use that. The broker starts up finally. However, almost all our application topics were lost. It took us a few days to recover the messy data caused by this accident.
RocketMQ is quite reliable and it is definitely unlikely to corrupt the commit log by itself.
And commit log file names are 20-digit w/ zero padding. There is no reason to consider irrelevant files, for instance 'the-bad-empty-file-fails-rocketmq-startup', as legitimate commit log and fails the startup. It makes sensible to igore files does not match the commit log name pattern and only fail on real commit log corruption.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] francisoliverlee commented on issue #3257: Ignore irrelevant commit log files on startup
Posted by GitBox <gi...@apache.org>.
francisoliverlee commented on issue #3257:
URL: https://github.com/apache/rocketmq/issues/3257#issuecomment-897773215
checked it on 4.8.0 , you are right. but data file not right is a FATAL ERROR, manually-check needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] schnell18 edited a comment on issue #3257: Ignore irrelevant commit log files on startup
Posted by GitBox <gi...@apache.org>.
schnell18 edited a comment on issue #3257:
URL: https://github.com/apache/rocketmq/issues/3257#issuecomment-898884407
Thank you very much for quick reponse. We had this problem on a 4.6.0 production environment and I also reproduced this problem on 4.8.0. I also looked at the code on master branch. It has same issue.
We lost quite a few data and spent days to recover due to this problem. The failure to start the broker w/ empty files under commit log directory surprised and confused us because other similar brokers started normally. Then our admin created an empty commit log directory and make the broker use that. The broker starts up finally. However, almost all our application topics were lost. It took us a few days to recover messy data caused by this accident.
RocketMQ is quite reliable and it is definitely unlikely to corrupt the commit log by itself.
And commit log file names are 20-digit w/ zero padding. There is no reason to make big fuss about irrelevant files, the 'the-bad-empty-file-fails-rocketmq-startup' in this case, and fails the startup. It makes perfect sense to igore files does not match the commit log name pattern and only fail on real commit log corruption.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] schnell18 edited a comment on issue #3257: Ignore irrelevant commit log files on startup
Posted by GitBox <gi...@apache.org>.
schnell18 edited a comment on issue #3257:
URL: https://github.com/apache/rocketmq/issues/3257#issuecomment-898884407
Thank you very much for quick reponse. We had this problem on a 4.6.0 production environment and I also reproduced this problem on 4.8.0. It seems to me this issue applies to code on master branch as well.
We lost quite a few data and spent days to recover due to this problem. The failure to start the broker w/ empty files under commit log directory surprised and confused us quite a lot. And we didn't see same problem on brokers in the same cluster and it WORKED previously. Then our admin created an empty commit log directory and made the broker use that. The broker started successfully at last. However, almost all our application topics were lost. It took us a few days to recover messy data caused by this accident.
RocketMQ is quite reliable and it is definitely unlikely to corrupt the commit log by itself.
And commit log file names are 20-digit w/ zero padding. There is no reason to make big fuss about irrelevant files, the 'the-bad-empty-file-fails-rocketmq-startup' in this case, and fails the startup. It makes perfect sense to igore files does not match the commit log name pattern and only fail on real commit log corruption.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] schnell18 edited a comment on issue #3257: Ignore irrelevant commit log files on startup
Posted by GitBox <gi...@apache.org>.
schnell18 edited a comment on issue #3257:
URL: https://github.com/apache/rocketmq/issues/3257#issuecomment-898884407
Thank you very much for quick reponse. We had this problem on a 4.6.0 production environment and I also reproduced this problem on 4.8.0. I also looked at the code on master branch. It has same issue.
We lost quite a few data and spent days to recover due to this problem. The failure to start the broker w/ empty files under commit log directory surprised and confused us because other similar brokers started normally. Then our admin created an empty commit log directory and make the broker use that. The broker starts up finally. However, almost all our application topics were lost. It took us a few days to recover messy data caused by this accident.
RocketMQ is quite reliable and it is definitely unlikely to corrupt the commit log by itself.
And commit log file names are 20-digit w/ zero padding. There is no reason to consider irrelevant files, for instance 'the-bad-empty-file-fails-rocketmq-startup', as legitimate commit log and fails the startup. It makes sensible to igore files does not match the commit log name pattern and only fail on real commit log corruption.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org