You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Jay Kreps (JIRA)" <ji...@apache.org> on 2013/07/18 22:10:48 UTC
[jira] [Commented] (KAFKA-980) Crash during log recovery can cause
full recovery to never run
[ https://issues.apache.org/jira/browse/KAFKA-980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13712740#comment-13712740 ]
Jay Kreps commented on KAFKA-980:
---------------------------------
Yeah this is a bug. You put down that it effects 0.8, where you able to reproduce it on 0.8?
> Crash during log recovery can cause full recovery to never run
> --------------------------------------------------------------
>
> Key: KAFKA-980
> URL: https://issues.apache.org/jira/browse/KAFKA-980
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8, 0.7.1
> Reporter: Blake Smith
>
> After an unclean shutdown of the Kafka server, if the broker throws an unhandled exception during log recovery, the broker can get in a state where recovery never runs on a log file.
> We saw this problem manifest in production and is summarized on the mailing list here: http://mail-archives.apache.org/mod_mbox/kafka-users/201307.mbox/%3CCAKSpikjgp2sW2ycuf86JrjtAPxWBp92OOEmigVed=u=JFoPvTA@mail.gmail.com%3E
> Because recovery state is not tracked explicitly, our kafka broker started writing data even when the log files were not fully recovered. It feels to me like a separate state flag for recovery should also be tracked in cases where recovery does not fully run. What do you guys think?
> Steps to reproduce:
> 1. Shutdown the kafka broker
> 2. Create a directory named 'bogus' under the kafka log directory (won't parse since it has no partition number)
> 3. Remove .kafka_cleanshutdown from the log directory to force a recovery
> 4. Start the kafka broker, observe:
> - Recovery will run on partition segments until it reaches the bogus directory
> - Exception will be thrown during log loading from the bogus directory
> - Kafka will initiate a clean shutdown after the exception is thrown
> 5. Once the Kafka server is cleanly shutdown, start it again, observe:
> - Recovery will not try to run, since kafka was shutdown cleanly
> - Some partition log files have never been recovered
> 6. Remove the bogus log directory
> 7. Start Kafka broker, observe:
> - Recovery will not run
> - Kafka will start cleanly and begin accepting writes again, even though recovery has never run and logs might be in a corrupt state
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira