[jira] [Commented] (KAFKA-1112) broker can not start itself after kafka is killed with -9

["Alex Tian (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14276497#comment-14276497 ] 

Alex Tian commented on KAFKA-1112:
----------------------------------

I also met similar issues in Kafka-2.9.2-0.8.1 ( after an uncleanShutDown --- kill -9. ). 
I am afraid that I cannot provide more logs since it was months ago: 
******************************************************************************************************************************
FATAL Fatal error during KafkaServerStable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
java.lang.IllegalArgumentException: requirement failed: Corrupt index found, index file (/home/storm/kafka_2.9.2-0.8.1/logs/tracker-1/00000000000006911940.index) has non-zero size but the last offset is 6911940 and the base offset is 6911940
******************************************************************************************************************************
Here is my analysis:
Once unclean shutdown, Kafka needs to rebuild index files for her logsegments when restarted.
If InvalidOffsetException is thrown when appending entries to an index file,    recoverLog()  function will handle it by truncating the log segment to baseOffset (==> lastIndexEntry == baseOffset ). 
However, since the index files are written by mmap, OS will flush the update to disks in time. It means the index file already has some entries now.
I am a beginner for scala, just guessing the logSegment with InvalidOffsetException thrown has been passed by the iterator, so it was not deleted in fact? (I mean all log segments after it will be deleted in recoverLog() in log.scala. )
This log segment missed to delete could not pass sanityCheck(), since it has index file with non-zero size but lastIndexEntry == baseOffset ( it was truncated to baseOffset when handling the InvalidOffsetException. ).
Quite sorry if any mistake above.


> broker can not start itself after kafka is killed with -9
> ---------------------------------------------------------
>
>                 Key: KAFKA-1112
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1112
>             Project: Kafka
>          Issue Type: Bug
>          Components: log
>    Affects Versions: 0.8.0, 0.8.1
>            Reporter: Kane Kim
>            Assignee: Jay Kreps
>            Priority: Critical
>             Fix For: 0.8.1
>
>         Attachments: KAFKA-1112-v1.patch, KAFKA-1112-v2.patch, KAFKA-1112-v3.patch, KAFKA-1112-v4.patch, KAFKA-1112.out
>
>
> When I kill kafka with -9, broker cannot start itself because of corrupted index logs. I think kafka should try to delete/rebuild indexes itself without manual intervention. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)