[jira] [Comment Edited] (KAFKA-2012) Broker should automatically handle corrupt index files

["Mayuresh Gharat (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-2012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14593812#comment-14593812 ] 

Mayuresh Gharat edited comment on KAFKA-2012 at 6/19/15 7:30 PM:
-----------------------------------------------------------------

Discussed this with [~jjkoshy]. This patch seems like a workaround and does not actually tell us why the file got corrupted in first place. We can probably have a config that can turn this code path ON or OFF, so that we can actually investigate when this happens. 
Let me know, I can open another ticket or use this : https://issues.apache.org/jira/browse/KAFKA-1554 to add that config.

This was discussed in KAFKA-1554 :

Joel Koshy added a comment - 14/Mar/15 01:10
That would be a work-around, but ideally we should figure out why it happened in the first place.

 Jun Rao added a comment - 09/Apr/15 02:06
Yes, I am not sure if auto fixing the index is better. People then may not realize if there is an issue. It would be better to figure out what's causing this.


Thanks,

Mayuresh



was (Author: mgharat):
Discussed this with [~jjkoshy]. This patch seems like a workaround and does not actually tell us why the file got corrupted in first place. We can probably have a config that can turn this code path ON or OFF, so that we can actually investigate when this happens. 
Let me know, I can open another ticket or use this : https://issues.apache.org/jira/browse/KAFKA-1554 to add that config.

Thanks,

Mayuresh


> Broker should automatically handle corrupt index files
> ------------------------------------------------------
>
>                 Key: KAFKA-2012
>                 URL: https://issues.apache.org/jira/browse/KAFKA-2012
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.8.1.1
>            Reporter: Todd Palino
>            Assignee: Manikumar Reddy
>             Fix For: 0.8.3
>
>         Attachments: KAFKA-2012.patch, KAFKA-2012_2015-06-19_18:55:11.patch, KAFKA-2012_2015-06-19_21:09:22.patch
>
>
> We had a bunch of unclean system shutdowns (power failure), which caused corruption on our disks holding log segments in many cases. While the broker is handling the log segment corruption properly (truncation), it is having problems with corruption in the index files. Additionally, this only seems to be happening on some startups (while we are upgrading).
> The broker should just do what I do when I hit a corrupt index file - remove it and rebuild it.
> 2015/03/09 17:58:53.873 FATAL [KafkaServerStartable] [main] [kafka-server] [] Fatal error during KafkaServerStartable startup. Prepare to shutdown
> java.lang.IllegalArgumentException: requirement failed: Corrupt index found, index file (/export/content/kafka/i001_caches/__consumer_offsets-39/00000000000000000000.index) has non-zero size but the last offset is -2121629628 and the base offset is 0
> 	at scala.Predef$.require(Predef.scala:233)
> 	at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352)
> 	at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:185)
> 	at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:184)
> 	at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> 	at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> 	at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> 	at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> 	at kafka.log.Log.loadSegments(Log.scala:184)
> 	at kafka.log.Log.<init>(Log.scala:82)
> 	at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:141)
> 	at kafka.utils.Utils$$anon$1.run(Utils.scala:54)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)