[jira] [Updated] (KAFKA-2012) Broker should automatically handle corrupt index files

["Manikumar Reddy (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-2012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Manikumar Reddy updated KAFKA-2012:
-----------------------------------
    Attachment: KAFKA-2012.patch

> Broker should automatically handle corrupt index files
> ------------------------------------------------------
>
>                 Key: KAFKA-2012
>                 URL: https://issues.apache.org/jira/browse/KAFKA-2012
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.8.1.1
>            Reporter: Todd Palino
>         Attachments: KAFKA-2012.patch
>
>
> We had a bunch of unclean system shutdowns (power failure), which caused corruption on our disks holding log segments in many cases. While the broker is handling the log segment corruption properly (truncation), it is having problems with corruption in the index files. Additionally, this only seems to be happening on some startups (while we are upgrading).
> The broker should just do what I do when I hit a corrupt index file - remove it and rebuild it.
> 2015/03/09 17:58:53.873 FATAL [KafkaServerStartable] [main] [kafka-server] [] Fatal error during KafkaServerStartable startup. Prepare to shutdown
> java.lang.IllegalArgumentException: requirement failed: Corrupt index found, index file (/export/content/kafka/i001_caches/__consumer_offsets-39/00000000000000000000.index) has non-zero size but the last offset is -2121629628 and the base offset is 0
> 	at scala.Predef$.require(Predef.scala:233)
> 	at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352)
> 	at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:185)
> 	at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:184)
> 	at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> 	at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> 	at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> 	at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> 	at kafka.log.Log.loadSegments(Log.scala:184)
> 	at kafka.log.Log.<init>(Log.scala:82)
> 	at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:141)
> 	at kafka.utils.Utils$$anon$1.run(Utils.scala:54)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)