[jira] [Created] (KAFKA-2012) Broker should automatically handle corrupt index files

["Todd Palino (JIRA)" <ji...@apache.org>]

Todd Palino created KAFKA-2012:
----------------------------------

             Summary: Broker should automatically handle corrupt index files
                 Key: KAFKA-2012
                 URL: https://issues.apache.org/jira/browse/KAFKA-2012
             Project: Kafka
          Issue Type: Bug
    Affects Versions: 0.8.1.1
            Reporter: Todd Palino


We had a bunch of unclean system shutdowns (power failure), which caused corruption on our disks holding log segments in many cases. While the broker is handling the log segment corruption properly (truncation), it is having problems with corruption in the index files. Additionally, this only seems to be happening on some startups (while we are upgrading).

The broker should just do what I do when I hit a corrupt index file - remove it and rebuild it.

2015/03/09 17:58:53.873 FATAL [KafkaServerStartable] [main] [kafka-server] [] Fatal error during KafkaServerStartable startup. Prepare to shutdown
java.lang.IllegalArgumentException: requirement failed: Corrupt index found, index file (/export/content/kafka/i001_caches/__consumer_offsets-39/00000000000000000000.index) has non-zero size but the last offset is -2121629628 and the base offset is 0
	at scala.Predef$.require(Predef.scala:233)
	at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352)
	at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:185)
	at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:184)
	at scala.collection.Iterator$class.foreach(Iterator.scala:727)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
	at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
	at kafka.log.Log.loadSegments(Log.scala:184)
	at kafka.log.Log.<init>(Log.scala:82)
	at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:141)
	at kafka.utils.Utils$$anon$1.run(Utils.scala:54)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)