[jira] [Resolved] (KAFKA-1791) Corrupt index after safe shutdown and restart

["Jay Kreps (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jay Kreps resolved KAFKA-1791.
------------------------------
    Resolution: Cannot Reproduce

> Corrupt index after safe shutdown and restart
> ---------------------------------------------
>
>                 Key: KAFKA-1791
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1791
>             Project: Kafka
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 0.8.1
>         Environment: Debian6 with Sun-Java6
>            Reporter: Vamsi Subhash Achanta
>            Priority: Critical
>         Attachments: 00000000000000000233.index, 00000000000000000233.log
>
>
> We have 3 kafka brokers - all VMs. One of the broker was stopped for around 30 minutes to fix a problem with the bare metal. Upon restart, after some time, the broker went out of file-descriptors (FDs) and started throwing errors. Upon restart, it started throwing this corrupted index exceptions. I followed the other JIRAs related to corrupted indices but the solutions mentioned there (deleting the index and restart) didn't work - the index gets created again. The other JIRAs solution of deleting those indexes which got wrongly compacted (> 10MB) didn't work either.
> What is the error? How can I fix this and bring back the broker? Thanks.
> INFO [2014-11-21 02:57:17,510] [main][] kafka.log.LogManager - Found clean shutdown file. Skipping recovery for all logs in data directory '/var/lib/fk-3p-kafka/logs'
>  INFO [2014-11-21 02:57:17,510] [main][] kafka.log.LogManager - Loading log 'kf.production.b2b.return_order.status-25'
> FATAL [2014-11-21 02:57:17,533] [main][] kafka.server.KafkaServerStartable - Fatal error during KafkaServerStable startup. Prepare to shutdown
> java.lang.IllegalArgumentException: requirement failed: Corrupt index found, index file (/var/lib/fk-3p-kafka/logs/kf.production.b2b.return_order.status-25/00000000000000000233.index) has non-zero size but the last offset is 233 and the base offset is 233
> 	at scala.Predef$.require(Predef.scala:145)
> 	at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352)
> 	at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:159)
> 	at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:158)
> 	at scala.collection.Iterator$class.foreach(Iterator.scala:631)
> 	at scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:474)
> 	at scala.collection.IterableLike$class.foreach(IterableLike.scala:79)
> 	at scala.collection.JavaConversions$JCollectionWrapper.foreach(JavaConversions.scala:495)
> 	at kafka.log.Log.loadSegments(Log.scala:158)
> 	at kafka.log.Log.<init>(Log.scala:64)
> 	at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:118)
> 	at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:113)
> 	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
> 	at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:34)
> 	at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:113)
> 	at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:105)
> 	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
> 	at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:32)
> 	at kafka.log.LogManager.loadLogs(LogManager.scala:105)
> 	at kafka.log.LogManager.<init>(LogManager.scala:57)
> 	at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:275)
> 	at kafka.server.KafkaServer.startup(KafkaServer.scala:72)
> 	at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
> 	at kafka.Kafka$.main(Kafka.scala:46)
> 	at kafka.Kafka.main(Kafka.scala)
>  INFO [2014-11-21 02:57:17,534] [main][] kafka.server.KafkaServer - [Kafka Server 2], shutting down
>  INFO [2014-11-21 02:57:17,538] [main][] kafka.server.KafkaServer - [Kafka Server 2], shut down completed
>  INFO [2014-11-21 02:57:17,539] [Thread-2][] kafka.server.KafkaServer - [Kafka Server 2], shutting down



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)