[jira] [Created] (KAFKA-3917) Some __consumer_offsets replicas grow way too big

["Maxim Vladimirskiy (JIRA)" <ji...@apache.org>]

Maxim Vladimirskiy created KAFKA-3917:
-----------------------------------------

             Summary: Some __consumer_offsets replicas grow way too big
                 Key: KAFKA-3917
                 URL: https://issues.apache.org/jira/browse/KAFKA-3917
             Project: Kafka
          Issue Type: Bug
          Components: core
    Affects Versions: 0.8.2.2
         Environment: Runs with Docker 1.10.1 in a container on 
Linux 3.13.0-77-generic #121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016 x86_64
            Reporter: Maxim Vladimirskiy


We noticed that some replicas of partitions of the __consumer_offsets topic grow way too big. Looking inside respective folders it became apparent that old segments had not been cleaned up. Please see below example of disk usage data for both affected and not affected partitions:

Not affected partitions:
    Partition: 0	Leader: 2	Replicas: 2,3,4	Isr: 2,4,3
        2: 49M
        3: 49M
        4: 49M

Affected partitions:
    Partition: 10	Leader: 2	Replicas: 2,0,1	Isr: 1,2,0
        0: 86M
        1: 22G <<< too big!
        2: 86M
    Partition: 38	Leader: 0	Replicas: 0,4,1	Isr: 1,0,4
        0: 43M
        1: 26G <<<  too big!
        4: 26G <<<  too big!

As you can see sometimes only one replica is affected, sometimes both replicas are affected.

When I try to restart a broker that has affected replicas it fails to start with an exception that looks like this:

[2016-06-28 23:15:20,441] ERROR There was an error in one of the threads during logs loading: java.lang.IllegalArgumentException: requirement failed: Corrupt index found, index file (/var/kafka/__consumer_offsets-38/00000000000000000000.index) has non-zero size but the last offset is -676703869 and the base offset is 0 (kafka.log.LogManager)
[2016-06-28 23:15:20,442] FATAL [Kafka Server 1], Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.lang.IllegalArgumentException: requirement failed: Corrupt index found, index file (/var/kafka/__consumer_offsets-38/00000000000000000000.index) has non-zero size but the last offset is -676703869 and the base offset is 0
        at scala.Predef$.require(Predef.scala:233)
        at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352)
        at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:184)
        at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:183)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at kafka.log.Log.loadSegments(Log.scala:183)
        at kafka.log.Log.<init>(Log.scala:67)
        at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:142)
        at kafka.utils.Utils$$anon$1.run(Utils.scala:54)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

After the content of the affected partition is deleted broker starts successfully. 
 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)