[jira] [Resolved] (KAFKA-3917) Some __consumer_offsets replicas grow way too big

["Ismael Juma (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-3917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ismael Juma resolved KAFKA-3917.
--------------------------------
    Resolution: Duplicate

This log cleaner issue has been fixed in more recent versions. I suggest upgrading.

> Some __consumer_offsets replicas grow way too big
> -------------------------------------------------
>
>                 Key: KAFKA-3917
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3917
>             Project: Kafka
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 0.8.2.2
>         Environment: Runs with Docker 1.10.1 in a container on 
> Linux 3.13.0-77-generic #121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016 x86_64
>            Reporter: Maxim Vladimirskiy
>              Labels: reliability
>
> We noticed that some replicas of partitions of the __consumer_offsets topic grow way too big. Looking inside respective folders it became apparent that old segments had not been cleaned up. Please see below example of disk usage data for both affected and not affected partitions:
> Not affected partitions:
>     Partition: 0	Leader: 2	Replicas: 2,3,4	Isr: 2,4,3
>         2: 49M
>         3: 49M
>         4: 49M
> Affected partitions:
>     Partition: 10	Leader: 2	Replicas: 2,0,1	Isr: 1,2,0
>         0: 86M
>         1: 22G <<< too big!
>         2: 86M
>     Partition: 38	Leader: 0	Replicas: 0,4,1	Isr: 1,0,4
>         0: 43M
>         1: 26G <<<  too big!
>         4: 26G <<<  too big!
> As you can see sometimes only one replica is affected, sometimes both replicas are affected.
> When I try to restart a broker that has affected replicas it fails to start with an exception that looks like this:
> [2016-06-28 23:15:20,441] ERROR There was an error in one of the threads during logs loading: java.lang.IllegalArgumentException: requirement failed: Corrupt index found, index file (/var/kafka/__consumer_offsets-38/00000000000000000000.index) has non-zero size but the last offset is -676703869 and the base offset is 0 (kafka.log.LogManager)
> [2016-06-28 23:15:20,442] FATAL [Kafka Server 1], Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
> java.lang.IllegalArgumentException: requirement failed: Corrupt index found, index file (/var/kafka/__consumer_offsets-38/00000000000000000000.index) has non-zero size but the last offset is -676703869 and the base offset is 0
>         at scala.Predef$.require(Predef.scala:233)
>         at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352)
>         at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:184)
>         at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:183)
>         at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>         at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
>         at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>         at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>         at kafka.log.Log.loadSegments(Log.scala:183)
>         at kafka.log.Log.<init>(Log.scala:67)
>         at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:142)
>         at kafka.utils.Utils$$anon$1.run(Utils.scala:54)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> After the content of the affected partition is deleted broker starts successfully. 
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)