[jira] [Created] (KAFKA-12868) log cleaner failed with java.lang.ArrayIndexOutOfBoundsException: -2147483648

["Yuneng Xie (Jira)" <ji...@apache.org>]

Yuneng Xie created KAFKA-12868:
----------------------------------

             Summary: log cleaner failed with java.lang.ArrayIndexOutOfBoundsException: -2147483648
                 Key: KAFKA-12868
                 URL: https://issues.apache.org/jira/browse/KAFKA-12868
             Project: Kafka
          Issue Type: Bug
          Components: log cleaner
    Affects Versions: 2.4.0
            Reporter: Yuneng Xie


our broker spent too long loading the offset partition.

```

[2021-05-30 03:18:20,505] INFO [GroupMetadataManager brokerId=2] Finished loading offsets and group metadata from __consumer_offsets-47 in 1236029 milliseconds. (kafka.coordinator.group.GroupMetadataManager)

```

 

so i checked the partition data , it's too big 

```

106G /var/lib/kafka/data/__consumer_offsets-47

```

 

the retention time is 7 days

```

log.retention.hours=168

```

 

and i found error log about log cleaner:

```

[2021-04-10 00:28:24,507] INFO Cleaner 0: Beginning cleaning of log __consumer_offsets-47. (kafka.log.LogCleaner)

[2021-04-10 00:28:24,507] INFO Cleaner 0: Building offset map for __consumer_offsets-47... (kafka.log.LogCleaner)

[2021-04-10 00:28:24,526] INFO Cleaner 0: Building offset map for log __consumer_offsets-47 for 17 segments in offset range [1010757586, 1020967392). (kafka.log.LogCleaner)

[2021-04-10 00:29:22,669] WARN [kafka-log-cleaner-thread-0]: Unexpected exception thrown when cleaning log Log(dir=/var/lib/kafka/data/__consumer_offsets-47, topic=__consumer_offsets, partition=47, highWatermark=1021411007, lastStableOffset=1021411007, logStartOffset=0, logEndOffset=1021411007). Marking its partition (__consumer_offsets-47) as uncleanable (kafka.log.LogCleaner)

kafka.log.LogCleaningException: -2147483648

        at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:348)

        at kafka.log.LogCleaner$CleanerThread.tryCleanFilthiestLog(LogCleaner.scala:324)

        at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:313)

        at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)

Caused by: java.lang.ArrayIndexOutOfBoundsException: -2147483648

        at kafka.utils.CoreUtils$.readInt(CoreUtils.scala:241)

        at kafka.log.SkimpyOffsetMap.positionOf(OffsetMap.scala:183)

        at kafka.log.SkimpyOffsetMap.put(OffsetMap.scala:101)

        at kafka.log.Cleaner.$anonfun$buildOffsetMapForSegment$2(LogCleaner.scala:947)

        at kafka.log.Cleaner.$anonfun$buildOffsetMapForSegment$2$adapted(LogCleaner.scala:944)

        at scala.collection.Iterator.foreach(Iterator.scala:941)

        at scala.collection.Iterator.foreach$(Iterator.scala:941)

        at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)

        at scala.collection.IterableLike.foreach(IterableLike.scala:74)

        at scala.collection.IterableLike.foreach$(IterableLike.scala:73)

        at scala.collection.AbstractIterable.foreach(Iterable.scala:56)

        at kafka.log.Cleaner.$anonfun$buildOffsetMapForSegment$1(LogCleaner.scala:944)

        at kafka.log.Cleaner.$anonfun$buildOffsetMapForSegment$1$adapted(LogCleaner.scala:933)

        at scala.collection.Iterator.foreach(Iterator.scala:941)

        at scala.collection.Iterator.foreach$(Iterator.scala:941)

        at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)

        at scala.collection.IterableLike.foreach(IterableLike.scala:74)

        at scala.collection.IterableLike.foreach$(IterableLike.scala:73)

        at scala.collection.AbstractIterable.foreach(Iterable.scala:56)

        at kafka.log.Cleaner.buildOffsetMapForSegment(LogCleaner.scala:933)

        at kafka.log.Cleaner.$anonfun$buildOffsetMap$3(LogCleaner.scala:894)

        at kafka.log.Cleaner.$anonfun$buildOffsetMap$3$adapted(LogCleaner.scala:890)

        at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:877)

        at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)

        at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)

        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)

        at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:876)

        at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:890)

        at kafka.log.Cleaner.doClean(LogCleaner.scala:514)

        at kafka.log.Cleaner.clean(LogCleaner.scala:502)

        at kafka.log.LogCleaner$CleanerThread.cleanLog(LogCleaner.scala:371)

        at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:344)

        ... 3 more

```

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)