LogCleaner keeps on stopping due to segment overflow exception

[elvinacamae <el...@yahoo.com.INVALID>]

Hello Kafka Users,We have an issue where the compacted topics are actually not being compacted and the logs are getting bigger and bigger. Turns out the LogCleaner has not been working and even if we restart the brokers the same issue appears on the LogCleaner.This is from the logs: [2021-02-05 02:48:14,287] INFO Cleaner 0: Caught segment overflow error during cleaning: Detected offset overflow at offset -1 in segment LogSegment(baseOffset=0, size=63468) (kafka.log.LogCleaner) [2021-02-05 02:48:14,327] ERROR [kafka-log-cleaner-thread-0]: Error due to (kafka.log.LogCleaner) java.lang.IllegalArgumentException: requirement failed: Split operation is only permitted for segments with overflow         at scala.Predef$.require(Predef.scala:224)         at kafka.log.Log.splitOverflowedSegment(Log.scala:1873)         at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:517)                 at kafka.log.Cleaner$$anonfun$doClean$4.apply(LogCleaner.scala:465)         at kafka.log.Cleaner$$anonfun$doClean$4.apply(LogCleaner.scala:464)         at scala.collection.immutable.List.foreach(List.scala:392)         at kafka.log.Cleaner.doClean(LogCleaner.scala:464)         at kafka.log.Cleaner.clean(LogCleaner.scala:442)         at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:303)         at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:289)         at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82) [2021-02-05 02:48:14,335] INFO [kafka-log-cleaner-thread-0]: Stopped (kafka.log.LogCleaner)I found this link on google where the user reported the same issue. I didn't find any jira tickets around the issue though. http://mail-archives.apache.org/mod_mbox/kafka-users/201902.mbox/%3C8a69819cce6d44ccaf694d1cd3b2c7a7@kasasi.de%3EI copied her in case she has resolved the issue and have any ideas to share :) I took a look at the last offset on the "problematic" segment but it doesn't look like it would cause an overflow given that baseOffset is 0.  sh-4.2$ /opt/confluent/bin/kafka-dump-log --files 00000000000000391241.log | tail -n1 baseOffset: 637828 lastOffset: 637832 count: 5 baseSequence: 55223 lastSequence: 55227 producerId: 36000 producerEpoch: 0 partitionLeaderEpoch: 48 isTransactional: false position: 1073555470 CreateTime: 1612466735652 isvalid: true size: 151936 magic: 2 compresscodec: NONE crc: 408829424Another thing that confuses me is that how come after identifying that there is an offset overflow the LogCleaner throws another exception due to java.lang.IllegalArgumentException: requirement failed: Split operation is only permitted for segments with overflow?I hope you can share some insight or guidance on the matter.Best Regards,Camille