You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Luke Chen (Jira)" <ji...@apache.org> on 2023/05/04 10:12:00 UTC
[jira] [Comment Edited] (KAFKA-14817) LogCleaner mark some partitions of __consumer_offsets as uncleanable
[ https://issues.apache.org/jira/browse/KAFKA-14817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17719250#comment-17719250 ]
Luke Chen edited comment on KAFKA-14817 at 5/4/23 10:11 AM:
------------------------------------------------------------
[~polaris.alioth] [~mrMigles] , thanks for reporting this issue. The error means we cannot recognize the batch in the log position (maybe corrupted), so that we cannot proceed with compaction for this partition logs. Like pointed out in this article [https://luppeng.wordpress.com/2022/08/21/possible-reasons-why-a-kafka-topic-is-not-being-compacted/] (point 3), letting Kafka do log recovery by itself should be able to fix it.
About how we can fix this permanently, I have to think about it.
But before that, if it's possible to help verify if the log segments appeared in the log is indeed corrupted?
You can verify it by running this command to dump the uncleanable logs?:
{code:java}
./bin/kafka-dump-log.sh --files /tmp/kraft-combined-logs/quickstart-events-0/00000000000000000000.log{code}
Of course, if it's allowed, it'll be better you upload the uncleanable logs for me to investigation. But I can understand if it's not permitted.
Thanks.
was (Author: showuon):
[~polaris.alioth] [~mrMigles] , thanks for reporting this issue. The error means we cannot recognize the batch in the log position (maybe corrupted), so that we cannot proceed with compaction for this partition logs. Like [~mrMigles] pointed out, letting Kafka do log recovery by itself should be able to fix it.
About how we can fix this permanently, I have to think about it.
But before that, if it's possible to help verify if the log segments appeared in the log is indeed corrupted?
You can verify it by running this command to dump the uncleanable logs?:
{code:java}
./bin/kafka-dump-log.sh --files /tmp/kraft-combined-logs/quickstart-events-0/00000000000000000000.log{code}
Of course, if it's allowed, it'll be better you upload the uncleanable logs for me to investigation. But I can understand if it's not permitted.
Thanks.
> LogCleaner mark some partitions of __consumer_offsets as uncleanable
> --------------------------------------------------------------------
>
> Key: KAFKA-14817
> URL: https://issues.apache.org/jira/browse/KAFKA-14817
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 2.8.1
> Reporter: ZhenChun Pan
> Priority: Major
>
> We find some patitions of topic __consumer_offsets can't retention it's log any more and takes up a lot of disk space. Then we found these patitions of topic __consumer_offsets have to mark as uncleanable in log-cleaner.log. The logs below:
> [2023-03-17 17:53:46,655] INFO Starting the log cleaner (kafka.log.LogCleaner)
> [2023-03-17 17:53:46,770] INFO [kafka-log-cleaner-thread-0]: Starting (kafka.log.LogCleaner)
> [2023-03-17 17:53:46,841] INFO Cleaner 0: Beginning cleaning of log __consumer_offsets-24. (kafka.log.LogCleaner)
> [2023-03-17 17:53:46,841] INFO Cleaner 0: Building offset map for __consumer_offsets-24... (kafka.log.LogCleaner)
> [2023-03-17 17:53:47,013] INFO Cleaner 0: Building offset map for log __consumer_offsets-24 for 5 segments in offset range [0, 2360519). (kafka.log.LogCleaner)
> [2023-03-17 17:53:47,394] INFO Cleaner 0: Growing cleaner I/O buffers from 262144 bytes to 524288 bytes. (kafka.log.LogCleaner)
> [2023-03-17 17:53:47,395] INFO Cleaner 0: Growing cleaner I/O buffers from 524288 bytes to 1048576 bytes. (kafka.log.LogCleaner)
> [2023-03-17 17:53:47,396] INFO Cleaner 0: Growing cleaner I/O buffers from 1048576 bytes to 2097152 bytes. (kafka.log.LogCleaner)
> [2023-03-17 17:53:47,401] INFO Cleaner 0: Growing cleaner I/O buffers from 2097152 bytes to 4194304 bytes. (kafka.log.LogCleaner)
> [2023-03-17 17:53:47,409] INFO Cleaner 0: Growing cleaner I/O buffers from 4194304 bytes to 8388608 bytes. (kafka.log.LogCleaner)
> [2023-03-17 17:53:47,434] INFO Cleaner 0: Growing cleaner I/O buffers from 8388608 bytes to 10485772 bytes. (kafka.log.LogCleaner)
> [2023-03-17 17:53:47,465] WARN [kafka-log-cleaner-thread-0]: Unexpected exception thrown when cleaning log Log(dir=/opt/kafka-service/data/__consumer_offsets-24, topic=__consumer_offsets, partition=24, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=2759760). Marking its partition (__consumer_offsets-24) as uncleanable (kafka.log.LogCleaner)
> kafka.log.LogCleaningException: Batch size 223 < buffer size 10485772, but not processed for log segment /opt/kafka-service/data/__consumer_offsets-24/00000000000000000000.log at position 31457091
> at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:356)
> at kafka.log.LogCleaner$CleanerThread.tryCleanFilthiestLog(LogCleaner.scala:332)
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:321)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
> Caused by: java.lang.IllegalStateException: Batch size 223 < buffer size 10485772, but not processed for log segment /opt/kafka-service/data/__consumer_offsets-24/00000000000000000000.log at position 31457091
> at kafka.log.Cleaner.growBuffersOrFail(LogCleaner.scala:745)
> at kafka.log.Cleaner.buildOffsetMapForSegment(LogCleaner.scala:983)
> at kafka.log.Cleaner.$anonfun$buildOffsetMap$5(LogCleaner.scala:908)
> at kafka.log.Cleaner.$anonfun$buildOffsetMap$5$adapted(LogCleaner.scala:904)
> at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:985)
> at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
> at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:984)
> at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:904)
> at kafka.log.Cleaner.doClean(LogCleaner.scala:523)
> at kafka.log.Cleaner.clean(LogCleaner.scala:511)
> at kafka.log.LogCleaner$CleanerThread.cleanLog(LogCleaner.scala:380)
> at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:352)
> ... 3 more
> [2023-03-17 17:54:02,477] INFO Cleaner 0: Beginning cleaning of log __consumer_offsets-33. (kafka.log.LogCleaner)
> [2023-03-17 17:54:02,477] INFO Cleaner 0: Building offset map for __consumer_offsets-33... (kafka.log.LogCleaner)
> [2023-03-17 17:54:02,511] INFO Cleaner 0: Building offset map for log __consumer_offsets-33 for 2361 segments in offset range [2946663171, 7343698202). (kafka.log.LogCleaner)
> [2023-03-17 17:54:03,235] WARN [kafka-log-cleaner-thread-0]: Unexpected exception thrown when cleaning log Log(dir=/opt/kafka-service/data/__consumer_offsets-33, topic=__consumer_offsets, partition=33, highWatermark=7344703164, lastStableOffset=7344703164, logStartOffset=0, logEndOffset=7344703164). Marking its partition (__consumer_offsets-33) as uncleanable (kafka.log.LogCleaner)
> kafka.log.LogCleaningException: Batch size 529 < buffer size 10485772, but not processed for log segment /opt/kafka-service/data/__consumer_offsets-33/00000000002946663171.log at position 70025136
> at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:356)
> at kafka.log.LogCleaner$CleanerThread.tryCleanFilthiestLog(LogCleaner.scala:332)
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:321)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
> Caused by: java.lang.IllegalStateException: Batch size 529 < buffer size 10485772, but not processed for log segment /opt/kafka-service/data/__consumer_offsets-33/00000000002946663171.log at position 70025136
> at kafka.log.Cleaner.growBuffersOrFail(LogCleaner.scala:745)
> at kafka.log.Cleaner.buildOffsetMapForSegment(LogCleaner.scala:983)
> at kafka.log.Cleaner.$anonfun$buildOffsetMap$5(LogCleaner.scala:908)
> at kafka.log.Cleaner.$anonfun$buildOffsetMap$5$adapted(LogCleaner.scala:904)
> at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:985)
> at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
> at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:984)
> at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:904)
> at kafka.log.Cleaner.doClean(LogCleaner.scala:523)
> at kafka.log.Cleaner.clean(LogCleaner.scala:511)
> at kafka.log.LogCleaner$CleanerThread.cleanLog(LogCleaner.scala:380)
> at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:352)
> ... 3 more
> We want to know why they marked as uncleanable and how to resume.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)