[jira] [Commented] (KAFKA-12946) __consumer_offsets topic with very big partitions

["Ron Dagostino (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-12946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17363155#comment-17363155 ] 

Ron Dagostino commented on KAFKA-12946:
---------------------------------------

If the partition isn't being cleaned then you can try setting min.cleanable.dirty.ratio=0 for the __consumer_offsets topic; this might allow it to get cleaned.  You can delete that config after a while to let the value default back.

Another possibility might exist if one of the follower replicas has a significantly smaller size than the leader; in such cases you can move leadership to the smaller replica and then reassign the follower replicas to new brokers so that they will copy the (much smaller-sized) data; then you can migrate the followers back to where they were originally and move the leader back to the original leader.  This solution will only work if you have more brokers than the replication factor.

Finally, take a look at https://cwiki.apache.org/confluence/display/KAFKA/KIP-664%3A+Provide+tooling+to+detect+and+abort+hanging+transactions.  You may not have any other options right now if it is a hanging transaction, but help is coming.

> __consumer_offsets topic with very big partitions
> -------------------------------------------------
>
>                 Key: KAFKA-12946
>                 URL: https://issues.apache.org/jira/browse/KAFKA-12946
>             Project: Kafka
>          Issue Type: Bug
>          Components: log cleaner
>    Affects Versions: 2.0.0
>            Reporter: Emi
>            Priority: Critical
>
> I am using Kafka 2.0.0 with java 8u191
>  There is a partitions of the __consumer_offsets topic that is 600 GB with 6000 segments older than 4 months. Other partitions of that topic are small: 20-30 MB.
> There are 60 consumer groups, 90 topics and 100 partitions per topic.
> There aren't errors in the logs. From the log of the logcleaner, I can see that partition is never touched from the logcleaner thread for the compaction, but it only add new segments.
>  How is this possible?
> There was another partition with the same problem, but after some months it has been compacted. Now there is only one partition with this problem, but this is bigger and keep growing
> I have used the kafka-dump-log tool to check these old segments and I can see many duplicates. So I would assume that is not compacted.
> My settings:
>  {{offsets.commit.required.acks = -1}}
>  {{[offsets.commit.timeout.ms|http://offsets.commit.timeout.ms/]}} = 5000
>  {{offsets.load.buffer.size = 5242880}}
>  {{[offsets.retention.check.interval.ms|http://offsets.retention.check.interval.ms/]}} = 600000
>  {{offsets.retention.minutes = 10080}}
>  {{offsets.topic.compression.codec = 0}}
>  {{offsets.topic.num.partitions = 50}}
>  {{offsets.topic.replication.factor = 3}}
>  {{offsets.topic.segment.bytes = 104857600}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)