[jira] [Commented] (KAFKA-3000) __consumer_offsets topic grows indefinitely

["Grant Henke (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-3000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15061105#comment-15061105 ] 

Grant Henke commented on KAFKA-3000:
------------------------------------

Do you have log.cleaner.enable=true configured on your brokers? The default is false.

Since __consumer_offsets is a compacted topic old segments aren't supposed to be deleted. Instead the log cleaner, if enabled, should compact the topic.

Discussion about changing the default to true, and more related handling can be found in KAFKA-2988 and this email thread in the developer list: http://search-hadoop.com/m/uyzND1XRTEVv2ToT1&subj=Consumer+Offsets+Compaction

> __consumer_offsets topic grows indefinitely
> -------------------------------------------
>
>                 Key: KAFKA-3000
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3000
>             Project: Kafka
>          Issue Type: Bug
>          Components: offset manager
>    Affects Versions: 0.8.2.1
>         Environment: Ubuntu 14.04, 5 node kafka cluster + 5 node ZooKeeper cluster. ZooKeeper and Kafka coexist on 5 cloud boxes.
>            Reporter: Maxim Vladimirskiy
>
> Old segments of the __consumer_offsets topic seem to be never deleted. As of Dec 16, 2015, there are segments dating back to when we started use Kafka - Oct 26, 2015. However the idx for the respective oldest segment file is just about an hour fresh. All offset related settings are default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)