You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Jay Kreps (JIRA)" <ji...@apache.org> on 2014/08/01 07:08:38 UTC
[jira] [Commented] (KAFKA-1489) Global threshold on data retention
size
[ https://issues.apache.org/jira/browse/KAFKA-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14081912#comment-14081912 ]
Jay Kreps commented on KAFKA-1489:
----------------------------------
Thinking about this I think the right way for this to work would be to introduce a new topic-level config, something like topic.retention.bytes. To use this the broker would divide this count by the number of partitions to compute an effective per-partition retention number. This would avoid uneven retention as described above.
> Global threshold on data retention size
> ---------------------------------------
>
> Key: KAFKA-1489
> URL: https://issues.apache.org/jira/browse/KAFKA-1489
> Project: Kafka
> Issue Type: New Feature
> Components: log
> Affects Versions: 0.8.1.1
> Reporter: Andras Sereny
> Assignee: Jay Kreps
> Labels: newbie
>
> Currently, Kafka has per topic settings to control the size of one single log (log.retention.bytes). With lots of topics of different volume and as they grow in number, it could become tedious to maintain topic level settings applying to a single log.
> Often, a chunk of disk space is dedicated to Kafka that hosts all logs stored, so it'd make sense to have a configurable threshold to control how much space *all* data in Kafka can take up.
> See also:
> http://mail-archives.apache.org/mod_mbox/kafka-users/201406.mbox/browser
> http://mail-archives.apache.org/mod_mbox/kafka-users/201311.mbox/%3C20131107015125.GC9718@jkoshy-ld.linkedin.biz%3E
--
This message was sent by Atlassian JIRA
(v6.2#6252)