You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by András Serény <se...@gravityrd.com> on 2014/06/10 17:22:55 UTC
Re: log.retention.size
Hi Kafka devs,
are there currently any plans to implement the global threshold feature?
Is there a JIRA about it?
We are considering to implement a solution for this issue (either inside
or outside of Kafka).
Thanks a lot,
András
On 5/30/2014 11:45 AM, András Serény wrote:
>
> Sorry for the delay on this.
>
> Yes, that's right -- it'd be just another term in the chain of 'or'
> conditions. Currently it's <time limit> OR <size limit>. With the
> global condition, it would be
> <time limit> OR <size limit> OR <global size limit>
>
> In my view, that's fairly simple and intuitive, hence a fine piece of
> logic.
>
> Regards,
> András
>
> On 5/27/2014 4:34 PM, Jun Rao wrote:
>> For log.retention.bytes.per.topic and log.retention.hours.per.topic, the
>> current interpretation is that those are tight bounds. In other
>> words, only
>> when those thresholds are violated, a segment is deleted. To further
>> satisfy log.retention.bytes.global, the per topic thresholds may no
>> longer
>> be tight, i.e., we may need to delete a segment even when the per topic
>> threshold is not violated.
>>
>> Thanks,
>>
>> Jun
>>
>>
>> On Tue, May 27, 2014 at 12:22 AM, András Serény
>> <sereny.andras@gravityrd.com
>>> wrote:
>>> No, I think more specific settings should get a chance first. I'm
>>> suggesting that provided that there is a segment rolled for a topic,
>>> *any
>>> *of log.retention.bytes.per.topic, log.retention.hours.per.topic, and a
>>> future log.retention.bytes.global violation would cause segments to be
>>> deleted.
>>>
>>> As far as I understand, the current logic says
>>>
>>> (1)
>>> for each topic, if there is a segment already rolled {
>>> mark segments eligible for deletion due to
>>> log.retention.hours.for.this.topic
>>> if log.retention.bytes.for.this.topic is still violated, mark
>>> segments eligible for deletion due to
>>> log.retention.bytes.for.this.topic
>>> }
>>>
>>> After this cleanup cycle, there could be another one, taking into
>>> account
>>> the global threshold. For instance, something along the lines of
>>>
>>> (2)
>>> if after (1) log.retention.bytes.global is still violated, for each
>>> topic,
>>> if there is a segment already rolled {
>>> calculate the required size for this topic (e.g. the proportional
>>> size,
>>> or simply (full size - threshold)/#topics ?)
>>> mark segments exceeding the required size for deletion
>>> }
>>>
>>> Regards,
>>> András
>>>
>>>
>>>
>>> On 5/23/2014 4:46 PM, Jun Rao wrote:
>>>
>>>> Yes, that's possible. There is a default log.retention.bytes for every
>>>> topic. By introducing a global threshold, we may have to delete
>>>> data from
>>>> logs whose size is smaller than log.retention.bytes. So, are you
>>>> saying
>>>> that the global threshold has precedence?
>>>>
>>>> Thanks,
>>>>
>>>> Jun
>>>>
>>>>
>>>> On Fri, May 23, 2014 at 2:26 AM, András Serény
>>>> <se...@gravityrd.com>wrote:
>>>>
>>>> Hi Kafka users,
>>>>> this feature would also be very useful for us. With lots of topics of
>>>>> different volume (and as they grow in number) it could become
>>>>> tedious to
>>>>> maintain topic level settings.
>>>>>
>>>>> As a start, I think uniform reduction is a good idea. Logs
>>>>> wouldn't be
>>>>> retained as long as you want, but that's already the case when a
>>>>> log.retention.bytes setting is specified. As for early rolling, I
>>>>> don't
>>>>> think it's necessary: currently, if there is no log segment
>>>>> eligible for
>>>>> deletion, log.retention.bytes and log.retention.hours settings
>>>>> won't kick
>>>>> in, so it's possible to exceed these limits, which is completely fine
>>>>> (please correct me if I'm mistaken here).
>>>>>
>>>>> All in all, introducing a global threshold doesn't seem to induce a
>>>>> considerable change in current retention logic.
>>>>>
>>>>> Regards,
>>>>> András
>>>>>
>>>>>
>>>>> On 5/8/2014 2:00 AM, vinh wrote:
>>>>>
>>>>> Agreed…a global knob is a bit tricky for exactly the reason you've
>>>>>> identified. Perhaps the problem could be simplified though by
>>>>>> considering
>>>>>> the context and purpose of Kafka. I would use a persistent message
>>>>>> queue
>>>>>> because I want to guarantee that data/messages don't get lost. But,
>>>>>> since
>>>>>> Kafka is not meant to be a long term storage solution (other
>>>>>> products
>>>>>> can
>>>>>> be used for that), I would clarify that guarantee to apply only
>>>>>> to the
>>>>>> most
>>>>>> recent messages up until a certain configured threshold (i.e. max 24
>>>>>> hrs,
>>>>>> max 500GB, etc). Once those thresholds are reached, old messages
>>>>>> are
>>>>>> deleted first.
>>>>>>
>>>>>> To ensure no message loss (up to a limit), I must ensure Kafka is
>>>>>> highly
>>>>>> available. There's a small a chance that the message deletion
>>>>>> rate is
>>>>>> the
>>>>>> same rate that receive rate. For example, when the incoming
>>>>>> volume is
>>>>>> so
>>>>>> high that the size threshold is reached before the time threshold.
>>>>>> But, I
>>>>>> may be ok with that because if Kafka goes down, it can cause
>>>>>> upstream
>>>>>> applications to fail. This can result in higher losses overall, and
>>>>>> particularly of the most *recent* messages.
>>>>>>
>>>>>> In other words, in a persistent but ephemeral message queue, I would
>>>>>> give
>>>>>> higher precedence to recent messages over older ones. On the flip
>>>>>> side, by
>>>>>> allowing Kafka to go down when a disk is full, applications are
>>>>>> forced
>>>>>> to
>>>>>> deal with the issue. This adds complexity to apps, but perhaps it's
>>>>>> not a
>>>>>> bad thing. After all, in scalability, all apps should be
>>>>>> designed to
>>>>>> handle failure.
>>>>>>
>>>>>> Having said that, next is to decide which messages to delete
>>>>>> first. I
>>>>>> believe that's a separate issue and has its own complexities, too.
>>>>>>
>>>>>> The main idea though is that a global knob would provide
>>>>>> flexibility,
>>>>>> even if not used. From an operation perspective, if we can't
>>>>>> ensure HA
>>>>>> for
>>>>>> all applications/components, it would be good if we can for at least
>>>>>> some
>>>>>> of the core ones, like Kafka. This is much easier said that done
>>>>>> though.
>>>>>>
>>>>>>
>>>>>> On May 5, 2014, at 9:16 AM, Jun Rao <ju...@gmail.com> wrote:
>>>>>>
>>>>>> Yes, your understanding is correct. A global knob that controls
>>>>>> aggregate
>>>>>>
>>>>>>> log size may make sense. What would be the expected behavior
>>>>>>> when that
>>>>>>> limit is reached? Would you reduce the retention uniformly
>>>>>>> across all
>>>>>>> topics? Then, it just means that some of the logs may not be
>>>>>>> retained
>>>>>>> as
>>>>>>> long as you want. Also, we need to think through what happens when
>>>>>>> every
>>>>>>> log has only 1 segment left and yet the total size still exceeds
>>>>>>> the
>>>>>>> limit.
>>>>>>> Do we roll log segments early?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Jun
>>>>>>>
>>>>>>>
>>>>>>> On Sun, May 4, 2014 at 4:31 AM, vinh <vi...@loggly.com> wrote:
>>>>>>>
>>>>>>> Thanks Jun. So if I understand this correctly, there really
>>>>>>> is no
>>>>>>>
>>>>>>>> master
>>>>>>>> property to control the total aggregate size of all Kafka data
>>>>>>>> files
>>>>>>>> on
>>>>>>>> a
>>>>>>>> broker.
>>>>>>>>
>>>>>>>> log.retention.size and log.file.size are great for managing
>>>>>>>> data at
>>>>>>>> the
>>>>>>>> application level. In our case, application needs change
>>>>>>>> frequently,
>>>>>>>> and
>>>>>>>> performance itself is an ever evolving feature. This means various
>>>>>>>> configs
>>>>>>>> are constantly changing, like topics, # of partitions, etc.
>>>>>>>>
>>>>>>>> What rarely changes though is provisioned hardware resources.
>>>>>>>> So a
>>>>>>>> setting to control the total aggregate size of Kafka logs (or
>>>>>>>> persisted
>>>>>>>> data, for better clarity) would definitely simplify things at an
>>>>>>>> operational level, regardless what happens at the application
>>>>>>>> level.
>>>>>>>>
>>>>>>>>
>>>>>>>> On May 2, 2014, at 7:49 AM, Jun Rao <ju...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> log.retention.size controls the total size in a log dir (per
>>>>>>>>
>>>>>>>>> partition). log.file.size
>>>>>>>>> controls the size of each log segment in the log dir.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Jun
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, May 1, 2014 at 9:31 PM, vinh <vi...@loggly.com> wrote:
>>>>>>>>>
>>>>>>>>> In the 0.7 docs, the description for log.retention.size and
>>>>>>>>> log.file.size
>>>>>>>>> sound very much the same. In particular, that they apply to a
>>>>>>>>> single
>>>>>>>>> log
>>>>>>>>> file (or log segment file).
>>>>>>>>>
>>>>>>>>>> http://kafka.apache.org/07/configuration.html
>>>>>>>>>>
>>>>>>>>>> I'm beginning to think there is no setting to control the max
>>>>>>>>>> aggregate
>>>>>>>>>> size of all logs. If this is correct, what would be a good
>>>>>>>>>> approach
>>>>>>>>>> to
>>>>>>>>>> enforce this requirement? In my particular scenario, I have
>>>>>>>>>> a lot
>>>>>>>>>> of
>>>>>>>>>>
>>>>>>>>>> data
>>>>>>>>> being written to Kafka at a very high rate. So a 1TB disk can
>>>>>>>>> easily
>>>>>>>>>
>>>>>>>>>> be
>>>>>>>>>> filled up in 24hrs or so. One option is to add more Kafka
>>>>>>>>>> brokers
>>>>>>>>>> to
>>>>>>>>>>
>>>>>>>>>> add
>>>>>>>>> more disk space to the pool, but I'd like to avoid that and
>>>>>>>>> see if I
>>>>>>>>>
>>>>>>>>>> can
>>>>>>>>>> simply configure Kafka to not write more than 1TB aggregate.
>>>>>>>>>> Else,
>>>>>>>>>>
>>>>>>>>>> Kafka
>>>>>>>>> will OOM and kill itself, and possibly the crash the node itself
>>>>>>>>>
>>>>>>>>>> because
>>>>>>>>>> the disk is full.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On May 1, 2014, at 9:21 PM, vinh <vi...@loggly.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Using Kafka 0.7.2, I have the following in server.properties:
>>>>>>>>>>
>>>>>>>>>>> log.retention.hours=48
>>>>>>>>>>> log.retention.size=107374182400
>>>>>>>>>>> log.file.size=536870912
>>>>>>>>>>>
>>>>>>>>>>> My interpretation of this is:
>>>>>>>>>>> a) a single log segment file over 48hrs old will be deleted
>>>>>>>>>>> b) the total combined size of *all* logs is 100GB
>>>>>>>>>>> c) a single log segment file is limited to 500MB in size
>>>>>>>>>>> before a
>>>>>>>>>>> new
>>>>>>>>>>>
>>>>>>>>>>> segment file is spawned spawning a new segment file
>>>>>>>>>> d) a "log file" can be composed of many "log segment files"
>>>>>>>>>>> But, even after setting the above, I find that the total
>>>>>>>>>>> combined
>>>>>>>>>>> size
>>>>>>>>>>>
>>>>>>>>>>> of all Kafka logs on disk is 200GB right now. Isn't
>>>>>>>>>> log.retention.size
>>>>>>>>>> supposed to limit it to 100GB? Am I missing something? The
>>>>>>>>>> docs
>>>>>>>>>> are
>>>>>>>>>>
>>>>>>>>>> not
>>>>>>>>> really clear, especially when it comes to distinguishing
>>>>>>>>> between a
>>>>>>>>> "log
>>>>>>>>>
>>>>>>>>>> file" and a "log segment file".
>>>>>>>>>>
>>>>>>>>>> I have disk monitoring. But like anything else in
>>>>>>>>>> software, even
>>>>>>>>>>> monitoring can fail. Via configuration, I'd like to make sure
>>>>>>>>>> that
>>>>>>>>>>
>>>>>>>>>> Kafka
>>>>>>>>> does not write more than the available disk space. Or
>>>>>>>>> something like
>>>>>>>>>
>>>>>>>>>> log4j, where I can set a max number of log files and the max
>>>>>>>>>> size
>>>>>>>>>> per
>>>>>>>>>>
>>>>>>>>>> file,
>>>>>>>>> which essentially allows me to set a max aggregate size limit
>>>>>>>>> across
>>>>>>>>>
>>>>>>>>>> all
>>>>>>>>>> logs.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>> -Vinh
>>>>>>>>>>>
>>>>>>>>>>>
>
Re: log.retention.size
Posted by András Serény <se...@gravityrd.com>.
Opened https://issues.apache.org/jira/browse/KAFKA-1489 .
Regards,
András
On 6/11/2014 6:19 AM, Jun Rao wrote:
> Could you file a jira to track this?
>
> Thanks,
>
> Jun
>
>
> On Tue, Jun 10, 2014 at 8:22 AM, András Serény <se...@gravityrd.com>
> wrote:
>
>> Hi Kafka devs,
>>
>> are there currently any plans to implement the global threshold feature?
>> Is there a JIRA about it?
>>
>> We are considering to implement a solution for this issue (either inside
>> or outside of Kafka).
>>
>> Thanks a lot,
>> András
>>
>>
>> On 5/30/2014 11:45 AM, András Serény wrote:
>>
>>> Sorry for the delay on this.
>>>
>>> Yes, that's right -- it'd be just another term in the chain of 'or'
>>> conditions. Currently it's <time limit> OR <size limit>. With the global
>>> condition, it would be
>>> <time limit> OR <size limit> OR <global size limit>
>>>
>>> In my view, that's fairly simple and intuitive, hence a fine piece of
>>> logic.
>>>
>>> Regards,
>>> András
>>>
>>> On 5/27/2014 4:34 PM, Jun Rao wrote:
>>>
>>>> For log.retention.bytes.per.topic and log.retention.hours.per.topic, the
>>>> current interpretation is that those are tight bounds. In other words,
>>>> only
>>>> when those thresholds are violated, a segment is deleted. To further
>>>> satisfy log.retention.bytes.global, the per topic thresholds may no
>>>> longer
>>>> be tight, i.e., we may need to delete a segment even when the per topic
>>>> threshold is not violated.
>>>>
>>>> Thanks,
>>>>
>>>> Jun
>>>>
>>>>
>>>> On Tue, May 27, 2014 at 12:22 AM, András Serény <
>>>> sereny.andras@gravityrd.com
>>>>
>>>>> wrote:
>>>>> No, I think more specific settings should get a chance first. I'm
>>>>> suggesting that provided that there is a segment rolled for a topic,
>>>>> *any
>>>>> *of log.retention.bytes.per.topic, log.retention.hours.per.topic, and a
>>>>> future log.retention.bytes.global violation would cause segments to be
>>>>> deleted.
>>>>>
>>>>> As far as I understand, the current logic says
>>>>>
>>>>> (1)
>>>>> for each topic, if there is a segment already rolled {
>>>>> mark segments eligible for deletion due to
>>>>> log.retention.hours.for.this.topic
>>>>> if log.retention.bytes.for.this.topic is still violated, mark
>>>>> segments eligible for deletion due to log.retention.bytes.for.this.
>>>>> topic
>>>>> }
>>>>>
>>>>> After this cleanup cycle, there could be another one, taking into
>>>>> account
>>>>> the global threshold. For instance, something along the lines of
>>>>>
>>>>> (2)
>>>>> if after (1) log.retention.bytes.global is still violated, for each
>>>>> topic,
>>>>> if there is a segment already rolled {
>>>>> calculate the required size for this topic (e.g. the proportional
>>>>> size,
>>>>> or simply (full size - threshold)/#topics ?)
>>>>> mark segments exceeding the required size for deletion
>>>>> }
>>>>>
>>>>> Regards,
>>>>> András
>>>>>
>>>>>
>>>>>
>>>>> On 5/23/2014 4:46 PM, Jun Rao wrote:
>>>>>
>>>>> Yes, that's possible. There is a default log.retention.bytes for every
>>>>>> topic. By introducing a global threshold, we may have to delete data
>>>>>> from
>>>>>> logs whose size is smaller than log.retention.bytes. So, are you saying
>>>>>> that the global threshold has precedence?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Jun
>>>>>>
>>>>>>
>>>>>> On Fri, May 23, 2014 at 2:26 AM, András Serény
>>>>>> <se...@gravityrd.com>wrote:
>>>>>>
>>>>>> Hi Kafka users,
>>>>>>
>>>>>>> this feature would also be very useful for us. With lots of topics of
>>>>>>> different volume (and as they grow in number) it could become tedious
>>>>>>> to
>>>>>>> maintain topic level settings.
>>>>>>>
>>>>>>> As a start, I think uniform reduction is a good idea. Logs wouldn't be
>>>>>>> retained as long as you want, but that's already the case when a
>>>>>>> log.retention.bytes setting is specified. As for early rolling, I
>>>>>>> don't
>>>>>>> think it's necessary: currently, if there is no log segment eligible
>>>>>>> for
>>>>>>> deletion, log.retention.bytes and log.retention.hours settings won't
>>>>>>> kick
>>>>>>> in, so it's possible to exceed these limits, which is completely fine
>>>>>>> (please correct me if I'm mistaken here).
>>>>>>>
>>>>>>> All in all, introducing a global threshold doesn't seem to induce a
>>>>>>> considerable change in current retention logic.
>>>>>>>
>>>>>>> Regards,
>>>>>>> András
>>>>>>>
>>>>>>>
>>>>>>> On 5/8/2014 2:00 AM, vinh wrote:
>>>>>>>
>>>>>>> Agreed…a global knob is a bit tricky for exactly the reason you've
>>>>>>>
>>>>>>>> identified. Perhaps the problem could be simplified though by
>>>>>>>> considering
>>>>>>>> the context and purpose of Kafka. I would use a persistent message
>>>>>>>> queue
>>>>>>>> because I want to guarantee that data/messages don't get lost. But,
>>>>>>>> since
>>>>>>>> Kafka is not meant to be a long term storage solution (other products
>>>>>>>> can
>>>>>>>> be used for that), I would clarify that guarantee to apply only to
>>>>>>>> the
>>>>>>>> most
>>>>>>>> recent messages up until a certain configured threshold (i.e. max 24
>>>>>>>> hrs,
>>>>>>>> max 500GB, etc). Once those thresholds are reached, old messages are
>>>>>>>> deleted first.
>>>>>>>>
>>>>>>>> To ensure no message loss (up to a limit), I must ensure Kafka is
>>>>>>>> highly
>>>>>>>> available. There's a small a chance that the message deletion rate
>>>>>>>> is
>>>>>>>> the
>>>>>>>> same rate that receive rate. For example, when the incoming volume
>>>>>>>> is
>>>>>>>> so
>>>>>>>> high that the size threshold is reached before the time threshold.
>>>>>>>> But, I
>>>>>>>> may be ok with that because if Kafka goes down, it can cause upstream
>>>>>>>> applications to fail. This can result in higher losses overall, and
>>>>>>>> particularly of the most *recent* messages.
>>>>>>>>
>>>>>>>> In other words, in a persistent but ephemeral message queue, I would
>>>>>>>> give
>>>>>>>> higher precedence to recent messages over older ones. On the flip
>>>>>>>> side, by
>>>>>>>> allowing Kafka to go down when a disk is full, applications are
>>>>>>>> forced
>>>>>>>> to
>>>>>>>> deal with the issue. This adds complexity to apps, but perhaps it's
>>>>>>>> not a
>>>>>>>> bad thing. After all, in scalability, all apps should be designed to
>>>>>>>> handle failure.
>>>>>>>>
>>>>>>>> Having said that, next is to decide which messages to delete first.
>>>>>>>> I
>>>>>>>> believe that's a separate issue and has its own complexities, too.
>>>>>>>>
>>>>>>>> The main idea though is that a global knob would provide flexibility,
>>>>>>>> even if not used. From an operation perspective, if we can't ensure
>>>>>>>> HA
>>>>>>>> for
>>>>>>>> all applications/components, it would be good if we can for at least
>>>>>>>> some
>>>>>>>> of the core ones, like Kafka. This is much easier said that done
>>>>>>>> though.
>>>>>>>>
>>>>>>>>
>>>>>>>> On May 5, 2014, at 9:16 AM, Jun Rao <ju...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Yes, your understanding is correct. A global knob that controls
>>>>>>>> aggregate
>>>>>>>>
>>>>>>>> log size may make sense. What would be the expected behavior when
>>>>>>>>> that
>>>>>>>>> limit is reached? Would you reduce the retention uniformly across
>>>>>>>>> all
>>>>>>>>> topics? Then, it just means that some of the logs may not be
>>>>>>>>> retained
>>>>>>>>> as
>>>>>>>>> long as you want. Also, we need to think through what happens when
>>>>>>>>> every
>>>>>>>>> log has only 1 segment left and yet the total size still exceeds the
>>>>>>>>> limit.
>>>>>>>>> Do we roll log segments early?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Jun
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sun, May 4, 2014 at 4:31 AM, vinh <vi...@loggly.com> wrote:
>>>>>>>>>
>>>>>>>>> Thanks Jun. So if I understand this correctly, there really is
>>>>>>>>> no
>>>>>>>>>
>>>>>>>>> master
>>>>>>>>>> property to control the total aggregate size of all Kafka data
>>>>>>>>>> files
>>>>>>>>>> on
>>>>>>>>>> a
>>>>>>>>>> broker.
>>>>>>>>>>
>>>>>>>>>> log.retention.size and log.file.size are great for managing data at
>>>>>>>>>> the
>>>>>>>>>> application level. In our case, application needs change
>>>>>>>>>> frequently,
>>>>>>>>>> and
>>>>>>>>>> performance itself is an ever evolving feature. This means various
>>>>>>>>>> configs
>>>>>>>>>> are constantly changing, like topics, # of partitions, etc.
>>>>>>>>>>
>>>>>>>>>> What rarely changes though is provisioned hardware resources. So a
>>>>>>>>>> setting to control the total aggregate size of Kafka logs (or
>>>>>>>>>> persisted
>>>>>>>>>> data, for better clarity) would definitely simplify things at an
>>>>>>>>>> operational level, regardless what happens at the application
>>>>>>>>>> level.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On May 2, 2014, at 7:49 AM, Jun Rao <ju...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> log.retention.size controls the total size in a log dir (per
>>>>>>>>>>
>>>>>>>>>> partition). log.file.size
>>>>>>>>>>> controls the size of each log segment in the log dir.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>>
>>>>>>>>>>> Jun
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, May 1, 2014 at 9:31 PM, vinh <vi...@loggly.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> In the 0.7 docs, the description for log.retention.size and
>>>>>>>>>>> log.file.size
>>>>>>>>>>> sound very much the same. In particular, that they apply to a
>>>>>>>>>>> single
>>>>>>>>>>> log
>>>>>>>>>>> file (or log segment file).
>>>>>>>>>>>
>>>>>>>>>>> http://kafka.apache.org/07/configuration.html
>>>>>>>>>>>> I'm beginning to think there is no setting to control the max
>>>>>>>>>>>> aggregate
>>>>>>>>>>>> size of all logs. If this is correct, what would be a good
>>>>>>>>>>>> approach
>>>>>>>>>>>> to
>>>>>>>>>>>> enforce this requirement? In my particular scenario, I have a
>>>>>>>>>>>> lot
>>>>>>>>>>>> of
>>>>>>>>>>>>
>>>>>>>>>>>> data
>>>>>>>>>>>>
>>>>>>>>>>> being written to Kafka at a very high rate. So a 1TB disk can
>>>>>>>>>>> easily
>>>>>>>>>>>
>>>>>>>>>>> be
>>>>>>>>>>>> filled up in 24hrs or so. One option is to add more Kafka
>>>>>>>>>>>> brokers
>>>>>>>>>>>> to
>>>>>>>>>>>>
>>>>>>>>>>>> add
>>>>>>>>>>>>
>>>>>>>>>>> more disk space to the pool, but I'd like to avoid that and see
>>>>>>>>>>> if I
>>>>>>>>>>>
>>>>>>>>>>> can
>>>>>>>>>>>> simply configure Kafka to not write more than 1TB aggregate.
>>>>>>>>>>>> Else,
>>>>>>>>>>>>
>>>>>>>>>>>> Kafka
>>>>>>>>>>>>
>>>>>>>>>>> will OOM and kill itself, and possibly the crash the node itself
>>>>>>>>>>>
>>>>>>>>>>> because
>>>>>>>>>>>> the disk is full.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On May 1, 2014, at 9:21 PM, vinh <vi...@loggly.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Using Kafka 0.7.2, I have the following in server.properties:
>>>>>>>>>>>>
>>>>>>>>>>>> log.retention.hours=48
>>>>>>>>>>>>> log.retention.size=107374182400
>>>>>>>>>>>>> log.file.size=536870912
>>>>>>>>>>>>>
>>>>>>>>>>>>> My interpretation of this is:
>>>>>>>>>>>>> a) a single log segment file over 48hrs old will be deleted
>>>>>>>>>>>>> b) the total combined size of *all* logs is 100GB
>>>>>>>>>>>>> c) a single log segment file is limited to 500MB in size before
>>>>>>>>>>>>> a
>>>>>>>>>>>>> new
>>>>>>>>>>>>>
>>>>>>>>>>>>> segment file is spawned spawning a new segment file
>>>>>>>>>>>>>
>>>>>>>>>>>> d) a "log file" can be composed of many "log segment files"
>>>>>>>>>>>>
>>>>>>>>>>>>> But, even after setting the above, I find that the total
>>>>>>>>>>>>> combined
>>>>>>>>>>>>> size
>>>>>>>>>>>>>
>>>>>>>>>>>>> of all Kafka logs on disk is 200GB right now. Isn't
>>>>>>>>>>>>>
>>>>>>>>>>>> log.retention.size
>>>>>>>>>>>> supposed to limit it to 100GB? Am I missing something? The docs
>>>>>>>>>>>> are
>>>>>>>>>>>>
>>>>>>>>>>>> not
>>>>>>>>>>>>
>>>>>>>>>>> really clear, especially when it comes to distinguishing between a
>>>>>>>>>>> "log
>>>>>>>>>>>
>>>>>>>>>>> file" and a "log segment file".
>>>>>>>>>>>> I have disk monitoring. But like anything else in software,
>>>>>>>>>>>> even
>>>>>>>>>>>>
>>>>>>>>>>>>> monitoring can fail. Via configuration, I'd like to make sure
>>>>>>>>>>>>>
>>>>>>>>>>>> that
>>>>>>>>>>>>
>>>>>>>>>>>> Kafka
>>>>>>>>>>>>
>>>>>>>>>>> does not write more than the available disk space. Or something
>>>>>>>>>>> like
>>>>>>>>>>>
>>>>>>>>>>> log4j, where I can set a max number of log files and the max size
>>>>>>>>>>>> per
>>>>>>>>>>>>
>>>>>>>>>>>> file,
>>>>>>>>>>>>
>>>>>>>>>>> which essentially allows me to set a max aggregate size limit
>>>>>>>>>>> across
>>>>>>>>>>>
>>>>>>>>>>> all
>>>>>>>>>>>> logs.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>
>>>>>>>>>>>>> -Vinh
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
Re: log.retention.size
Posted by Jun Rao <ju...@gmail.com>.
Could you file a jira to track this?
Thanks,
Jun
On Tue, Jun 10, 2014 at 8:22 AM, András Serény <se...@gravityrd.com>
wrote:
> Hi Kafka devs,
>
> are there currently any plans to implement the global threshold feature?
> Is there a JIRA about it?
>
> We are considering to implement a solution for this issue (either inside
> or outside of Kafka).
>
> Thanks a lot,
> András
>
>
> On 5/30/2014 11:45 AM, András Serény wrote:
>
>>
>> Sorry for the delay on this.
>>
>> Yes, that's right -- it'd be just another term in the chain of 'or'
>> conditions. Currently it's <time limit> OR <size limit>. With the global
>> condition, it would be
>> <time limit> OR <size limit> OR <global size limit>
>>
>> In my view, that's fairly simple and intuitive, hence a fine piece of
>> logic.
>>
>> Regards,
>> András
>>
>> On 5/27/2014 4:34 PM, Jun Rao wrote:
>>
>>> For log.retention.bytes.per.topic and log.retention.hours.per.topic, the
>>> current interpretation is that those are tight bounds. In other words,
>>> only
>>> when those thresholds are violated, a segment is deleted. To further
>>> satisfy log.retention.bytes.global, the per topic thresholds may no
>>> longer
>>> be tight, i.e., we may need to delete a segment even when the per topic
>>> threshold is not violated.
>>>
>>> Thanks,
>>>
>>> Jun
>>>
>>>
>>> On Tue, May 27, 2014 at 12:22 AM, András Serény <
>>> sereny.andras@gravityrd.com
>>>
>>>> wrote:
>>>> No, I think more specific settings should get a chance first. I'm
>>>> suggesting that provided that there is a segment rolled for a topic,
>>>> *any
>>>> *of log.retention.bytes.per.topic, log.retention.hours.per.topic, and a
>>>> future log.retention.bytes.global violation would cause segments to be
>>>> deleted.
>>>>
>>>> As far as I understand, the current logic says
>>>>
>>>> (1)
>>>> for each topic, if there is a segment already rolled {
>>>> mark segments eligible for deletion due to
>>>> log.retention.hours.for.this.topic
>>>> if log.retention.bytes.for.this.topic is still violated, mark
>>>> segments eligible for deletion due to log.retention.bytes.for.this.
>>>> topic
>>>> }
>>>>
>>>> After this cleanup cycle, there could be another one, taking into
>>>> account
>>>> the global threshold. For instance, something along the lines of
>>>>
>>>> (2)
>>>> if after (1) log.retention.bytes.global is still violated, for each
>>>> topic,
>>>> if there is a segment already rolled {
>>>> calculate the required size for this topic (e.g. the proportional
>>>> size,
>>>> or simply (full size - threshold)/#topics ?)
>>>> mark segments exceeding the required size for deletion
>>>> }
>>>>
>>>> Regards,
>>>> András
>>>>
>>>>
>>>>
>>>> On 5/23/2014 4:46 PM, Jun Rao wrote:
>>>>
>>>> Yes, that's possible. There is a default log.retention.bytes for every
>>>>> topic. By introducing a global threshold, we may have to delete data
>>>>> from
>>>>> logs whose size is smaller than log.retention.bytes. So, are you saying
>>>>> that the global threshold has precedence?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jun
>>>>>
>>>>>
>>>>> On Fri, May 23, 2014 at 2:26 AM, András Serény
>>>>> <se...@gravityrd.com>wrote:
>>>>>
>>>>> Hi Kafka users,
>>>>>
>>>>>> this feature would also be very useful for us. With lots of topics of
>>>>>> different volume (and as they grow in number) it could become tedious
>>>>>> to
>>>>>> maintain topic level settings.
>>>>>>
>>>>>> As a start, I think uniform reduction is a good idea. Logs wouldn't be
>>>>>> retained as long as you want, but that's already the case when a
>>>>>> log.retention.bytes setting is specified. As for early rolling, I
>>>>>> don't
>>>>>> think it's necessary: currently, if there is no log segment eligible
>>>>>> for
>>>>>> deletion, log.retention.bytes and log.retention.hours settings won't
>>>>>> kick
>>>>>> in, so it's possible to exceed these limits, which is completely fine
>>>>>> (please correct me if I'm mistaken here).
>>>>>>
>>>>>> All in all, introducing a global threshold doesn't seem to induce a
>>>>>> considerable change in current retention logic.
>>>>>>
>>>>>> Regards,
>>>>>> András
>>>>>>
>>>>>>
>>>>>> On 5/8/2014 2:00 AM, vinh wrote:
>>>>>>
>>>>>> Agreed…a global knob is a bit tricky for exactly the reason you've
>>>>>>
>>>>>>> identified. Perhaps the problem could be simplified though by
>>>>>>> considering
>>>>>>> the context and purpose of Kafka. I would use a persistent message
>>>>>>> queue
>>>>>>> because I want to guarantee that data/messages don't get lost. But,
>>>>>>> since
>>>>>>> Kafka is not meant to be a long term storage solution (other products
>>>>>>> can
>>>>>>> be used for that), I would clarify that guarantee to apply only to
>>>>>>> the
>>>>>>> most
>>>>>>> recent messages up until a certain configured threshold (i.e. max 24
>>>>>>> hrs,
>>>>>>> max 500GB, etc). Once those thresholds are reached, old messages are
>>>>>>> deleted first.
>>>>>>>
>>>>>>> To ensure no message loss (up to a limit), I must ensure Kafka is
>>>>>>> highly
>>>>>>> available. There's a small a chance that the message deletion rate
>>>>>>> is
>>>>>>> the
>>>>>>> same rate that receive rate. For example, when the incoming volume
>>>>>>> is
>>>>>>> so
>>>>>>> high that the size threshold is reached before the time threshold.
>>>>>>> But, I
>>>>>>> may be ok with that because if Kafka goes down, it can cause upstream
>>>>>>> applications to fail. This can result in higher losses overall, and
>>>>>>> particularly of the most *recent* messages.
>>>>>>>
>>>>>>> In other words, in a persistent but ephemeral message queue, I would
>>>>>>> give
>>>>>>> higher precedence to recent messages over older ones. On the flip
>>>>>>> side, by
>>>>>>> allowing Kafka to go down when a disk is full, applications are
>>>>>>> forced
>>>>>>> to
>>>>>>> deal with the issue. This adds complexity to apps, but perhaps it's
>>>>>>> not a
>>>>>>> bad thing. After all, in scalability, all apps should be designed to
>>>>>>> handle failure.
>>>>>>>
>>>>>>> Having said that, next is to decide which messages to delete first.
>>>>>>> I
>>>>>>> believe that's a separate issue and has its own complexities, too.
>>>>>>>
>>>>>>> The main idea though is that a global knob would provide flexibility,
>>>>>>> even if not used. From an operation perspective, if we can't ensure
>>>>>>> HA
>>>>>>> for
>>>>>>> all applications/components, it would be good if we can for at least
>>>>>>> some
>>>>>>> of the core ones, like Kafka. This is much easier said that done
>>>>>>> though.
>>>>>>>
>>>>>>>
>>>>>>> On May 5, 2014, at 9:16 AM, Jun Rao <ju...@gmail.com> wrote:
>>>>>>>
>>>>>>> Yes, your understanding is correct. A global knob that controls
>>>>>>> aggregate
>>>>>>>
>>>>>>> log size may make sense. What would be the expected behavior when
>>>>>>>> that
>>>>>>>> limit is reached? Would you reduce the retention uniformly across
>>>>>>>> all
>>>>>>>> topics? Then, it just means that some of the logs may not be
>>>>>>>> retained
>>>>>>>> as
>>>>>>>> long as you want. Also, we need to think through what happens when
>>>>>>>> every
>>>>>>>> log has only 1 segment left and yet the total size still exceeds the
>>>>>>>> limit.
>>>>>>>> Do we roll log segments early?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Jun
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, May 4, 2014 at 4:31 AM, vinh <vi...@loggly.com> wrote:
>>>>>>>>
>>>>>>>> Thanks Jun. So if I understand this correctly, there really is
>>>>>>>> no
>>>>>>>>
>>>>>>>> master
>>>>>>>>> property to control the total aggregate size of all Kafka data
>>>>>>>>> files
>>>>>>>>> on
>>>>>>>>> a
>>>>>>>>> broker.
>>>>>>>>>
>>>>>>>>> log.retention.size and log.file.size are great for managing data at
>>>>>>>>> the
>>>>>>>>> application level. In our case, application needs change
>>>>>>>>> frequently,
>>>>>>>>> and
>>>>>>>>> performance itself is an ever evolving feature. This means various
>>>>>>>>> configs
>>>>>>>>> are constantly changing, like topics, # of partitions, etc.
>>>>>>>>>
>>>>>>>>> What rarely changes though is provisioned hardware resources. So a
>>>>>>>>> setting to control the total aggregate size of Kafka logs (or
>>>>>>>>> persisted
>>>>>>>>> data, for better clarity) would definitely simplify things at an
>>>>>>>>> operational level, regardless what happens at the application
>>>>>>>>> level.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On May 2, 2014, at 7:49 AM, Jun Rao <ju...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> log.retention.size controls the total size in a log dir (per
>>>>>>>>>
>>>>>>>>> partition). log.file.size
>>>>>>>>>> controls the size of each log segment in the log dir.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Jun
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, May 1, 2014 at 9:31 PM, vinh <vi...@loggly.com> wrote:
>>>>>>>>>>
>>>>>>>>>> In the 0.7 docs, the description for log.retention.size and
>>>>>>>>>> log.file.size
>>>>>>>>>> sound very much the same. In particular, that they apply to a
>>>>>>>>>> single
>>>>>>>>>> log
>>>>>>>>>> file (or log segment file).
>>>>>>>>>>
>>>>>>>>>> http://kafka.apache.org/07/configuration.html
>>>>>>>>>>>
>>>>>>>>>>> I'm beginning to think there is no setting to control the max
>>>>>>>>>>> aggregate
>>>>>>>>>>> size of all logs. If this is correct, what would be a good
>>>>>>>>>>> approach
>>>>>>>>>>> to
>>>>>>>>>>> enforce this requirement? In my particular scenario, I have a
>>>>>>>>>>> lot
>>>>>>>>>>> of
>>>>>>>>>>>
>>>>>>>>>>> data
>>>>>>>>>>>
>>>>>>>>>> being written to Kafka at a very high rate. So a 1TB disk can
>>>>>>>>>> easily
>>>>>>>>>>
>>>>>>>>>> be
>>>>>>>>>>> filled up in 24hrs or so. One option is to add more Kafka
>>>>>>>>>>> brokers
>>>>>>>>>>> to
>>>>>>>>>>>
>>>>>>>>>>> add
>>>>>>>>>>>
>>>>>>>>>> more disk space to the pool, but I'd like to avoid that and see
>>>>>>>>>> if I
>>>>>>>>>>
>>>>>>>>>> can
>>>>>>>>>>> simply configure Kafka to not write more than 1TB aggregate.
>>>>>>>>>>> Else,
>>>>>>>>>>>
>>>>>>>>>>> Kafka
>>>>>>>>>>>
>>>>>>>>>> will OOM and kill itself, and possibly the crash the node itself
>>>>>>>>>>
>>>>>>>>>> because
>>>>>>>>>>> the disk is full.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On May 1, 2014, at 9:21 PM, vinh <vi...@loggly.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Using Kafka 0.7.2, I have the following in server.properties:
>>>>>>>>>>>
>>>>>>>>>>> log.retention.hours=48
>>>>>>>>>>>> log.retention.size=107374182400
>>>>>>>>>>>> log.file.size=536870912
>>>>>>>>>>>>
>>>>>>>>>>>> My interpretation of this is:
>>>>>>>>>>>> a) a single log segment file over 48hrs old will be deleted
>>>>>>>>>>>> b) the total combined size of *all* logs is 100GB
>>>>>>>>>>>> c) a single log segment file is limited to 500MB in size before
>>>>>>>>>>>> a
>>>>>>>>>>>> new
>>>>>>>>>>>>
>>>>>>>>>>>> segment file is spawned spawning a new segment file
>>>>>>>>>>>>
>>>>>>>>>>> d) a "log file" can be composed of many "log segment files"
>>>>>>>>>>>
>>>>>>>>>>>> But, even after setting the above, I find that the total
>>>>>>>>>>>> combined
>>>>>>>>>>>> size
>>>>>>>>>>>>
>>>>>>>>>>>> of all Kafka logs on disk is 200GB right now. Isn't
>>>>>>>>>>>>
>>>>>>>>>>> log.retention.size
>>>>>>>>>>> supposed to limit it to 100GB? Am I missing something? The docs
>>>>>>>>>>> are
>>>>>>>>>>>
>>>>>>>>>>> not
>>>>>>>>>>>
>>>>>>>>>> really clear, especially when it comes to distinguishing between a
>>>>>>>>>> "log
>>>>>>>>>>
>>>>>>>>>> file" and a "log segment file".
>>>>>>>>>>>
>>>>>>>>>>> I have disk monitoring. But like anything else in software,
>>>>>>>>>>> even
>>>>>>>>>>>
>>>>>>>>>>>> monitoring can fail. Via configuration, I'd like to make sure
>>>>>>>>>>>>
>>>>>>>>>>> that
>>>>>>>>>>>
>>>>>>>>>>> Kafka
>>>>>>>>>>>
>>>>>>>>>> does not write more than the available disk space. Or something
>>>>>>>>>> like
>>>>>>>>>>
>>>>>>>>>> log4j, where I can set a max number of log files and the max size
>>>>>>>>>>> per
>>>>>>>>>>>
>>>>>>>>>>> file,
>>>>>>>>>>>
>>>>>>>>>> which essentially allows me to set a max aggregate size limit
>>>>>>>>>> across
>>>>>>>>>>
>>>>>>>>>> all
>>>>>>>>>>> logs.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>>
>>>>>>>>>>>> -Vinh
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>
>