log.retention.size

[vinh <vi...@loggly.com>]

Using Kafka 0.7.2, I have the following in server.properties:

log.retention.hours=48
log.retention.size=107374182400
log.file.size=536870912

My interpretation of this is:
a) a single log segment file over 48hrs old will be deleted
b) the total combined size of *all* logs is 100GB
c) a single log segment file is limited to 500MB in size before a new segment file is spawned spawning a new segment file
d) a "log file" can be composed of many "log segment files"

But, even after setting the above, I find that the total combined size of all Kafka logs on disk is 200GB right now.  Isn't log.retention.size supposed to limit it to 100GB?  Am I missing something?  The docs are not really clear, especially when it comes to distinguishing between a "log file" and a "log segment file".

I have disk monitoring.  But like anything else in software, even monitoring can fail.  Via configuration, I'd like to make sure that Kafka does not write more than the available disk space.  Or something like log4j, where I can set a max number of log files and the max size per file, which essentially allows me to set a max aggregate size limit across all logs.

Thanks,
-Vinh

Re: log.retention.size

[András Serény <se...@gravityrd.com>]

Opened https://issues.apache.org/jira/browse/KAFKA-1489 .

Regards,
András

On 6/11/2014 6:19 AM, Jun Rao wrote:
> Could you file a jira to track this?
>
> Thanks,
>
> Jun
>
>
> On Tue, Jun 10, 2014 at 8:22 AM, András Serény <se...@gravityrd.com>
> wrote:
>
>> Hi Kafka devs,
>>
>> are there currently any plans to implement the global threshold feature?
>> Is there a JIRA about it?
>>
>> We are considering to implement a solution for this issue (either inside
>> or outside of Kafka).
>>
>> Thanks a lot,
>> András
>>
>>
>> On 5/30/2014 11:45 AM, András Serény wrote:
>>
>>> Sorry for the delay on this.
>>>
>>> Yes, that's right -- it'd be just another term in the chain of 'or'
>>> conditions. Currently it's <time limit> OR <size limit>. With the global
>>> condition, it would be
>>> <time limit> OR <size limit> OR <global size limit>
>>>
>>> In my view, that's fairly simple and intuitive, hence a fine piece of
>>> logic.
>>>
>>> Regards,
>>> András
>>>
>>> On 5/27/2014 4:34 PM, Jun Rao wrote:
>>>
>>>> For log.retention.bytes.per.topic and log.retention.hours.per.topic, the
>>>> current interpretation is that those are tight bounds. In other words,
>>>> only
>>>> when those thresholds are violated, a segment is deleted. To further
>>>> satisfy log.retention.bytes.global, the per topic thresholds may no
>>>> longer
>>>> be tight, i.e., we may need to delete a segment even when the per topic
>>>> threshold is not violated.
>>>>
>>>> Thanks,
>>>>
>>>> Jun
>>>>
>>>>
>>>> On Tue, May 27, 2014 at 12:22 AM, András Serény <
>>>> sereny.andras@gravityrd.com
>>>>
>>>>> wrote:
>>>>> No, I think more specific settings should get a chance first. I'm
>>>>> suggesting that provided that there is a segment rolled for a topic,
>>>>> *any
>>>>> *of log.retention.bytes.per.topic, log.retention.hours.per.topic, and a
>>>>> future log.retention.bytes.global violation would cause segments to be
>>>>> deleted.
>>>>>
>>>>> As far as I understand, the current logic says
>>>>>
>>>>> (1)
>>>>> for each topic, if there is a segment already rolled {
>>>>>       mark segments eligible for deletion due to
>>>>> log.retention.hours.for.this.topic
>>>>>       if log.retention.bytes.for.this.topic is still violated, mark
>>>>> segments eligible for deletion due to log.retention.bytes.for.this.
>>>>> topic
>>>>> }
>>>>>
>>>>> After this cleanup cycle, there could be another one,  taking into
>>>>> account
>>>>> the global threshold. For instance, something along the lines of
>>>>>
>>>>> (2)
>>>>> if after (1) log.retention.bytes.global is still violated, for each
>>>>> topic,
>>>>> if there is a segment already rolled {
>>>>>     calculate the required size for this topic (e.g. the proportional
>>>>> size,
>>>>> or simply (full size - threshold)/#topics ?)
>>>>>     mark segments exceeding the required size for deletion
>>>>> }
>>>>>
>>>>> Regards,
>>>>> András
>>>>>
>>>>>
>>>>>
>>>>> On 5/23/2014 4:46 PM, Jun Rao wrote:
>>>>>
>>>>>   Yes, that's possible. There is a default log.retention.bytes for every
>>>>>> topic. By introducing a global threshold, we may have to delete data
>>>>>> from
>>>>>> logs whose size is smaller than log.retention.bytes. So, are you saying
>>>>>> that the global threshold has precedence?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Jun
>>>>>>
>>>>>>
>>>>>> On Fri, May 23, 2014 at 2:26 AM, András Serény
>>>>>> <se...@gravityrd.com>wrote:
>>>>>>
>>>>>>    Hi Kafka users,
>>>>>>
>>>>>>> this feature would also be very useful for us. With lots of topics of
>>>>>>> different volume (and as they grow in number) it could become tedious
>>>>>>> to
>>>>>>> maintain topic level settings.
>>>>>>>
>>>>>>> As a start, I think uniform reduction is a good idea. Logs wouldn't be
>>>>>>> retained as long as you want, but that's already the case when a
>>>>>>> log.retention.bytes setting is specified. As for early rolling, I
>>>>>>> don't
>>>>>>> think it's necessary: currently, if there is no log segment eligible
>>>>>>> for
>>>>>>> deletion, log.retention.bytes and log.retention.hours settings won't
>>>>>>> kick
>>>>>>> in, so it's possible to exceed these limits, which is completely fine
>>>>>>> (please correct me if I'm mistaken here).
>>>>>>>
>>>>>>> All in all, introducing a global threshold doesn't seem to induce a
>>>>>>> considerable change in current retention logic.
>>>>>>>
>>>>>>> Regards,
>>>>>>> András
>>>>>>>
>>>>>>>
>>>>>>> On 5/8/2014 2:00 AM, vinh wrote:
>>>>>>>
>>>>>>>    Agreed…a global knob is a bit tricky for exactly the reason you've
>>>>>>>
>>>>>>>> identified.  Perhaps the problem could be simplified though by
>>>>>>>> considering
>>>>>>>> the context and purpose of Kafka.  I would use a persistent message
>>>>>>>> queue
>>>>>>>> because I want to guarantee that data/messages don't get lost.  But,
>>>>>>>> since
>>>>>>>> Kafka is not meant to be a long term storage solution (other products
>>>>>>>> can
>>>>>>>> be used for that), I would clarify that guarantee to apply only to
>>>>>>>> the
>>>>>>>> most
>>>>>>>> recent messages up until a certain configured threshold (i.e. max 24
>>>>>>>> hrs,
>>>>>>>> max 500GB, etc).  Once those thresholds are reached, old messages are
>>>>>>>> deleted first.
>>>>>>>>
>>>>>>>> To ensure no message loss (up to a limit), I must ensure Kafka is
>>>>>>>> highly
>>>>>>>> available.  There's a small a chance that the message deletion rate
>>>>>>>> is
>>>>>>>> the
>>>>>>>> same rate that receive rate.  For example, when the incoming volume
>>>>>>>> is
>>>>>>>> so
>>>>>>>> high that the size threshold is reached before the time threshold.
>>>>>>>>    But, I
>>>>>>>> may be ok with that because if Kafka goes down, it can cause upstream
>>>>>>>> applications to fail.  This can result in higher losses overall, and
>>>>>>>> particularly of the most *recent* messages.
>>>>>>>>
>>>>>>>> In other words, in a persistent but ephemeral message queue, I would
>>>>>>>> give
>>>>>>>> higher precedence to recent messages over older ones. On the flip
>>>>>>>> side, by
>>>>>>>> allowing Kafka to go down when a disk is full, applications are
>>>>>>>> forced
>>>>>>>> to
>>>>>>>> deal with the issue.  This adds complexity to apps, but perhaps it's
>>>>>>>> not a
>>>>>>>> bad thing.  After all, in scalability, all apps should be designed to
>>>>>>>> handle failure.
>>>>>>>>
>>>>>>>> Having said that, next is to decide which messages to delete first.
>>>>>>>>   I
>>>>>>>> believe that's a separate issue and has its own complexities, too.
>>>>>>>>
>>>>>>>> The main idea though is that a global knob would provide flexibility,
>>>>>>>> even if not used.  From an operation perspective, if we can't ensure
>>>>>>>> HA
>>>>>>>> for
>>>>>>>> all applications/components, it would be good if we can for at least
>>>>>>>> some
>>>>>>>> of the core ones, like Kafka.  This is much easier said that done
>>>>>>>> though.
>>>>>>>>
>>>>>>>>
>>>>>>>> On May 5, 2014, at 9:16 AM, Jun Rao <ju...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>     Yes, your understanding is correct. A global knob that controls
>>>>>>>> aggregate
>>>>>>>>
>>>>>>>>   log size may make sense. What would be the expected behavior when
>>>>>>>>> that
>>>>>>>>> limit is reached? Would you reduce the retention uniformly across
>>>>>>>>> all
>>>>>>>>> topics? Then, it just means that some of the logs may not be
>>>>>>>>> retained
>>>>>>>>> as
>>>>>>>>> long as you want. Also, we need to think through what happens when
>>>>>>>>> every
>>>>>>>>> log has only 1 segment left and yet the total size still exceeds the
>>>>>>>>> limit.
>>>>>>>>> Do we roll log segments early?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Jun
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sun, May 4, 2014 at 4:31 AM, vinh <vi...@loggly.com> wrote:
>>>>>>>>>
>>>>>>>>>     Thanks Jun.  So if I understand this correctly, there really is
>>>>>>>>> no
>>>>>>>>>
>>>>>>>>>   master
>>>>>>>>>> property to control the total aggregate size of all Kafka data
>>>>>>>>>> files
>>>>>>>>>> on
>>>>>>>>>> a
>>>>>>>>>> broker.
>>>>>>>>>>
>>>>>>>>>> log.retention.size and log.file.size are great for managing data at
>>>>>>>>>> the
>>>>>>>>>> application level.  In our case, application needs change
>>>>>>>>>> frequently,
>>>>>>>>>> and
>>>>>>>>>> performance itself is an ever evolving feature. This means various
>>>>>>>>>> configs
>>>>>>>>>> are constantly changing, like topics, # of partitions, etc.
>>>>>>>>>>
>>>>>>>>>> What rarely changes though is provisioned hardware resources.  So a
>>>>>>>>>> setting to control the total aggregate size of Kafka logs (or
>>>>>>>>>> persisted
>>>>>>>>>> data, for better clarity) would definitely simplify things at an
>>>>>>>>>> operational level, regardless what happens at the application
>>>>>>>>>> level.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On May 2, 2014, at 7:49 AM, Jun Rao <ju...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>     log.retention.size controls the total size in a log dir (per
>>>>>>>>>>
>>>>>>>>>>   partition). log.file.size
>>>>>>>>>>> controls the size of each log segment in the log dir.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>>
>>>>>>>>>>> Jun
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, May 1, 2014 at 9:31 PM, vinh <vi...@loggly.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>     In the 0.7 docs, the description for log.retention.size and
>>>>>>>>>>> log.file.size
>>>>>>>>>>> sound very much the same.  In particular, that they apply to a
>>>>>>>>>>> single
>>>>>>>>>>> log
>>>>>>>>>>> file (or log segment file).
>>>>>>>>>>>
>>>>>>>>>>>   http://kafka.apache.org/07/configuration.html
>>>>>>>>>>>> I'm beginning to think there is no setting to control the max
>>>>>>>>>>>> aggregate
>>>>>>>>>>>> size of all logs.  If this is correct, what would be a good
>>>>>>>>>>>> approach
>>>>>>>>>>>> to
>>>>>>>>>>>> enforce this requirement?  In my particular scenario, I have a
>>>>>>>>>>>> lot
>>>>>>>>>>>> of
>>>>>>>>>>>>
>>>>>>>>>>>>    data
>>>>>>>>>>>>
>>>>>>>>>>> being written to Kafka at a very high rate.  So a 1TB disk can
>>>>>>>>>>> easily
>>>>>>>>>>>
>>>>>>>>>>>   be
>>>>>>>>>>>> filled up in 24hrs or so.  One option is to add more Kafka
>>>>>>>>>>>> brokers
>>>>>>>>>>>> to
>>>>>>>>>>>>
>>>>>>>>>>>>    add
>>>>>>>>>>>>
>>>>>>>>>>> more disk space to the pool, but I'd like to avoid that and see
>>>>>>>>>>> if I
>>>>>>>>>>>
>>>>>>>>>>>   can
>>>>>>>>>>>> simply configure Kafka to not write more than 1TB aggregate.
>>>>>>>>>>>>   Else,
>>>>>>>>>>>>
>>>>>>>>>>>>    Kafka
>>>>>>>>>>>>
>>>>>>>>>>> will OOM and kill itself, and possibly the crash the node itself
>>>>>>>>>>>
>>>>>>>>>>>   because
>>>>>>>>>>>> the disk is full.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On May 1, 2014, at 9:21 PM, vinh <vi...@loggly.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>     Using Kafka 0.7.2, I have the following in server.properties:
>>>>>>>>>>>>
>>>>>>>>>>>>   log.retention.hours=48
>>>>>>>>>>>>> log.retention.size=107374182400
>>>>>>>>>>>>> log.file.size=536870912
>>>>>>>>>>>>>
>>>>>>>>>>>>> My interpretation of this is:
>>>>>>>>>>>>> a) a single log segment file over 48hrs old will be deleted
>>>>>>>>>>>>> b) the total combined size of *all* logs is 100GB
>>>>>>>>>>>>> c) a single log segment file is limited to 500MB in size before
>>>>>>>>>>>>> a
>>>>>>>>>>>>> new
>>>>>>>>>>>>>
>>>>>>>>>>>>>    segment file is spawned spawning a new segment file
>>>>>>>>>>>>>
>>>>>>>>>>>>    d) a "log file" can be composed of many "log segment files"
>>>>>>>>>>>>
>>>>>>>>>>>>> But, even after setting the above, I find that the total
>>>>>>>>>>>>> combined
>>>>>>>>>>>>> size
>>>>>>>>>>>>>
>>>>>>>>>>>>>    of all Kafka logs on disk is 200GB right now.  Isn't
>>>>>>>>>>>>>
>>>>>>>>>>>> log.retention.size
>>>>>>>>>>>> supposed to limit it to 100GB?  Am I missing something?  The docs
>>>>>>>>>>>> are
>>>>>>>>>>>>
>>>>>>>>>>>>    not
>>>>>>>>>>>>
>>>>>>>>>>> really clear, especially when it comes to distinguishing between a
>>>>>>>>>>> "log
>>>>>>>>>>>
>>>>>>>>>>>   file" and a "log segment file".
>>>>>>>>>>>>    I have disk monitoring.  But like anything else in software,
>>>>>>>>>>>> even
>>>>>>>>>>>>
>>>>>>>>>>>>>    monitoring can fail. Via configuration, I'd like to make sure
>>>>>>>>>>>>>
>>>>>>>>>>>> that
>>>>>>>>>>>>
>>>>>>>>>>>>    Kafka
>>>>>>>>>>>>
>>>>>>>>>>> does not write more than the available disk space.  Or something
>>>>>>>>>>> like
>>>>>>>>>>>
>>>>>>>>>>>   log4j, where I can set a max number of log files and the max size
>>>>>>>>>>>> per
>>>>>>>>>>>>
>>>>>>>>>>>>    file,
>>>>>>>>>>>>
>>>>>>>>>>> which essentially allows me to set a max aggregate size limit
>>>>>>>>>>> across
>>>>>>>>>>>
>>>>>>>>>>>   all
>>>>>>>>>>>> logs.
>>>>>>>>>>>>
>>>>>>>>>>>>    Thanks,
>>>>>>>>>>>>
>>>>>>>>>>>>> -Vinh
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>

Re: log.retention.size

[Jun Rao <ju...@gmail.com>]

Could you file a jira to track this?

Thanks,

Jun


On Tue, Jun 10, 2014 at 8:22 AM, András Serény <se...@gravityrd.com>
wrote:

> Hi Kafka devs,
>
> are there currently any plans to implement the global threshold feature?
> Is there a JIRA about it?
>
> We are considering to implement a solution for this issue (either inside
> or outside of Kafka).
>
> Thanks a lot,
> András
>
>
> On 5/30/2014 11:45 AM, András Serény wrote:
>
>>
>> Sorry for the delay on this.
>>
>> Yes, that's right -- it'd be just another term in the chain of 'or'
>> conditions. Currently it's <time limit> OR <size limit>. With the global
>> condition, it would be
>> <time limit> OR <size limit> OR <global size limit>
>>
>> In my view, that's fairly simple and intuitive, hence a fine piece of
>> logic.
>>
>> Regards,
>> András
>>
>> On 5/27/2014 4:34 PM, Jun Rao wrote:
>>
>>> For log.retention.bytes.per.topic and log.retention.hours.per.topic, the
>>> current interpretation is that those are tight bounds. In other words,
>>> only
>>> when those thresholds are violated, a segment is deleted. To further
>>> satisfy log.retention.bytes.global, the per topic thresholds may no
>>> longer
>>> be tight, i.e., we may need to delete a segment even when the per topic
>>> threshold is not violated.
>>>
>>> Thanks,
>>>
>>> Jun
>>>
>>>
>>> On Tue, May 27, 2014 at 12:22 AM, András Serény <
>>> sereny.andras@gravityrd.com
>>>
>>>> wrote:
>>>> No, I think more specific settings should get a chance first. I'm
>>>> suggesting that provided that there is a segment rolled for a topic,
>>>> *any
>>>> *of log.retention.bytes.per.topic, log.retention.hours.per.topic, and a
>>>> future log.retention.bytes.global violation would cause segments to be
>>>> deleted.
>>>>
>>>> As far as I understand, the current logic says
>>>>
>>>> (1)
>>>> for each topic, if there is a segment already rolled {
>>>>      mark segments eligible for deletion due to
>>>> log.retention.hours.for.this.topic
>>>>      if log.retention.bytes.for.this.topic is still violated, mark
>>>> segments eligible for deletion due to log.retention.bytes.for.this.
>>>> topic
>>>> }
>>>>
>>>> After this cleanup cycle, there could be another one,  taking into
>>>> account
>>>> the global threshold. For instance, something along the lines of
>>>>
>>>> (2)
>>>> if after (1) log.retention.bytes.global is still violated, for each
>>>> topic,
>>>> if there is a segment already rolled {
>>>>    calculate the required size for this topic (e.g. the proportional
>>>> size,
>>>> or simply (full size - threshold)/#topics ?)
>>>>    mark segments exceeding the required size for deletion
>>>> }
>>>>
>>>> Regards,
>>>> András
>>>>
>>>>
>>>>
>>>> On 5/23/2014 4:46 PM, Jun Rao wrote:
>>>>
>>>>  Yes, that's possible. There is a default log.retention.bytes for every
>>>>> topic. By introducing a global threshold, we may have to delete data
>>>>> from
>>>>> logs whose size is smaller than log.retention.bytes. So, are you saying
>>>>> that the global threshold has precedence?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jun
>>>>>
>>>>>
>>>>> On Fri, May 23, 2014 at 2:26 AM, András Serény
>>>>> <se...@gravityrd.com>wrote:
>>>>>
>>>>>   Hi Kafka users,
>>>>>
>>>>>> this feature would also be very useful for us. With lots of topics of
>>>>>> different volume (and as they grow in number) it could become tedious
>>>>>> to
>>>>>> maintain topic level settings.
>>>>>>
>>>>>> As a start, I think uniform reduction is a good idea. Logs wouldn't be
>>>>>> retained as long as you want, but that's already the case when a
>>>>>> log.retention.bytes setting is specified. As for early rolling, I
>>>>>> don't
>>>>>> think it's necessary: currently, if there is no log segment eligible
>>>>>> for
>>>>>> deletion, log.retention.bytes and log.retention.hours settings won't
>>>>>> kick
>>>>>> in, so it's possible to exceed these limits, which is completely fine
>>>>>> (please correct me if I'm mistaken here).
>>>>>>
>>>>>> All in all, introducing a global threshold doesn't seem to induce a
>>>>>> considerable change in current retention logic.
>>>>>>
>>>>>> Regards,
>>>>>> András
>>>>>>
>>>>>>
>>>>>> On 5/8/2014 2:00 AM, vinh wrote:
>>>>>>
>>>>>>   Agreed…a global knob is a bit tricky for exactly the reason you've
>>>>>>
>>>>>>> identified.  Perhaps the problem could be simplified though by
>>>>>>> considering
>>>>>>> the context and purpose of Kafka.  I would use a persistent message
>>>>>>> queue
>>>>>>> because I want to guarantee that data/messages don't get lost.  But,
>>>>>>> since
>>>>>>> Kafka is not meant to be a long term storage solution (other products
>>>>>>> can
>>>>>>> be used for that), I would clarify that guarantee to apply only to
>>>>>>> the
>>>>>>> most
>>>>>>> recent messages up until a certain configured threshold (i.e. max 24
>>>>>>> hrs,
>>>>>>> max 500GB, etc).  Once those thresholds are reached, old messages are
>>>>>>> deleted first.
>>>>>>>
>>>>>>> To ensure no message loss (up to a limit), I must ensure Kafka is
>>>>>>> highly
>>>>>>> available.  There's a small a chance that the message deletion rate
>>>>>>> is
>>>>>>> the
>>>>>>> same rate that receive rate.  For example, when the incoming volume
>>>>>>> is
>>>>>>> so
>>>>>>> high that the size threshold is reached before the time threshold.
>>>>>>>   But, I
>>>>>>> may be ok with that because if Kafka goes down, it can cause upstream
>>>>>>> applications to fail.  This can result in higher losses overall, and
>>>>>>> particularly of the most *recent* messages.
>>>>>>>
>>>>>>> In other words, in a persistent but ephemeral message queue, I would
>>>>>>> give
>>>>>>> higher precedence to recent messages over older ones. On the flip
>>>>>>> side, by
>>>>>>> allowing Kafka to go down when a disk is full, applications are
>>>>>>> forced
>>>>>>> to
>>>>>>> deal with the issue.  This adds complexity to apps, but perhaps it's
>>>>>>> not a
>>>>>>> bad thing.  After all, in scalability, all apps should be designed to
>>>>>>> handle failure.
>>>>>>>
>>>>>>> Having said that, next is to decide which messages to delete first.
>>>>>>>  I
>>>>>>> believe that's a separate issue and has its own complexities, too.
>>>>>>>
>>>>>>> The main idea though is that a global knob would provide flexibility,
>>>>>>> even if not used.  From an operation perspective, if we can't ensure
>>>>>>> HA
>>>>>>> for
>>>>>>> all applications/components, it would be good if we can for at least
>>>>>>> some
>>>>>>> of the core ones, like Kafka.  This is much easier said that done
>>>>>>> though.
>>>>>>>
>>>>>>>
>>>>>>> On May 5, 2014, at 9:16 AM, Jun Rao <ju...@gmail.com> wrote:
>>>>>>>
>>>>>>>    Yes, your understanding is correct. A global knob that controls
>>>>>>> aggregate
>>>>>>>
>>>>>>>  log size may make sense. What would be the expected behavior when
>>>>>>>> that
>>>>>>>> limit is reached? Would you reduce the retention uniformly across
>>>>>>>> all
>>>>>>>> topics? Then, it just means that some of the logs may not be
>>>>>>>> retained
>>>>>>>> as
>>>>>>>> long as you want. Also, we need to think through what happens when
>>>>>>>> every
>>>>>>>> log has only 1 segment left and yet the total size still exceeds the
>>>>>>>> limit.
>>>>>>>> Do we roll log segments early?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Jun
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, May 4, 2014 at 4:31 AM, vinh <vi...@loggly.com> wrote:
>>>>>>>>
>>>>>>>>    Thanks Jun.  So if I understand this correctly, there really is
>>>>>>>> no
>>>>>>>>
>>>>>>>>  master
>>>>>>>>> property to control the total aggregate size of all Kafka data
>>>>>>>>> files
>>>>>>>>> on
>>>>>>>>> a
>>>>>>>>> broker.
>>>>>>>>>
>>>>>>>>> log.retention.size and log.file.size are great for managing data at
>>>>>>>>> the
>>>>>>>>> application level.  In our case, application needs change
>>>>>>>>> frequently,
>>>>>>>>> and
>>>>>>>>> performance itself is an ever evolving feature. This means various
>>>>>>>>> configs
>>>>>>>>> are constantly changing, like topics, # of partitions, etc.
>>>>>>>>>
>>>>>>>>> What rarely changes though is provisioned hardware resources.  So a
>>>>>>>>> setting to control the total aggregate size of Kafka logs (or
>>>>>>>>> persisted
>>>>>>>>> data, for better clarity) would definitely simplify things at an
>>>>>>>>> operational level, regardless what happens at the application
>>>>>>>>> level.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On May 2, 2014, at 7:49 AM, Jun Rao <ju...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>    log.retention.size controls the total size in a log dir (per
>>>>>>>>>
>>>>>>>>>  partition). log.file.size
>>>>>>>>>> controls the size of each log segment in the log dir.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Jun
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, May 1, 2014 at 9:31 PM, vinh <vi...@loggly.com> wrote:
>>>>>>>>>>
>>>>>>>>>>    In the 0.7 docs, the description for log.retention.size and
>>>>>>>>>> log.file.size
>>>>>>>>>> sound very much the same.  In particular, that they apply to a
>>>>>>>>>> single
>>>>>>>>>> log
>>>>>>>>>> file (or log segment file).
>>>>>>>>>>
>>>>>>>>>>  http://kafka.apache.org/07/configuration.html
>>>>>>>>>>>
>>>>>>>>>>> I'm beginning to think there is no setting to control the max
>>>>>>>>>>> aggregate
>>>>>>>>>>> size of all logs.  If this is correct, what would be a good
>>>>>>>>>>> approach
>>>>>>>>>>> to
>>>>>>>>>>> enforce this requirement?  In my particular scenario, I have a
>>>>>>>>>>> lot
>>>>>>>>>>> of
>>>>>>>>>>>
>>>>>>>>>>>   data
>>>>>>>>>>>
>>>>>>>>>> being written to Kafka at a very high rate.  So a 1TB disk can
>>>>>>>>>> easily
>>>>>>>>>>
>>>>>>>>>>  be
>>>>>>>>>>> filled up in 24hrs or so.  One option is to add more Kafka
>>>>>>>>>>> brokers
>>>>>>>>>>> to
>>>>>>>>>>>
>>>>>>>>>>>   add
>>>>>>>>>>>
>>>>>>>>>> more disk space to the pool, but I'd like to avoid that and see
>>>>>>>>>> if I
>>>>>>>>>>
>>>>>>>>>>  can
>>>>>>>>>>> simply configure Kafka to not write more than 1TB aggregate.
>>>>>>>>>>>  Else,
>>>>>>>>>>>
>>>>>>>>>>>   Kafka
>>>>>>>>>>>
>>>>>>>>>> will OOM and kill itself, and possibly the crash the node itself
>>>>>>>>>>
>>>>>>>>>>  because
>>>>>>>>>>> the disk is full.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On May 1, 2014, at 9:21 PM, vinh <vi...@loggly.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>    Using Kafka 0.7.2, I have the following in server.properties:
>>>>>>>>>>>
>>>>>>>>>>>  log.retention.hours=48
>>>>>>>>>>>> log.retention.size=107374182400
>>>>>>>>>>>> log.file.size=536870912
>>>>>>>>>>>>
>>>>>>>>>>>> My interpretation of this is:
>>>>>>>>>>>> a) a single log segment file over 48hrs old will be deleted
>>>>>>>>>>>> b) the total combined size of *all* logs is 100GB
>>>>>>>>>>>> c) a single log segment file is limited to 500MB in size before
>>>>>>>>>>>> a
>>>>>>>>>>>> new
>>>>>>>>>>>>
>>>>>>>>>>>>   segment file is spawned spawning a new segment file
>>>>>>>>>>>>
>>>>>>>>>>>   d) a "log file" can be composed of many "log segment files"
>>>>>>>>>>>
>>>>>>>>>>>> But, even after setting the above, I find that the total
>>>>>>>>>>>> combined
>>>>>>>>>>>> size
>>>>>>>>>>>>
>>>>>>>>>>>>   of all Kafka logs on disk is 200GB right now.  Isn't
>>>>>>>>>>>>
>>>>>>>>>>> log.retention.size
>>>>>>>>>>> supposed to limit it to 100GB?  Am I missing something?  The docs
>>>>>>>>>>> are
>>>>>>>>>>>
>>>>>>>>>>>   not
>>>>>>>>>>>
>>>>>>>>>> really clear, especially when it comes to distinguishing between a
>>>>>>>>>> "log
>>>>>>>>>>
>>>>>>>>>>  file" and a "log segment file".
>>>>>>>>>>>
>>>>>>>>>>>   I have disk monitoring.  But like anything else in software,
>>>>>>>>>>> even
>>>>>>>>>>>
>>>>>>>>>>>>   monitoring can fail. Via configuration, I'd like to make sure
>>>>>>>>>>>>
>>>>>>>>>>> that
>>>>>>>>>>>
>>>>>>>>>>>   Kafka
>>>>>>>>>>>
>>>>>>>>>> does not write more than the available disk space.  Or something
>>>>>>>>>> like
>>>>>>>>>>
>>>>>>>>>>  log4j, where I can set a max number of log files and the max size
>>>>>>>>>>> per
>>>>>>>>>>>
>>>>>>>>>>>   file,
>>>>>>>>>>>
>>>>>>>>>> which essentially allows me to set a max aggregate size limit
>>>>>>>>>> across
>>>>>>>>>>
>>>>>>>>>>  all
>>>>>>>>>>> logs.
>>>>>>>>>>>
>>>>>>>>>>>   Thanks,
>>>>>>>>>>>
>>>>>>>>>>>> -Vinh
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>
>

Re: log.retention.size

[András Serény <se...@gravityrd.com>]

Hi Kafka devs,

are there currently any plans to implement the global threshold feature? 
Is there a JIRA about it?

We are considering to implement a solution for this issue (either inside 
or outside of Kafka).

Thanks a lot,
András

On 5/30/2014 11:45 AM, András Serény wrote:
>
> Sorry for the delay on this.
>
> Yes, that's right -- it'd be just another term in the chain of 'or' 
> conditions. Currently it's <time limit> OR <size limit>. With the 
> global condition, it would be
> <time limit> OR <size limit> OR <global size limit>
>
> In my view, that's fairly simple and intuitive, hence a fine piece of 
> logic.
>
> Regards,
> András
>
> On 5/27/2014 4:34 PM, Jun Rao wrote:
>> For log.retention.bytes.per.topic and log.retention.hours.per.topic, the
>> current interpretation is that those are tight bounds. In other 
>> words, only
>> when those thresholds are violated, a segment is deleted. To further
>> satisfy log.retention.bytes.global, the per topic thresholds may no 
>> longer
>> be tight, i.e., we may need to delete a segment even when the per topic
>> threshold is not violated.
>>
>> Thanks,
>>
>> Jun
>>
>>
>> On Tue, May 27, 2014 at 12:22 AM, András Serény 
>> <sereny.andras@gravityrd.com
>>> wrote:
>>> No, I think more specific settings should get a chance first. I'm
>>> suggesting that provided that there is a segment rolled for a topic, 
>>> *any
>>> *of log.retention.bytes.per.topic, log.retention.hours.per.topic, and a
>>> future log.retention.bytes.global violation would cause segments to be
>>> deleted.
>>>
>>> As far as I understand, the current logic says
>>>
>>> (1)
>>> for each topic, if there is a segment already rolled {
>>>      mark segments eligible for deletion due to
>>> log.retention.hours.for.this.topic
>>>      if log.retention.bytes.for.this.topic is still violated, mark
>>> segments eligible for deletion due to 
>>> log.retention.bytes.for.this.topic
>>> }
>>>
>>> After this cleanup cycle, there could be another one,  taking into 
>>> account
>>> the global threshold. For instance, something along the lines of
>>>
>>> (2)
>>> if after (1) log.retention.bytes.global is still violated, for each 
>>> topic,
>>> if there is a segment already rolled {
>>>    calculate the required size for this topic (e.g. the proportional 
>>> size,
>>> or simply (full size - threshold)/#topics ?)
>>>    mark segments exceeding the required size for deletion
>>> }
>>>
>>> Regards,
>>> András
>>>
>>>
>>>
>>> On 5/23/2014 4:46 PM, Jun Rao wrote:
>>>
>>>> Yes, that's possible. There is a default log.retention.bytes for every
>>>> topic. By introducing a global threshold, we may have to delete 
>>>> data from
>>>> logs whose size is smaller than log.retention.bytes. So, are you 
>>>> saying
>>>> that the global threshold has precedence?
>>>>
>>>> Thanks,
>>>>
>>>> Jun
>>>>
>>>>
>>>> On Fri, May 23, 2014 at 2:26 AM, András Serény
>>>> <se...@gravityrd.com>wrote:
>>>>
>>>>   Hi Kafka users,
>>>>> this feature would also be very useful for us. With lots of topics of
>>>>> different volume (and as they grow in number) it could become 
>>>>> tedious to
>>>>> maintain topic level settings.
>>>>>
>>>>> As a start, I think uniform reduction is a good idea. Logs 
>>>>> wouldn't be
>>>>> retained as long as you want, but that's already the case when a
>>>>> log.retention.bytes setting is specified. As for early rolling, I 
>>>>> don't
>>>>> think it's necessary: currently, if there is no log segment 
>>>>> eligible for
>>>>> deletion, log.retention.bytes and log.retention.hours settings 
>>>>> won't kick
>>>>> in, so it's possible to exceed these limits, which is completely fine
>>>>> (please correct me if I'm mistaken here).
>>>>>
>>>>> All in all, introducing a global threshold doesn't seem to induce a
>>>>> considerable change in current retention logic.
>>>>>
>>>>> Regards,
>>>>> András
>>>>>
>>>>>
>>>>> On 5/8/2014 2:00 AM, vinh wrote:
>>>>>
>>>>>   Agreed…a global knob is a bit tricky for exactly the reason you've
>>>>>> identified.  Perhaps the problem could be simplified though by
>>>>>> considering
>>>>>> the context and purpose of Kafka.  I would use a persistent message
>>>>>> queue
>>>>>> because I want to guarantee that data/messages don't get lost.  But,
>>>>>> since
>>>>>> Kafka is not meant to be a long term storage solution (other 
>>>>>> products
>>>>>> can
>>>>>> be used for that), I would clarify that guarantee to apply only 
>>>>>> to the
>>>>>> most
>>>>>> recent messages up until a certain configured threshold (i.e. max 24
>>>>>> hrs,
>>>>>> max 500GB, etc).  Once those thresholds are reached, old messages 
>>>>>> are
>>>>>> deleted first.
>>>>>>
>>>>>> To ensure no message loss (up to a limit), I must ensure Kafka is 
>>>>>> highly
>>>>>> available.  There's a small a chance that the message deletion 
>>>>>> rate is
>>>>>> the
>>>>>> same rate that receive rate.  For example, when the incoming 
>>>>>> volume is
>>>>>> so
>>>>>> high that the size threshold is reached before the time threshold.
>>>>>>   But, I
>>>>>> may be ok with that because if Kafka goes down, it can cause 
>>>>>> upstream
>>>>>> applications to fail.  This can result in higher losses overall, and
>>>>>> particularly of the most *recent* messages.
>>>>>>
>>>>>> In other words, in a persistent but ephemeral message queue, I would
>>>>>> give
>>>>>> higher precedence to recent messages over older ones. On the flip
>>>>>> side, by
>>>>>> allowing Kafka to go down when a disk is full, applications are 
>>>>>> forced
>>>>>> to
>>>>>> deal with the issue.  This adds complexity to apps, but perhaps it's
>>>>>> not a
>>>>>> bad thing.  After all, in scalability, all apps should be 
>>>>>> designed to
>>>>>> handle failure.
>>>>>>
>>>>>> Having said that, next is to decide which messages to delete 
>>>>>> first.  I
>>>>>> believe that's a separate issue and has its own complexities, too.
>>>>>>
>>>>>> The main idea though is that a global knob would provide 
>>>>>> flexibility,
>>>>>> even if not used.  From an operation perspective, if we can't 
>>>>>> ensure HA
>>>>>> for
>>>>>> all applications/components, it would be good if we can for at least
>>>>>> some
>>>>>> of the core ones, like Kafka.  This is much easier said that done
>>>>>> though.
>>>>>>
>>>>>>
>>>>>> On May 5, 2014, at 9:16 AM, Jun Rao <ju...@gmail.com> wrote:
>>>>>>
>>>>>>    Yes, your understanding is correct. A global knob that controls
>>>>>> aggregate
>>>>>>
>>>>>>> log size may make sense. What would be the expected behavior 
>>>>>>> when that
>>>>>>> limit is reached? Would you reduce the retention uniformly 
>>>>>>> across all
>>>>>>> topics? Then, it just means that some of the logs may not be 
>>>>>>> retained
>>>>>>> as
>>>>>>> long as you want. Also, we need to think through what happens when
>>>>>>> every
>>>>>>> log has only 1 segment left and yet the total size still exceeds 
>>>>>>> the
>>>>>>> limit.
>>>>>>> Do we roll log segments early?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Jun
>>>>>>>
>>>>>>>
>>>>>>> On Sun, May 4, 2014 at 4:31 AM, vinh <vi...@loggly.com> wrote:
>>>>>>>
>>>>>>>    Thanks Jun.  So if I understand this correctly, there really 
>>>>>>> is no
>>>>>>>
>>>>>>>> master
>>>>>>>> property to control the total aggregate size of all Kafka data 
>>>>>>>> files
>>>>>>>> on
>>>>>>>> a
>>>>>>>> broker.
>>>>>>>>
>>>>>>>> log.retention.size and log.file.size are great for managing 
>>>>>>>> data at
>>>>>>>> the
>>>>>>>> application level.  In our case, application needs change 
>>>>>>>> frequently,
>>>>>>>> and
>>>>>>>> performance itself is an ever evolving feature. This means various
>>>>>>>> configs
>>>>>>>> are constantly changing, like topics, # of partitions, etc.
>>>>>>>>
>>>>>>>> What rarely changes though is provisioned hardware resources.  
>>>>>>>> So a
>>>>>>>> setting to control the total aggregate size of Kafka logs (or
>>>>>>>> persisted
>>>>>>>> data, for better clarity) would definitely simplify things at an
>>>>>>>> operational level, regardless what happens at the application 
>>>>>>>> level.
>>>>>>>>
>>>>>>>>
>>>>>>>> On May 2, 2014, at 7:49 AM, Jun Rao <ju...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>    log.retention.size controls the total size in a log dir (per
>>>>>>>>
>>>>>>>>> partition). log.file.size
>>>>>>>>> controls the size of each log segment in the log dir.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Jun
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, May 1, 2014 at 9:31 PM, vinh <vi...@loggly.com> wrote:
>>>>>>>>>
>>>>>>>>>    In the 0.7 docs, the description for log.retention.size and
>>>>>>>>> log.file.size
>>>>>>>>> sound very much the same.  In particular, that they apply to a 
>>>>>>>>> single
>>>>>>>>> log
>>>>>>>>> file (or log segment file).
>>>>>>>>>
>>>>>>>>>> http://kafka.apache.org/07/configuration.html
>>>>>>>>>>
>>>>>>>>>> I'm beginning to think there is no setting to control the max
>>>>>>>>>> aggregate
>>>>>>>>>> size of all logs.  If this is correct, what would be a good 
>>>>>>>>>> approach
>>>>>>>>>> to
>>>>>>>>>> enforce this requirement?  In my particular scenario, I have 
>>>>>>>>>> a lot
>>>>>>>>>> of
>>>>>>>>>>
>>>>>>>>>>   data
>>>>>>>>> being written to Kafka at a very high rate.  So a 1TB disk can 
>>>>>>>>> easily
>>>>>>>>>
>>>>>>>>>> be
>>>>>>>>>> filled up in 24hrs or so.  One option is to add more Kafka 
>>>>>>>>>> brokers
>>>>>>>>>> to
>>>>>>>>>>
>>>>>>>>>>   add
>>>>>>>>> more disk space to the pool, but I'd like to avoid that and 
>>>>>>>>> see if I
>>>>>>>>>
>>>>>>>>>> can
>>>>>>>>>> simply configure Kafka to not write more than 1TB aggregate.  
>>>>>>>>>> Else,
>>>>>>>>>>
>>>>>>>>>>   Kafka
>>>>>>>>> will OOM and kill itself, and possibly the crash the node itself
>>>>>>>>>
>>>>>>>>>> because
>>>>>>>>>> the disk is full.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On May 1, 2014, at 9:21 PM, vinh <vi...@loggly.com> wrote:
>>>>>>>>>>
>>>>>>>>>>    Using Kafka 0.7.2, I have the following in server.properties:
>>>>>>>>>>
>>>>>>>>>>> log.retention.hours=48
>>>>>>>>>>> log.retention.size=107374182400
>>>>>>>>>>> log.file.size=536870912
>>>>>>>>>>>
>>>>>>>>>>> My interpretation of this is:
>>>>>>>>>>> a) a single log segment file over 48hrs old will be deleted
>>>>>>>>>>> b) the total combined size of *all* logs is 100GB
>>>>>>>>>>> c) a single log segment file is limited to 500MB in size 
>>>>>>>>>>> before a
>>>>>>>>>>> new
>>>>>>>>>>>
>>>>>>>>>>>   segment file is spawned spawning a new segment file
>>>>>>>>>>   d) a "log file" can be composed of many "log segment files"
>>>>>>>>>>> But, even after setting the above, I find that the total 
>>>>>>>>>>> combined
>>>>>>>>>>> size
>>>>>>>>>>>
>>>>>>>>>>>   of all Kafka logs on disk is 200GB right now.  Isn't
>>>>>>>>>> log.retention.size
>>>>>>>>>> supposed to limit it to 100GB?  Am I missing something?  The 
>>>>>>>>>> docs
>>>>>>>>>> are
>>>>>>>>>>
>>>>>>>>>>   not
>>>>>>>>> really clear, especially when it comes to distinguishing 
>>>>>>>>> between a
>>>>>>>>> "log
>>>>>>>>>
>>>>>>>>>> file" and a "log segment file".
>>>>>>>>>>
>>>>>>>>>>   I have disk monitoring.  But like anything else in 
>>>>>>>>>> software, even
>>>>>>>>>>>   monitoring can fail. Via configuration, I'd like to make sure
>>>>>>>>>> that
>>>>>>>>>>
>>>>>>>>>>   Kafka
>>>>>>>>> does not write more than the available disk space.  Or 
>>>>>>>>> something like
>>>>>>>>>
>>>>>>>>>> log4j, where I can set a max number of log files and the max 
>>>>>>>>>> size
>>>>>>>>>> per
>>>>>>>>>>
>>>>>>>>>>   file,
>>>>>>>>> which essentially allows me to set a max aggregate size limit 
>>>>>>>>> across
>>>>>>>>>
>>>>>>>>>> all
>>>>>>>>>> logs.
>>>>>>>>>>
>>>>>>>>>>   Thanks,
>>>>>>>>>>> -Vinh
>>>>>>>>>>>
>>>>>>>>>>>
>

Re: log.retention.size

[András Serény <se...@gravityrd.com>]

Sorry for the delay on this.

Yes, that's right -- it'd be just another term in the chain of  'or' 
conditions. Currently it's <time limit> OR <size limit>. With the global 
condition, it would be
<time limit> OR <size limit> OR <global size limit>

In my view, that's fairly simple and intuitive, hence a fine piece of logic.

Regards,
András

On 5/27/2014 4:34 PM, Jun Rao wrote:
> For log.retention.bytes.per.topic and log.retention.hours.per.topic, the
> current interpretation is that those are tight bounds. In other words, only
> when those thresholds are violated, a segment is deleted. To further
> satisfy log.retention.bytes.global, the per topic thresholds may no longer
> be tight, i.e., we may need to delete a segment even when the per topic
> threshold is not violated.
>
> Thanks,
>
> Jun
>
>
> On Tue, May 27, 2014 at 12:22 AM, András Serény <sereny.andras@gravityrd.com
>> wrote:
>> No, I think more specific settings should get a chance first. I'm
>> suggesting that provided that there is a segment rolled for a topic, *any
>> *of log.retention.bytes.per.topic, log.retention.hours.per.topic, and a
>> future log.retention.bytes.global violation would cause segments to be
>> deleted.
>>
>> As far as I understand, the current logic says
>>
>> (1)
>> for each topic, if there is a segment already rolled {
>>      mark segments eligible for deletion due to
>> log.retention.hours.for.this.topic
>>      if log.retention.bytes.for.this.topic is still violated, mark
>> segments eligible for deletion due to log.retention.bytes.for.this.topic
>> }
>>
>> After this cleanup cycle, there could be another one,  taking into account
>> the global threshold. For instance, something along the lines of
>>
>> (2)
>> if after (1) log.retention.bytes.global is still violated, for each topic,
>> if there is a segment already rolled {
>>    calculate the required size for this topic (e.g. the proportional size,
>> or simply (full size - threshold)/#topics ?)
>>    mark segments exceeding the required size for deletion
>> }
>>
>> Regards,
>> András
>>
>>
>>
>> On 5/23/2014 4:46 PM, Jun Rao wrote:
>>
>>> Yes, that's possible. There is a default log.retention.bytes for every
>>> topic. By introducing a global threshold, we may have to delete data from
>>> logs whose size is smaller than log.retention.bytes. So, are you saying
>>> that the global threshold has precedence?
>>>
>>> Thanks,
>>>
>>> Jun
>>>
>>>
>>> On Fri, May 23, 2014 at 2:26 AM, András Serény
>>> <se...@gravityrd.com>wrote:
>>>
>>>   Hi Kafka users,
>>>> this feature would also be very useful for us. With lots of topics of
>>>> different volume (and as they grow in number) it could become tedious to
>>>> maintain topic level settings.
>>>>
>>>> As a start, I think uniform reduction is a good idea. Logs wouldn't be
>>>> retained as long as you want, but that's already the case when a
>>>> log.retention.bytes setting is specified. As for early rolling, I don't
>>>> think it's necessary: currently, if there is no log segment eligible for
>>>> deletion, log.retention.bytes and log.retention.hours settings won't kick
>>>> in, so it's possible to exceed these limits, which is completely fine
>>>> (please correct me if I'm mistaken here).
>>>>
>>>> All in all, introducing a global threshold doesn't seem to induce a
>>>> considerable change in current retention logic.
>>>>
>>>> Regards,
>>>> András
>>>>
>>>>
>>>> On 5/8/2014 2:00 AM, vinh wrote:
>>>>
>>>>   Agreed…a global knob is a bit tricky for exactly the reason you've
>>>>> identified.  Perhaps the problem could be simplified though by
>>>>> considering
>>>>> the context and purpose of Kafka.  I would use a persistent message
>>>>> queue
>>>>> because I want to guarantee that data/messages don't get lost.  But,
>>>>> since
>>>>> Kafka is not meant to be a long term storage solution (other products
>>>>> can
>>>>> be used for that), I would clarify that guarantee to apply only to the
>>>>> most
>>>>> recent messages up until a certain configured threshold (i.e. max 24
>>>>> hrs,
>>>>> max 500GB, etc).  Once those thresholds are reached, old messages are
>>>>> deleted first.
>>>>>
>>>>> To ensure no message loss (up to a limit), I must ensure Kafka is highly
>>>>> available.  There's a small a chance that the message deletion rate is
>>>>> the
>>>>> same rate that receive rate.  For example, when the incoming volume is
>>>>> so
>>>>> high that the size threshold is reached before the time threshold.
>>>>>   But, I
>>>>> may be ok with that because if Kafka goes down, it can cause upstream
>>>>> applications to fail.  This can result in higher losses overall, and
>>>>> particularly of the most *recent* messages.
>>>>>
>>>>> In other words, in a persistent but ephemeral message queue, I would
>>>>> give
>>>>> higher precedence to recent messages over older ones.  On the flip
>>>>> side, by
>>>>> allowing Kafka to go down when a disk is full, applications are forced
>>>>> to
>>>>> deal with the issue.  This adds complexity to apps, but perhaps it's
>>>>> not a
>>>>> bad thing.  After all, in scalability, all apps should be designed to
>>>>> handle failure.
>>>>>
>>>>> Having said that, next is to decide which messages to delete first.  I
>>>>> believe that's a separate issue and has its own complexities, too.
>>>>>
>>>>> The main idea though is that a global knob would provide flexibility,
>>>>> even if not used.  From an operation perspective, if we can't ensure HA
>>>>> for
>>>>> all applications/components, it would be good if we can for at least
>>>>> some
>>>>> of the core ones, like Kafka.  This is much easier said that done
>>>>> though.
>>>>>
>>>>>
>>>>> On May 5, 2014, at 9:16 AM, Jun Rao <ju...@gmail.com> wrote:
>>>>>
>>>>>    Yes, your understanding is correct. A global knob that controls
>>>>> aggregate
>>>>>
>>>>>> log size may make sense. What would be the expected behavior when that
>>>>>> limit is reached? Would you reduce the retention uniformly across all
>>>>>> topics? Then, it just means that some of the logs may not be retained
>>>>>> as
>>>>>> long as you want. Also, we need to think through what happens when
>>>>>> every
>>>>>> log has only 1 segment left and yet the total size still exceeds the
>>>>>> limit.
>>>>>> Do we roll log segments early?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Jun
>>>>>>
>>>>>>
>>>>>> On Sun, May 4, 2014 at 4:31 AM, vinh <vi...@loggly.com> wrote:
>>>>>>
>>>>>>    Thanks Jun.  So if I understand this correctly, there really is no
>>>>>>
>>>>>>> master
>>>>>>> property to control the total aggregate size of all Kafka data files
>>>>>>> on
>>>>>>> a
>>>>>>> broker.
>>>>>>>
>>>>>>> log.retention.size and log.file.size are great for managing data at
>>>>>>> the
>>>>>>> application level.  In our case, application needs change frequently,
>>>>>>> and
>>>>>>> performance itself is an ever evolving feature.  This means various
>>>>>>> configs
>>>>>>> are constantly changing, like topics, # of partitions, etc.
>>>>>>>
>>>>>>> What rarely changes though is provisioned hardware resources.  So a
>>>>>>> setting to control the total aggregate size of Kafka logs (or
>>>>>>> persisted
>>>>>>> data, for better clarity) would definitely simplify things at an
>>>>>>> operational level, regardless what happens at the application level.
>>>>>>>
>>>>>>>
>>>>>>> On May 2, 2014, at 7:49 AM, Jun Rao <ju...@gmail.com> wrote:
>>>>>>>
>>>>>>>    log.retention.size controls the total size in a log dir (per
>>>>>>>
>>>>>>>> partition). log.file.size
>>>>>>>> controls the size of each log segment in the log dir.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Jun
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, May 1, 2014 at 9:31 PM, vinh <vi...@loggly.com> wrote:
>>>>>>>>
>>>>>>>>    In the 0.7 docs, the description for log.retention.size and
>>>>>>>> log.file.size
>>>>>>>> sound very much the same.  In particular, that they apply to a single
>>>>>>>> log
>>>>>>>> file (or log segment file).
>>>>>>>>
>>>>>>>>> http://kafka.apache.org/07/configuration.html
>>>>>>>>>
>>>>>>>>> I'm beginning to think there is no setting to control the max
>>>>>>>>> aggregate
>>>>>>>>> size of all logs.  If this is correct, what would be a good approach
>>>>>>>>> to
>>>>>>>>> enforce this requirement?  In my particular scenario, I have a lot
>>>>>>>>> of
>>>>>>>>>
>>>>>>>>>   data
>>>>>>>> being written to Kafka at a very high rate.  So a 1TB disk can easily
>>>>>>>>
>>>>>>>>> be
>>>>>>>>> filled up in 24hrs or so.  One option is to add more Kafka brokers
>>>>>>>>> to
>>>>>>>>>
>>>>>>>>>   add
>>>>>>>> more disk space to the pool, but I'd like to avoid that and see if I
>>>>>>>>
>>>>>>>>> can
>>>>>>>>> simply configure Kafka to not write more than 1TB aggregate.  Else,
>>>>>>>>>
>>>>>>>>>   Kafka
>>>>>>>> will OOM and kill itself, and possibly the crash the node itself
>>>>>>>>
>>>>>>>>> because
>>>>>>>>> the disk is full.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On May 1, 2014, at 9:21 PM, vinh <vi...@loggly.com> wrote:
>>>>>>>>>
>>>>>>>>>    Using Kafka 0.7.2, I have the following in server.properties:
>>>>>>>>>
>>>>>>>>>> log.retention.hours=48
>>>>>>>>>> log.retention.size=107374182400
>>>>>>>>>> log.file.size=536870912
>>>>>>>>>>
>>>>>>>>>> My interpretation of this is:
>>>>>>>>>> a) a single log segment file over 48hrs old will be deleted
>>>>>>>>>> b) the total combined size of *all* logs is 100GB
>>>>>>>>>> c) a single log segment file is limited to 500MB in size before a
>>>>>>>>>> new
>>>>>>>>>>
>>>>>>>>>>   segment file is spawned spawning a new segment file
>>>>>>>>>   d) a "log file" can be composed of many "log segment files"
>>>>>>>>>> But, even after setting the above, I find that the total combined
>>>>>>>>>> size
>>>>>>>>>>
>>>>>>>>>>   of all Kafka logs on disk is 200GB right now.  Isn't
>>>>>>>>> log.retention.size
>>>>>>>>> supposed to limit it to 100GB?  Am I missing something?  The docs
>>>>>>>>> are
>>>>>>>>>
>>>>>>>>>   not
>>>>>>>> really clear, especially when it comes to distinguishing between a
>>>>>>>> "log
>>>>>>>>
>>>>>>>>> file" and a "log segment file".
>>>>>>>>>
>>>>>>>>>   I have disk monitoring.  But like anything else in software, even
>>>>>>>>>>   monitoring can fail.  Via configuration, I'd like to make sure
>>>>>>>>> that
>>>>>>>>>
>>>>>>>>>   Kafka
>>>>>>>> does not write more than the available disk space.  Or something like
>>>>>>>>
>>>>>>>>> log4j, where I can set a max number of log files and the max size
>>>>>>>>> per
>>>>>>>>>
>>>>>>>>>   file,
>>>>>>>> which essentially allows me to set a max aggregate size limit across
>>>>>>>>
>>>>>>>>> all
>>>>>>>>> logs.
>>>>>>>>>
>>>>>>>>>   Thanks,
>>>>>>>>>> -Vinh
>>>>>>>>>>
>>>>>>>>>>

Re: log.retention.size

[Jun Rao <ju...@gmail.com>]

For log.retention.bytes.per.topic and log.retention.hours.per.topic, the
current interpretation is that those are tight bounds. In other words, only
when those thresholds are violated, a segment is deleted. To further
satisfy log.retention.bytes.global, the per topic thresholds may no longer
be tight, i.e., we may need to delete a segment even when the per topic
threshold is not violated.

Thanks,

Jun


On Tue, May 27, 2014 at 12:22 AM, András Serény <sereny.andras@gravityrd.com
> wrote:

>
> No, I think more specific settings should get a chance first. I'm
> suggesting that provided that there is a segment rolled for a topic, *any
> *of log.retention.bytes.per.topic, log.retention.hours.per.topic, and a
> future log.retention.bytes.global violation would cause segments to be
> deleted.
>
> As far as I understand, the current logic says
>
> (1)
> for each topic, if there is a segment already rolled {
>     mark segments eligible for deletion due to
> log.retention.hours.for.this.topic
>     if log.retention.bytes.for.this.topic is still violated, mark
> segments eligible for deletion due to log.retention.bytes.for.this.topic
> }
>
> After this cleanup cycle, there could be another one,  taking into account
> the global threshold. For instance, something along the lines of
>
> (2)
> if after (1) log.retention.bytes.global is still violated, for each topic,
> if there is a segment already rolled {
>   calculate the required size for this topic (e.g. the proportional size,
> or simply (full size - threshold)/#topics ?)
>   mark segments exceeding the required size for deletion
> }
>
> Regards,
> András
>
>
>
> On 5/23/2014 4:46 PM, Jun Rao wrote:
>
>> Yes, that's possible. There is a default log.retention.bytes for every
>> topic. By introducing a global threshold, we may have to delete data from
>> logs whose size is smaller than log.retention.bytes. So, are you saying
>> that the global threshold has precedence?
>>
>> Thanks,
>>
>> Jun
>>
>>
>> On Fri, May 23, 2014 at 2:26 AM, András Serény
>> <se...@gravityrd.com>wrote:
>>
>>  Hi Kafka users,
>>>
>>> this feature would also be very useful for us. With lots of topics of
>>> different volume (and as they grow in number) it could become tedious to
>>> maintain topic level settings.
>>>
>>> As a start, I think uniform reduction is a good idea. Logs wouldn't be
>>> retained as long as you want, but that's already the case when a
>>> log.retention.bytes setting is specified. As for early rolling, I don't
>>> think it's necessary: currently, if there is no log segment eligible for
>>> deletion, log.retention.bytes and log.retention.hours settings won't kick
>>> in, so it's possible to exceed these limits, which is completely fine
>>> (please correct me if I'm mistaken here).
>>>
>>> All in all, introducing a global threshold doesn't seem to induce a
>>> considerable change in current retention logic.
>>>
>>> Regards,
>>> András
>>>
>>>
>>> On 5/8/2014 2:00 AM, vinh wrote:
>>>
>>>  Agreed…a global knob is a bit tricky for exactly the reason you've
>>>> identified.  Perhaps the problem could be simplified though by
>>>> considering
>>>> the context and purpose of Kafka.  I would use a persistent message
>>>> queue
>>>> because I want to guarantee that data/messages don't get lost.  But,
>>>> since
>>>> Kafka is not meant to be a long term storage solution (other products
>>>> can
>>>> be used for that), I would clarify that guarantee to apply only to the
>>>> most
>>>> recent messages up until a certain configured threshold (i.e. max 24
>>>> hrs,
>>>> max 500GB, etc).  Once those thresholds are reached, old messages are
>>>> deleted first.
>>>>
>>>> To ensure no message loss (up to a limit), I must ensure Kafka is highly
>>>> available.  There's a small a chance that the message deletion rate is
>>>> the
>>>> same rate that receive rate.  For example, when the incoming volume is
>>>> so
>>>> high that the size threshold is reached before the time threshold.
>>>>  But, I
>>>> may be ok with that because if Kafka goes down, it can cause upstream
>>>> applications to fail.  This can result in higher losses overall, and
>>>> particularly of the most *recent* messages.
>>>>
>>>> In other words, in a persistent but ephemeral message queue, I would
>>>> give
>>>> higher precedence to recent messages over older ones.  On the flip
>>>> side, by
>>>> allowing Kafka to go down when a disk is full, applications are forced
>>>> to
>>>> deal with the issue.  This adds complexity to apps, but perhaps it's
>>>> not a
>>>> bad thing.  After all, in scalability, all apps should be designed to
>>>> handle failure.
>>>>
>>>> Having said that, next is to decide which messages to delete first.  I
>>>> believe that's a separate issue and has its own complexities, too.
>>>>
>>>> The main idea though is that a global knob would provide flexibility,
>>>> even if not used.  From an operation perspective, if we can't ensure HA
>>>> for
>>>> all applications/components, it would be good if we can for at least
>>>> some
>>>> of the core ones, like Kafka.  This is much easier said that done
>>>> though.
>>>>
>>>>
>>>> On May 5, 2014, at 9:16 AM, Jun Rao <ju...@gmail.com> wrote:
>>>>
>>>>   Yes, your understanding is correct. A global knob that controls
>>>> aggregate
>>>>
>>>>> log size may make sense. What would be the expected behavior when that
>>>>> limit is reached? Would you reduce the retention uniformly across all
>>>>> topics? Then, it just means that some of the logs may not be retained
>>>>> as
>>>>> long as you want. Also, we need to think through what happens when
>>>>> every
>>>>> log has only 1 segment left and yet the total size still exceeds the
>>>>> limit.
>>>>> Do we roll log segments early?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jun
>>>>>
>>>>>
>>>>> On Sun, May 4, 2014 at 4:31 AM, vinh <vi...@loggly.com> wrote:
>>>>>
>>>>>   Thanks Jun.  So if I understand this correctly, there really is no
>>>>>
>>>>>> master
>>>>>> property to control the total aggregate size of all Kafka data files
>>>>>> on
>>>>>> a
>>>>>> broker.
>>>>>>
>>>>>> log.retention.size and log.file.size are great for managing data at
>>>>>> the
>>>>>> application level.  In our case, application needs change frequently,
>>>>>> and
>>>>>> performance itself is an ever evolving feature.  This means various
>>>>>> configs
>>>>>> are constantly changing, like topics, # of partitions, etc.
>>>>>>
>>>>>> What rarely changes though is provisioned hardware resources.  So a
>>>>>> setting to control the total aggregate size of Kafka logs (or
>>>>>> persisted
>>>>>> data, for better clarity) would definitely simplify things at an
>>>>>> operational level, regardless what happens at the application level.
>>>>>>
>>>>>>
>>>>>> On May 2, 2014, at 7:49 AM, Jun Rao <ju...@gmail.com> wrote:
>>>>>>
>>>>>>   log.retention.size controls the total size in a log dir (per
>>>>>>
>>>>>>> partition). log.file.size
>>>>>>> controls the size of each log segment in the log dir.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Jun
>>>>>>>
>>>>>>>
>>>>>>> On Thu, May 1, 2014 at 9:31 PM, vinh <vi...@loggly.com> wrote:
>>>>>>>
>>>>>>>   In the 0.7 docs, the description for log.retention.size and
>>>>>>> log.file.size
>>>>>>> sound very much the same.  In particular, that they apply to a single
>>>>>>> log
>>>>>>> file (or log segment file).
>>>>>>>
>>>>>>>> http://kafka.apache.org/07/configuration.html
>>>>>>>>
>>>>>>>> I'm beginning to think there is no setting to control the max
>>>>>>>> aggregate
>>>>>>>> size of all logs.  If this is correct, what would be a good approach
>>>>>>>> to
>>>>>>>> enforce this requirement?  In my particular scenario, I have a lot
>>>>>>>> of
>>>>>>>>
>>>>>>>>  data
>>>>>>> being written to Kafka at a very high rate.  So a 1TB disk can easily
>>>>>>>
>>>>>>>> be
>>>>>>>> filled up in 24hrs or so.  One option is to add more Kafka brokers
>>>>>>>> to
>>>>>>>>
>>>>>>>>  add
>>>>>>> more disk space to the pool, but I'd like to avoid that and see if I
>>>>>>>
>>>>>>>> can
>>>>>>>> simply configure Kafka to not write more than 1TB aggregate.  Else,
>>>>>>>>
>>>>>>>>  Kafka
>>>>>>> will OOM and kill itself, and possibly the crash the node itself
>>>>>>>
>>>>>>>> because
>>>>>>>> the disk is full.
>>>>>>>>
>>>>>>>>
>>>>>>>> On May 1, 2014, at 9:21 PM, vinh <vi...@loggly.com> wrote:
>>>>>>>>
>>>>>>>>   Using Kafka 0.7.2, I have the following in server.properties:
>>>>>>>>
>>>>>>>>> log.retention.hours=48
>>>>>>>>> log.retention.size=107374182400
>>>>>>>>> log.file.size=536870912
>>>>>>>>>
>>>>>>>>> My interpretation of this is:
>>>>>>>>> a) a single log segment file over 48hrs old will be deleted
>>>>>>>>> b) the total combined size of *all* logs is 100GB
>>>>>>>>> c) a single log segment file is limited to 500MB in size before a
>>>>>>>>> new
>>>>>>>>>
>>>>>>>>>  segment file is spawned spawning a new segment file
>>>>>>>>
>>>>>>>>  d) a "log file" can be composed of many "log segment files"
>>>>>>>>>
>>>>>>>>> But, even after setting the above, I find that the total combined
>>>>>>>>> size
>>>>>>>>>
>>>>>>>>>  of all Kafka logs on disk is 200GB right now.  Isn't
>>>>>>>> log.retention.size
>>>>>>>> supposed to limit it to 100GB?  Am I missing something?  The docs
>>>>>>>> are
>>>>>>>>
>>>>>>>>  not
>>>>>>> really clear, especially when it comes to distinguishing between a
>>>>>>> "log
>>>>>>>
>>>>>>>> file" and a "log segment file".
>>>>>>>>
>>>>>>>>  I have disk monitoring.  But like anything else in software, even
>>>>>>>>>
>>>>>>>>>  monitoring can fail.  Via configuration, I'd like to make sure
>>>>>>>> that
>>>>>>>>
>>>>>>>>  Kafka
>>>>>>> does not write more than the available disk space.  Or something like
>>>>>>>
>>>>>>>> log4j, where I can set a max number of log files and the max size
>>>>>>>> per
>>>>>>>>
>>>>>>>>  file,
>>>>>>> which essentially allows me to set a max aggregate size limit across
>>>>>>>
>>>>>>>> all
>>>>>>>> logs.
>>>>>>>>
>>>>>>>>  Thanks,
>>>>>>>>> -Vinh
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>

Re: log.retention.size

[András Serény <se...@gravityrd.com>]

No, I think more specific settings should get a chance first. I'm 
suggesting that provided that there is a segment rolled for a topic, 
*any *of log.retention.bytes.per.topic, log.retention.hours.per.topic, 
and a future log.retention.bytes.global violation would cause segments 
to be deleted.

As far as I understand, the current logic says

(1)
for each topic, if there is a segment already rolled {
     mark segments eligible for deletion due to 
log.retention.hours.for.this.topic
     if log.retention.bytes.for.this.topic is still violated, mark 
segments eligible for deletion due to log.retention.bytes.for.this.topic
}

After this cleanup cycle, there could be another one,  taking into 
account the global threshold. For instance, something along the lines of

(2)
if after (1) log.retention.bytes.global is still violated, for each 
topic, if there is a segment already rolled {
   calculate the required size for this topic (e.g. the proportional 
size, or simply (full size - threshold)/#topics ?)
   mark segments exceeding the required size for deletion
}

Regards,
András


On 5/23/2014 4:46 PM, Jun Rao wrote:
> Yes, that's possible. There is a default log.retention.bytes for every
> topic. By introducing a global threshold, we may have to delete data from
> logs whose size is smaller than log.retention.bytes. So, are you saying
> that the global threshold has precedence?
>
> Thanks,
>
> Jun
>
>
> On Fri, May 23, 2014 at 2:26 AM, András Serény
> <se...@gravityrd.com>wrote:
>
>> Hi Kafka users,
>>
>> this feature would also be very useful for us. With lots of topics of
>> different volume (and as they grow in number) it could become tedious to
>> maintain topic level settings.
>>
>> As a start, I think uniform reduction is a good idea. Logs wouldn't be
>> retained as long as you want, but that's already the case when a
>> log.retention.bytes setting is specified. As for early rolling, I don't
>> think it's necessary: currently, if there is no log segment eligible for
>> deletion, log.retention.bytes and log.retention.hours settings won't kick
>> in, so it's possible to exceed these limits, which is completely fine
>> (please correct me if I'm mistaken here).
>>
>> All in all, introducing a global threshold doesn't seem to induce a
>> considerable change in current retention logic.
>>
>> Regards,
>> András
>>
>>
>> On 5/8/2014 2:00 AM, vinh wrote:
>>
>>> Agreed…a global knob is a bit tricky for exactly the reason you've
>>> identified.  Perhaps the problem could be simplified though by considering
>>> the context and purpose of Kafka.  I would use a persistent message queue
>>> because I want to guarantee that data/messages don't get lost.  But, since
>>> Kafka is not meant to be a long term storage solution (other products can
>>> be used for that), I would clarify that guarantee to apply only to the most
>>> recent messages up until a certain configured threshold (i.e. max 24 hrs,
>>> max 500GB, etc).  Once those thresholds are reached, old messages are
>>> deleted first.
>>>
>>> To ensure no message loss (up to a limit), I must ensure Kafka is highly
>>> available.  There's a small a chance that the message deletion rate is the
>>> same rate that receive rate.  For example, when the incoming volume is so
>>> high that the size threshold is reached before the time threshold.  But, I
>>> may be ok with that because if Kafka goes down, it can cause upstream
>>> applications to fail.  This can result in higher losses overall, and
>>> particularly of the most *recent* messages.
>>>
>>> In other words, in a persistent but ephemeral message queue, I would give
>>> higher precedence to recent messages over older ones.  On the flip side, by
>>> allowing Kafka to go down when a disk is full, applications are forced to
>>> deal with the issue.  This adds complexity to apps, but perhaps it's not a
>>> bad thing.  After all, in scalability, all apps should be designed to
>>> handle failure.
>>>
>>> Having said that, next is to decide which messages to delete first.  I
>>> believe that's a separate issue and has its own complexities, too.
>>>
>>> The main idea though is that a global knob would provide flexibility,
>>> even if not used.  From an operation perspective, if we can't ensure HA for
>>> all applications/components, it would be good if we can for at least some
>>> of the core ones, like Kafka.  This is much easier said that done though.
>>>
>>>
>>> On May 5, 2014, at 9:16 AM, Jun Rao <ju...@gmail.com> wrote:
>>>
>>>   Yes, your understanding is correct. A global knob that controls aggregate
>>>> log size may make sense. What would be the expected behavior when that
>>>> limit is reached? Would you reduce the retention uniformly across all
>>>> topics? Then, it just means that some of the logs may not be retained as
>>>> long as you want. Also, we need to think through what happens when every
>>>> log has only 1 segment left and yet the total size still exceeds the
>>>> limit.
>>>> Do we roll log segments early?
>>>>
>>>> Thanks,
>>>>
>>>> Jun
>>>>
>>>>
>>>> On Sun, May 4, 2014 at 4:31 AM, vinh <vi...@loggly.com> wrote:
>>>>
>>>>   Thanks Jun.  So if I understand this correctly, there really is no
>>>>> master
>>>>> property to control the total aggregate size of all Kafka data files on
>>>>> a
>>>>> broker.
>>>>>
>>>>> log.retention.size and log.file.size are great for managing data at the
>>>>> application level.  In our case, application needs change frequently,
>>>>> and
>>>>> performance itself is an ever evolving feature.  This means various
>>>>> configs
>>>>> are constantly changing, like topics, # of partitions, etc.
>>>>>
>>>>> What rarely changes though is provisioned hardware resources.  So a
>>>>> setting to control the total aggregate size of Kafka logs (or persisted
>>>>> data, for better clarity) would definitely simplify things at an
>>>>> operational level, regardless what happens at the application level.
>>>>>
>>>>>
>>>>> On May 2, 2014, at 7:49 AM, Jun Rao <ju...@gmail.com> wrote:
>>>>>
>>>>>   log.retention.size controls the total size in a log dir (per
>>>>>> partition). log.file.size
>>>>>> controls the size of each log segment in the log dir.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Jun
>>>>>>
>>>>>>
>>>>>> On Thu, May 1, 2014 at 9:31 PM, vinh <vi...@loggly.com> wrote:
>>>>>>
>>>>>>   In the 0.7 docs, the description for log.retention.size and
>>>>>> log.file.size
>>>>>> sound very much the same.  In particular, that they apply to a single
>>>>>> log
>>>>>> file (or log segment file).
>>>>>>> http://kafka.apache.org/07/configuration.html
>>>>>>>
>>>>>>> I'm beginning to think there is no setting to control the max
>>>>>>> aggregate
>>>>>>> size of all logs.  If this is correct, what would be a good approach
>>>>>>> to
>>>>>>> enforce this requirement?  In my particular scenario, I have a lot of
>>>>>>>
>>>>>> data
>>>>>> being written to Kafka at a very high rate.  So a 1TB disk can easily
>>>>>>> be
>>>>>>> filled up in 24hrs or so.  One option is to add more Kafka brokers to
>>>>>>>
>>>>>> add
>>>>>> more disk space to the pool, but I'd like to avoid that and see if I
>>>>>>> can
>>>>>>> simply configure Kafka to not write more than 1TB aggregate.  Else,
>>>>>>>
>>>>>> Kafka
>>>>>> will OOM and kill itself, and possibly the crash the node itself
>>>>>>> because
>>>>>>> the disk is full.
>>>>>>>
>>>>>>>
>>>>>>> On May 1, 2014, at 9:21 PM, vinh <vi...@loggly.com> wrote:
>>>>>>>
>>>>>>>   Using Kafka 0.7.2, I have the following in server.properties:
>>>>>>>> log.retention.hours=48
>>>>>>>> log.retention.size=107374182400
>>>>>>>> log.file.size=536870912
>>>>>>>>
>>>>>>>> My interpretation of this is:
>>>>>>>> a) a single log segment file over 48hrs old will be deleted
>>>>>>>> b) the total combined size of *all* logs is 100GB
>>>>>>>> c) a single log segment file is limited to 500MB in size before a new
>>>>>>>>
>>>>>>> segment file is spawned spawning a new segment file
>>>>>>>
>>>>>>>> d) a "log file" can be composed of many "log segment files"
>>>>>>>>
>>>>>>>> But, even after setting the above, I find that the total combined
>>>>>>>> size
>>>>>>>>
>>>>>>> of all Kafka logs on disk is 200GB right now.  Isn't
>>>>>>> log.retention.size
>>>>>>> supposed to limit it to 100GB?  Am I missing something?  The docs are
>>>>>>>
>>>>>> not
>>>>>> really clear, especially when it comes to distinguishing between a "log
>>>>>>> file" and a "log segment file".
>>>>>>>
>>>>>>>> I have disk monitoring.  But like anything else in software, even
>>>>>>>>
>>>>>>> monitoring can fail.  Via configuration, I'd like to make sure that
>>>>>>>
>>>>>> Kafka
>>>>>> does not write more than the available disk space.  Or something like
>>>>>>> log4j, where I can set a max number of log files and the max size per
>>>>>>>
>>>>>> file,
>>>>>> which essentially allows me to set a max aggregate size limit across
>>>>>>> all
>>>>>>> logs.
>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> -Vinh
>>>>>>>>
>>>>>>>

Re: log.retention.size

[Jun Rao <ju...@gmail.com>]

Yes, that's possible. There is a default log.retention.bytes for every
topic. By introducing a global threshold, we may have to delete data from
logs whose size is smaller than log.retention.bytes. So, are you saying
that the global threshold has precedence?

Thanks,

Jun


On Fri, May 23, 2014 at 2:26 AM, András Serény
<se...@gravityrd.com>wrote:

>
> Hi Kafka users,
>
> this feature would also be very useful for us. With lots of topics of
> different volume (and as they grow in number) it could become tedious to
> maintain topic level settings.
>
> As a start, I think uniform reduction is a good idea. Logs wouldn't be
> retained as long as you want, but that's already the case when a
> log.retention.bytes setting is specified. As for early rolling, I don't
> think it's necessary: currently, if there is no log segment eligible for
> deletion, log.retention.bytes and log.retention.hours settings won't kick
> in, so it's possible to exceed these limits, which is completely fine
> (please correct me if I'm mistaken here).
>
> All in all, introducing a global threshold doesn't seem to induce a
> considerable change in current retention logic.
>
> Regards,
> András
>
>
> On 5/8/2014 2:00 AM, vinh wrote:
>
>> Agreed…a global knob is a bit tricky for exactly the reason you've
>> identified.  Perhaps the problem could be simplified though by considering
>> the context and purpose of Kafka.  I would use a persistent message queue
>> because I want to guarantee that data/messages don't get lost.  But, since
>> Kafka is not meant to be a long term storage solution (other products can
>> be used for that), I would clarify that guarantee to apply only to the most
>> recent messages up until a certain configured threshold (i.e. max 24 hrs,
>> max 500GB, etc).  Once those thresholds are reached, old messages are
>> deleted first.
>>
>> To ensure no message loss (up to a limit), I must ensure Kafka is highly
>> available.  There's a small a chance that the message deletion rate is the
>> same rate that receive rate.  For example, when the incoming volume is so
>> high that the size threshold is reached before the time threshold.  But, I
>> may be ok with that because if Kafka goes down, it can cause upstream
>> applications to fail.  This can result in higher losses overall, and
>> particularly of the most *recent* messages.
>>
>> In other words, in a persistent but ephemeral message queue, I would give
>> higher precedence to recent messages over older ones.  On the flip side, by
>> allowing Kafka to go down when a disk is full, applications are forced to
>> deal with the issue.  This adds complexity to apps, but perhaps it's not a
>> bad thing.  After all, in scalability, all apps should be designed to
>> handle failure.
>>
>> Having said that, next is to decide which messages to delete first.  I
>> believe that's a separate issue and has its own complexities, too.
>>
>> The main idea though is that a global knob would provide flexibility,
>> even if not used.  From an operation perspective, if we can't ensure HA for
>> all applications/components, it would be good if we can for at least some
>> of the core ones, like Kafka.  This is much easier said that done though.
>>
>>
>> On May 5, 2014, at 9:16 AM, Jun Rao <ju...@gmail.com> wrote:
>>
>>  Yes, your understanding is correct. A global knob that controls aggregate
>>> log size may make sense. What would be the expected behavior when that
>>> limit is reached? Would you reduce the retention uniformly across all
>>> topics? Then, it just means that some of the logs may not be retained as
>>> long as you want. Also, we need to think through what happens when every
>>> log has only 1 segment left and yet the total size still exceeds the
>>> limit.
>>> Do we roll log segments early?
>>>
>>> Thanks,
>>>
>>> Jun
>>>
>>>
>>> On Sun, May 4, 2014 at 4:31 AM, vinh <vi...@loggly.com> wrote:
>>>
>>>  Thanks Jun.  So if I understand this correctly, there really is no
>>>> master
>>>> property to control the total aggregate size of all Kafka data files on
>>>> a
>>>> broker.
>>>>
>>>> log.retention.size and log.file.size are great for managing data at the
>>>> application level.  In our case, application needs change frequently,
>>>> and
>>>> performance itself is an ever evolving feature.  This means various
>>>> configs
>>>> are constantly changing, like topics, # of partitions, etc.
>>>>
>>>> What rarely changes though is provisioned hardware resources.  So a
>>>> setting to control the total aggregate size of Kafka logs (or persisted
>>>> data, for better clarity) would definitely simplify things at an
>>>> operational level, regardless what happens at the application level.
>>>>
>>>>
>>>> On May 2, 2014, at 7:49 AM, Jun Rao <ju...@gmail.com> wrote:
>>>>
>>>>  log.retention.size controls the total size in a log dir (per
>>>>> partition). log.file.size
>>>>> controls the size of each log segment in the log dir.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jun
>>>>>
>>>>>
>>>>> On Thu, May 1, 2014 at 9:31 PM, vinh <vi...@loggly.com> wrote:
>>>>>
>>>>>  In the 0.7 docs, the description for log.retention.size and
>>>>>>
>>>>> log.file.size
>>>>
>>>>> sound very much the same.  In particular, that they apply to a single
>>>>>>
>>>>> log
>>>>
>>>>> file (or log segment file).
>>>>>>
>>>>>> http://kafka.apache.org/07/configuration.html
>>>>>>
>>>>>> I'm beginning to think there is no setting to control the max
>>>>>> aggregate
>>>>>> size of all logs.  If this is correct, what would be a good approach
>>>>>> to
>>>>>> enforce this requirement?  In my particular scenario, I have a lot of
>>>>>>
>>>>> data
>>>>
>>>>> being written to Kafka at a very high rate.  So a 1TB disk can easily
>>>>>> be
>>>>>> filled up in 24hrs or so.  One option is to add more Kafka brokers to
>>>>>>
>>>>> add
>>>>
>>>>> more disk space to the pool, but I'd like to avoid that and see if I
>>>>>> can
>>>>>> simply configure Kafka to not write more than 1TB aggregate.  Else,
>>>>>>
>>>>> Kafka
>>>>
>>>>> will OOM and kill itself, and possibly the crash the node itself
>>>>>> because
>>>>>> the disk is full.
>>>>>>
>>>>>>
>>>>>> On May 1, 2014, at 9:21 PM, vinh <vi...@loggly.com> wrote:
>>>>>>
>>>>>>  Using Kafka 0.7.2, I have the following in server.properties:
>>>>>>>
>>>>>>> log.retention.hours=48
>>>>>>> log.retention.size=107374182400
>>>>>>> log.file.size=536870912
>>>>>>>
>>>>>>> My interpretation of this is:
>>>>>>> a) a single log segment file over 48hrs old will be deleted
>>>>>>> b) the total combined size of *all* logs is 100GB
>>>>>>> c) a single log segment file is limited to 500MB in size before a new
>>>>>>>
>>>>>> segment file is spawned spawning a new segment file
>>>>>>
>>>>>>> d) a "log file" can be composed of many "log segment files"
>>>>>>>
>>>>>>> But, even after setting the above, I find that the total combined
>>>>>>> size
>>>>>>>
>>>>>> of all Kafka logs on disk is 200GB right now.  Isn't
>>>>>> log.retention.size
>>>>>> supposed to limit it to 100GB?  Am I missing something?  The docs are
>>>>>>
>>>>> not
>>>>
>>>>> really clear, especially when it comes to distinguishing between a "log
>>>>>> file" and a "log segment file".
>>>>>>
>>>>>>> I have disk monitoring.  But like anything else in software, even
>>>>>>>
>>>>>> monitoring can fail.  Via configuration, I'd like to make sure that
>>>>>>
>>>>> Kafka
>>>>
>>>>> does not write more than the available disk space.  Or something like
>>>>>> log4j, where I can set a max number of log files and the max size per
>>>>>>
>>>>> file,
>>>>
>>>>> which essentially allows me to set a max aggregate size limit across
>>>>>> all
>>>>>> logs.
>>>>>>
>>>>>>> Thanks,
>>>>>>> -Vinh
>>>>>>>
>>>>>>
>>>>>>
>>>>
>

Re: log.retention.size

[András Serény <se...@gravityrd.com>]

Hi Kafka users,

this feature would also be very useful for us. With lots of topics of 
different volume (and as they grow in number) it could become tedious to 
maintain topic level settings.

As a start, I think uniform reduction is a good idea. Logs wouldn't be 
retained as long as you want, but that's already the case when a 
log.retention.bytes setting is specified. As for early rolling, I don't 
think it's necessary: currently, if there is no log segment eligible for 
deletion, log.retention.bytes and log.retention.hours settings won't 
kick in, so it's possible to exceed these limits, which is completely 
fine (please correct me if I'm mistaken here).

All in all, introducing a global threshold doesn't seem to induce a 
considerable change in current retention logic.

Regards,
András

On 5/8/2014 2:00 AM, vinh wrote:
> Agreed…a global knob is a bit tricky for exactly the reason you've identified.  Perhaps the problem could be simplified though by considering the context and purpose of Kafka.  I would use a persistent message queue because I want to guarantee that data/messages don't get lost.  But, since Kafka is not meant to be a long term storage solution (other products can be used for that), I would clarify that guarantee to apply only to the most recent messages up until a certain configured threshold (i.e. max 24 hrs, max 500GB, etc).  Once those thresholds are reached, old messages are deleted first.
>
> To ensure no message loss (up to a limit), I must ensure Kafka is highly available.  There's a small a chance that the message deletion rate is the same rate that receive rate.  For example, when the incoming volume is so high that the size threshold is reached before the time threshold.  But, I may be ok with that because if Kafka goes down, it can cause upstream applications to fail.  This can result in higher losses overall, and particularly of the most *recent* messages.
>
> In other words, in a persistent but ephemeral message queue, I would give higher precedence to recent messages over older ones.  On the flip side, by allowing Kafka to go down when a disk is full, applications are forced to deal with the issue.  This adds complexity to apps, but perhaps it's not a bad thing.  After all, in scalability, all apps should be designed to handle failure.
>
> Having said that, next is to decide which messages to delete first.  I believe that's a separate issue and has its own complexities, too.
>
> The main idea though is that a global knob would provide flexibility, even if not used.  From an operation perspective, if we can't ensure HA for all applications/components, it would be good if we can for at least some of the core ones, like Kafka.  This is much easier said that done though.
>
>
> On May 5, 2014, at 9:16 AM, Jun Rao <ju...@gmail.com> wrote:
>
>> Yes, your understanding is correct. A global knob that controls aggregate
>> log size may make sense. What would be the expected behavior when that
>> limit is reached? Would you reduce the retention uniformly across all
>> topics? Then, it just means that some of the logs may not be retained as
>> long as you want. Also, we need to think through what happens when every
>> log has only 1 segment left and yet the total size still exceeds the limit.
>> Do we roll log segments early?
>>
>> Thanks,
>>
>> Jun
>>
>>
>> On Sun, May 4, 2014 at 4:31 AM, vinh <vi...@loggly.com> wrote:
>>
>>> Thanks Jun.  So if I understand this correctly, there really is no master
>>> property to control the total aggregate size of all Kafka data files on a
>>> broker.
>>>
>>> log.retention.size and log.file.size are great for managing data at the
>>> application level.  In our case, application needs change frequently, and
>>> performance itself is an ever evolving feature.  This means various configs
>>> are constantly changing, like topics, # of partitions, etc.
>>>
>>> What rarely changes though is provisioned hardware resources.  So a
>>> setting to control the total aggregate size of Kafka logs (or persisted
>>> data, for better clarity) would definitely simplify things at an
>>> operational level, regardless what happens at the application level.
>>>
>>>
>>> On May 2, 2014, at 7:49 AM, Jun Rao <ju...@gmail.com> wrote:
>>>
>>>> log.retention.size controls the total size in a log dir (per
>>>> partition). log.file.size
>>>> controls the size of each log segment in the log dir.
>>>>
>>>> Thanks,
>>>>
>>>> Jun
>>>>
>>>>
>>>> On Thu, May 1, 2014 at 9:31 PM, vinh <vi...@loggly.com> wrote:
>>>>
>>>>> In the 0.7 docs, the description for log.retention.size and
>>> log.file.size
>>>>> sound very much the same.  In particular, that they apply to a single
>>> log
>>>>> file (or log segment file).
>>>>>
>>>>> http://kafka.apache.org/07/configuration.html
>>>>>
>>>>> I'm beginning to think there is no setting to control the max aggregate
>>>>> size of all logs.  If this is correct, what would be a good approach to
>>>>> enforce this requirement?  In my particular scenario, I have a lot of
>>> data
>>>>> being written to Kafka at a very high rate.  So a 1TB disk can easily be
>>>>> filled up in 24hrs or so.  One option is to add more Kafka brokers to
>>> add
>>>>> more disk space to the pool, but I'd like to avoid that and see if I can
>>>>> simply configure Kafka to not write more than 1TB aggregate.  Else,
>>> Kafka
>>>>> will OOM and kill itself, and possibly the crash the node itself because
>>>>> the disk is full.
>>>>>
>>>>>
>>>>> On May 1, 2014, at 9:21 PM, vinh <vi...@loggly.com> wrote:
>>>>>
>>>>>> Using Kafka 0.7.2, I have the following in server.properties:
>>>>>>
>>>>>> log.retention.hours=48
>>>>>> log.retention.size=107374182400
>>>>>> log.file.size=536870912
>>>>>>
>>>>>> My interpretation of this is:
>>>>>> a) a single log segment file over 48hrs old will be deleted
>>>>>> b) the total combined size of *all* logs is 100GB
>>>>>> c) a single log segment file is limited to 500MB in size before a new
>>>>> segment file is spawned spawning a new segment file
>>>>>> d) a "log file" can be composed of many "log segment files"
>>>>>>
>>>>>> But, even after setting the above, I find that the total combined size
>>>>> of all Kafka logs on disk is 200GB right now.  Isn't log.retention.size
>>>>> supposed to limit it to 100GB?  Am I missing something?  The docs are
>>> not
>>>>> really clear, especially when it comes to distinguishing between a "log
>>>>> file" and a "log segment file".
>>>>>> I have disk monitoring.  But like anything else in software, even
>>>>> monitoring can fail.  Via configuration, I'd like to make sure that
>>> Kafka
>>>>> does not write more than the available disk space.  Or something like
>>>>> log4j, where I can set a max number of log files and the max size per
>>> file,
>>>>> which essentially allows me to set a max aggregate size limit across all
>>>>> logs.
>>>>>> Thanks,
>>>>>> -Vinh
>>>>>
>>>

Re: log.retention.size

[vinh <vi...@loggly.com>]

Agreed…a global knob is a bit tricky for exactly the reason you've identified.  Perhaps the problem could be simplified though by considering the context and purpose of Kafka.  I would use a persistent message queue because I want to guarantee that data/messages don't get lost.  But, since Kafka is not meant to be a long term storage solution (other products can be used for that), I would clarify that guarantee to apply only to the most recent messages up until a certain configured threshold (i.e. max 24 hrs, max 500GB, etc).  Once those thresholds are reached, old messages are deleted first.

To ensure no message loss (up to a limit), I must ensure Kafka is highly available.  There's a small a chance that the message deletion rate is the same rate that receive rate.  For example, when the incoming volume is so high that the size threshold is reached before the time threshold.  But, I may be ok with that because if Kafka goes down, it can cause upstream applications to fail.  This can result in higher losses overall, and particularly of the most *recent* messages.

In other words, in a persistent but ephemeral message queue, I would give higher precedence to recent messages over older ones.  On the flip side, by allowing Kafka to go down when a disk is full, applications are forced to deal with the issue.  This adds complexity to apps, but perhaps it's not a bad thing.  After all, in scalability, all apps should be designed to handle failure.

Having said that, next is to decide which messages to delete first.  I believe that's a separate issue and has its own complexities, too.

The main idea though is that a global knob would provide flexibility, even if not used.  From an operation perspective, if we can't ensure HA for all applications/components, it would be good if we can for at least some of the core ones, like Kafka.  This is much easier said that done though.


On May 5, 2014, at 9:16 AM, Jun Rao <ju...@gmail.com> wrote:

> Yes, your understanding is correct. A global knob that controls aggregate
> log size may make sense. What would be the expected behavior when that
> limit is reached? Would you reduce the retention uniformly across all
> topics? Then, it just means that some of the logs may not be retained as
> long as you want. Also, we need to think through what happens when every
> log has only 1 segment left and yet the total size still exceeds the limit.
> Do we roll log segments early?
> 
> Thanks,
> 
> Jun
> 
> 
> On Sun, May 4, 2014 at 4:31 AM, vinh <vi...@loggly.com> wrote:
> 
>> Thanks Jun.  So if I understand this correctly, there really is no master
>> property to control the total aggregate size of all Kafka data files on a
>> broker.
>> 
>> log.retention.size and log.file.size are great for managing data at the
>> application level.  In our case, application needs change frequently, and
>> performance itself is an ever evolving feature.  This means various configs
>> are constantly changing, like topics, # of partitions, etc.
>> 
>> What rarely changes though is provisioned hardware resources.  So a
>> setting to control the total aggregate size of Kafka logs (or persisted
>> data, for better clarity) would definitely simplify things at an
>> operational level, regardless what happens at the application level.
>> 
>> 
>> On May 2, 2014, at 7:49 AM, Jun Rao <ju...@gmail.com> wrote:
>> 
>>> log.retention.size controls the total size in a log dir (per
>>> partition). log.file.size
>>> controls the size of each log segment in the log dir.
>>> 
>>> Thanks,
>>> 
>>> Jun
>>> 
>>> 
>>> On Thu, May 1, 2014 at 9:31 PM, vinh <vi...@loggly.com> wrote:
>>> 
>>>> In the 0.7 docs, the description for log.retention.size and
>> log.file.size
>>>> sound very much the same.  In particular, that they apply to a single
>> log
>>>> file (or log segment file).
>>>> 
>>>> http://kafka.apache.org/07/configuration.html
>>>> 
>>>> I'm beginning to think there is no setting to control the max aggregate
>>>> size of all logs.  If this is correct, what would be a good approach to
>>>> enforce this requirement?  In my particular scenario, I have a lot of
>> data
>>>> being written to Kafka at a very high rate.  So a 1TB disk can easily be
>>>> filled up in 24hrs or so.  One option is to add more Kafka brokers to
>> add
>>>> more disk space to the pool, but I'd like to avoid that and see if I can
>>>> simply configure Kafka to not write more than 1TB aggregate.  Else,
>> Kafka
>>>> will OOM and kill itself, and possibly the crash the node itself because
>>>> the disk is full.
>>>> 
>>>> 
>>>> On May 1, 2014, at 9:21 PM, vinh <vi...@loggly.com> wrote:
>>>> 
>>>>> Using Kafka 0.7.2, I have the following in server.properties:
>>>>> 
>>>>> log.retention.hours=48
>>>>> log.retention.size=107374182400
>>>>> log.file.size=536870912
>>>>> 
>>>>> My interpretation of this is:
>>>>> a) a single log segment file over 48hrs old will be deleted
>>>>> b) the total combined size of *all* logs is 100GB
>>>>> c) a single log segment file is limited to 500MB in size before a new
>>>> segment file is spawned spawning a new segment file
>>>>> d) a "log file" can be composed of many "log segment files"
>>>>> 
>>>>> But, even after setting the above, I find that the total combined size
>>>> of all Kafka logs on disk is 200GB right now.  Isn't log.retention.size
>>>> supposed to limit it to 100GB?  Am I missing something?  The docs are
>> not
>>>> really clear, especially when it comes to distinguishing between a "log
>>>> file" and a "log segment file".
>>>>> 
>>>>> I have disk monitoring.  But like anything else in software, even
>>>> monitoring can fail.  Via configuration, I'd like to make sure that
>> Kafka
>>>> does not write more than the available disk space.  Or something like
>>>> log4j, where I can set a max number of log files and the max size per
>> file,
>>>> which essentially allows me to set a max aggregate size limit across all
>>>> logs.
>>>>> 
>>>>> Thanks,
>>>>> -Vinh
>>>> 
>>>> 
>> 
>> 

Re: log.retention.size

[Jun Rao <ju...@gmail.com>]

Yes, your understanding is correct. A global knob that controls aggregate
log size may make sense. What would be the expected behavior when that
limit is reached? Would you reduce the retention uniformly across all
topics? Then, it just means that some of the logs may not be retained as
long as you want. Also, we need to think through what happens when every
log has only 1 segment left and yet the total size still exceeds the limit.
Do we roll log segments early?

Thanks,

Jun


On Sun, May 4, 2014 at 4:31 AM, vinh <vi...@loggly.com> wrote:

> Thanks Jun.  So if I understand this correctly, there really is no master
> property to control the total aggregate size of all Kafka data files on a
> broker.
>
> log.retention.size and log.file.size are great for managing data at the
> application level.  In our case, application needs change frequently, and
> performance itself is an ever evolving feature.  This means various configs
> are constantly changing, like topics, # of partitions, etc.
>
> What rarely changes though is provisioned hardware resources.  So a
> setting to control the total aggregate size of Kafka logs (or persisted
> data, for better clarity) would definitely simplify things at an
> operational level, regardless what happens at the application level.
>
>
> On May 2, 2014, at 7:49 AM, Jun Rao <ju...@gmail.com> wrote:
>
> > log.retention.size controls the total size in a log dir (per
> > partition). log.file.size
> > controls the size of each log segment in the log dir.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Thu, May 1, 2014 at 9:31 PM, vinh <vi...@loggly.com> wrote:
> >
> >> In the 0.7 docs, the description for log.retention.size and
> log.file.size
> >> sound very much the same.  In particular, that they apply to a single
> log
> >> file (or log segment file).
> >>
> >> http://kafka.apache.org/07/configuration.html
> >>
> >> I'm beginning to think there is no setting to control the max aggregate
> >> size of all logs.  If this is correct, what would be a good approach to
> >> enforce this requirement?  In my particular scenario, I have a lot of
> data
> >> being written to Kafka at a very high rate.  So a 1TB disk can easily be
> >> filled up in 24hrs or so.  One option is to add more Kafka brokers to
> add
> >> more disk space to the pool, but I'd like to avoid that and see if I can
> >> simply configure Kafka to not write more than 1TB aggregate.  Else,
> Kafka
> >> will OOM and kill itself, and possibly the crash the node itself because
> >> the disk is full.
> >>
> >>
> >> On May 1, 2014, at 9:21 PM, vinh <vi...@loggly.com> wrote:
> >>
> >>> Using Kafka 0.7.2, I have the following in server.properties:
> >>>
> >>> log.retention.hours=48
> >>> log.retention.size=107374182400
> >>> log.file.size=536870912
> >>>
> >>> My interpretation of this is:
> >>> a) a single log segment file over 48hrs old will be deleted
> >>> b) the total combined size of *all* logs is 100GB
> >>> c) a single log segment file is limited to 500MB in size before a new
> >> segment file is spawned spawning a new segment file
> >>> d) a "log file" can be composed of many "log segment files"
> >>>
> >>> But, even after setting the above, I find that the total combined size
> >> of all Kafka logs on disk is 200GB right now.  Isn't log.retention.size
> >> supposed to limit it to 100GB?  Am I missing something?  The docs are
> not
> >> really clear, especially when it comes to distinguishing between a "log
> >> file" and a "log segment file".
> >>>
> >>> I have disk monitoring.  But like anything else in software, even
> >> monitoring can fail.  Via configuration, I'd like to make sure that
> Kafka
> >> does not write more than the available disk space.  Or something like
> >> log4j, where I can set a max number of log files and the max size per
> file,
> >> which essentially allows me to set a max aggregate size limit across all
> >> logs.
> >>>
> >>> Thanks,
> >>> -Vinh
> >>
> >>
>
>

Re: log.retention.size

[vinh <vi...@loggly.com>]

Thanks Jun.  So if I understand this correctly, there really is no master property to control the total aggregate size of all Kafka data files on a broker.

log.retention.size and log.file.size are great for managing data at the application level.  In our case, application needs change frequently, and performance itself is an ever evolving feature.  This means various configs are constantly changing, like topics, # of partitions, etc.

What rarely changes though is provisioned hardware resources.  So a setting to control the total aggregate size of Kafka logs (or persisted data, for better clarity) would definitely simplify things at an operational level, regardless what happens at the application level.


On May 2, 2014, at 7:49 AM, Jun Rao <ju...@gmail.com> wrote:

> log.retention.size controls the total size in a log dir (per
> partition). log.file.size
> controls the size of each log segment in the log dir.
> 
> Thanks,
> 
> Jun
> 
> 
> On Thu, May 1, 2014 at 9:31 PM, vinh <vi...@loggly.com> wrote:
> 
>> In the 0.7 docs, the description for log.retention.size and log.file.size
>> sound very much the same.  In particular, that they apply to a single log
>> file (or log segment file).
>> 
>> http://kafka.apache.org/07/configuration.html
>> 
>> I'm beginning to think there is no setting to control the max aggregate
>> size of all logs.  If this is correct, what would be a good approach to
>> enforce this requirement?  In my particular scenario, I have a lot of data
>> being written to Kafka at a very high rate.  So a 1TB disk can easily be
>> filled up in 24hrs or so.  One option is to add more Kafka brokers to add
>> more disk space to the pool, but I'd like to avoid that and see if I can
>> simply configure Kafka to not write more than 1TB aggregate.  Else, Kafka
>> will OOM and kill itself, and possibly the crash the node itself because
>> the disk is full.
>> 
>> 
>> On May 1, 2014, at 9:21 PM, vinh <vi...@loggly.com> wrote:
>> 
>>> Using Kafka 0.7.2, I have the following in server.properties:
>>> 
>>> log.retention.hours=48
>>> log.retention.size=107374182400
>>> log.file.size=536870912
>>> 
>>> My interpretation of this is:
>>> a) a single log segment file over 48hrs old will be deleted
>>> b) the total combined size of *all* logs is 100GB
>>> c) a single log segment file is limited to 500MB in size before a new
>> segment file is spawned spawning a new segment file
>>> d) a "log file" can be composed of many "log segment files"
>>> 
>>> But, even after setting the above, I find that the total combined size
>> of all Kafka logs on disk is 200GB right now.  Isn't log.retention.size
>> supposed to limit it to 100GB?  Am I missing something?  The docs are not
>> really clear, especially when it comes to distinguishing between a "log
>> file" and a "log segment file".
>>> 
>>> I have disk monitoring.  But like anything else in software, even
>> monitoring can fail.  Via configuration, I'd like to make sure that Kafka
>> does not write more than the available disk space.  Or something like
>> log4j, where I can set a max number of log files and the max size per file,
>> which essentially allows me to set a max aggregate size limit across all
>> logs.
>>> 
>>> Thanks,
>>> -Vinh
>> 
>> 

Re: log.retention.size

[Jun Rao <ju...@gmail.com>]

log.retention.size controls the total size in a log dir (per
partition). log.file.size
controls the size of each log segment in the log dir.

Thanks,

Jun


On Thu, May 1, 2014 at 9:31 PM, vinh <vi...@loggly.com> wrote:

> In the 0.7 docs, the description for log.retention.size and log.file.size
> sound very much the same.  In particular, that they apply to a single log
> file (or log segment file).
>
> http://kafka.apache.org/07/configuration.html
>
> I'm beginning to think there is no setting to control the max aggregate
> size of all logs.  If this is correct, what would be a good approach to
> enforce this requirement?  In my particular scenario, I have a lot of data
> being written to Kafka at a very high rate.  So a 1TB disk can easily be
> filled up in 24hrs or so.  One option is to add more Kafka brokers to add
> more disk space to the pool, but I'd like to avoid that and see if I can
> simply configure Kafka to not write more than 1TB aggregate.  Else, Kafka
> will OOM and kill itself, and possibly the crash the node itself because
> the disk is full.
>
>
> On May 1, 2014, at 9:21 PM, vinh <vi...@loggly.com> wrote:
>
> > Using Kafka 0.7.2, I have the following in server.properties:
> >
> > log.retention.hours=48
> > log.retention.size=107374182400
> > log.file.size=536870912
> >
> > My interpretation of this is:
> > a) a single log segment file over 48hrs old will be deleted
> > b) the total combined size of *all* logs is 100GB
> > c) a single log segment file is limited to 500MB in size before a new
> segment file is spawned spawning a new segment file
> > d) a "log file" can be composed of many "log segment files"
> >
> > But, even after setting the above, I find that the total combined size
> of all Kafka logs on disk is 200GB right now.  Isn't log.retention.size
> supposed to limit it to 100GB?  Am I missing something?  The docs are not
> really clear, especially when it comes to distinguishing between a "log
> file" and a "log segment file".
> >
> > I have disk monitoring.  But like anything else in software, even
> monitoring can fail.  Via configuration, I'd like to make sure that Kafka
> does not write more than the available disk space.  Or something like
> log4j, where I can set a max number of log files and the max size per file,
> which essentially allows me to set a max aggregate size limit across all
> logs.
> >
> > Thanks,
> > -Vinh
>
>

Re: log.retention.size

[vinh <vi...@loggly.com>]

In the 0.7 docs, the description for log.retention.size and log.file.size sound very much the same.  In particular, that they apply to a single log file (or log segment file).

http://kafka.apache.org/07/configuration.html

I'm beginning to think there is no setting to control the max aggregate size of all logs.  If this is correct, what would be a good approach to enforce this requirement?  In my particular scenario, I have a lot of data being written to Kafka at a very high rate.  So a 1TB disk can easily be filled up in 24hrs or so.  One option is to add more Kafka brokers to add more disk space to the pool, but I'd like to avoid that and see if I can simply configure Kafka to not write more than 1TB aggregate.  Else, Kafka will OOM and kill itself, and possibly the crash the node itself because the disk is full.


On May 1, 2014, at 9:21 PM, vinh <vi...@loggly.com> wrote:

> Using Kafka 0.7.2, I have the following in server.properties:
> 
> log.retention.hours=48
> log.retention.size=107374182400
> log.file.size=536870912
> 
> My interpretation of this is:
> a) a single log segment file over 48hrs old will be deleted
> b) the total combined size of *all* logs is 100GB
> c) a single log segment file is limited to 500MB in size before a new segment file is spawned spawning a new segment file
> d) a "log file" can be composed of many "log segment files"
> 
> But, even after setting the above, I find that the total combined size of all Kafka logs on disk is 200GB right now.  Isn't log.retention.size supposed to limit it to 100GB?  Am I missing something?  The docs are not really clear, especially when it comes to distinguishing between a "log file" and a "log segment file".
> 
> I have disk monitoring.  But like anything else in software, even monitoring can fail.  Via configuration, I'd like to make sure that Kafka does not write more than the available disk space.  Or something like log4j, where I can set a max number of log files and the max size per file, which essentially allows me to set a max aggregate size limit across all logs.
> 
> Thanks,
> -Vinh