Re: compaction not happening for change log topic

[Sachin Mittal <sj...@gmail.com>]

Ok I have gone through the docs and looked the log.cleaner properties.
The use case we have is that say for a change log topic we have at a point
of time
(k, v1).
When for same key we now have
(k, v2) we really don't want (k, v1) to be retained and get cleaned up as
soon as possible.

So I see this:
1. log.cleaner.delete.retention.ms 86400000
I suppose we need to reduce this value to much less say 600000 (10 minutes
or so from default 1 day)

2. log.cleaner.min.compaction.lag.ms
I think default value of 0 is OK given we want old record be clean as soon
as possible?

3. log.cleaner.min.cleanable.ratio
I have not understood this one. What do we mean by dirty log?

What we have observed is that our log segment retention time for change log
topic is 30 minutes.
So after that time old log segment does get deleted, but what we see is all
the segment file sizes are 1GB.
Very rarely we see an old segment size less than 1 GB.
So does this mean that logs are not getting compacted or the messages are
marked for deletion but somehow log cleaner is not cleaning up those
messages.

Please let me know is these are the properties we need to adjust for a more
efficient cleanup of old messages for same key.

Thanks
Sachin




On Sat, Apr 1, 2017 at 4:54 AM, Matthias J. Sax <ma...@confluent.io>
wrote:

> That's a topic config you need to set at the broker side:
>
> See config parameter `log.cleaner.*` in
> http://kafka.apache.org/documentation/#brokerconfigs
>
>
> -Matthias
>
> On 3/31/17 11:49 AM, Sachin Mittal wrote:
> > Hi,
> > I have noticed that many times change log topics don't get compacted. The
> > segment log file is always 1 GB.
> >
> > So I would like to know how and when compaction comes into play.
> > is there a way we can get the topic compacted say trigger compaction
> after
> > x seconds of a given message or a given segment file creation.
> >
> > Also what could be the reason for change log topic not getting compacted.
> >
> > Thanks
> > Sachin
> >
>
>

Re: compaction not happening for change log topic

[Eno Thereska <en...@gmail.com>]

Hi Sachin,

I’ve found this useful: http://stackoverflow.com/questions/34281643/how-to-test-whether-log-compaction-is-working-or-not-in-kafka, <http://stackoverflow.com/questions/34281643/how-to-test-whether-log-compaction-is-working-or-not-in-kafka,as> as well as this http://www.shayne.me/blog/2015/2015-06-25-everything-about-kafka-part-2/ <http://www.shayne.me/blog/2015/2015-06-25-everything-about-kafka-part-2/> (the log compaction part).

Thanks
Eno

> On Apr 1, 2017, at 6:17 AM, Sachin Mittal <sj...@gmail.com> wrote:
> 
> Ok I have gone through the docs and looked the log.cleaner properties.
> The use case we have is that say for a change log topic we have at a point
> of time
> (k, v1).
> When for same key we now have
> (k, v2) we really don't want (k, v1) to be retained and get cleaned up as
> soon as possible.
> 
> So I see this:
> 1. log.cleaner.delete.retention.ms 86400000
> I suppose we need to reduce this value to much less say 600000 (10 minutes
> or so from default 1 day)
> 
> 2. log.cleaner.min.compaction.lag.ms
> I think default value of 0 is OK given we want old record be clean as soon
> as possible?
> 
> 3. log.cleaner.min.cleanable.ratio
> I have not understood this one. What do we mean by dirty log?
> 
> What we have observed is that our log segment retention time for change log
> topic is 30 minutes.
> So after that time old log segment does get deleted, but what we see is all
> the segment file sizes are 1GB.
> Very rarely we see an old segment size less than 1 GB.
> So does this mean that logs are not getting compacted or the messages are
> marked for deletion but somehow log cleaner is not cleaning up those
> messages.
> 
> Please let me know is these are the properties we need to adjust for a more
> efficient cleanup of old messages for same key.
> 
> Thanks
> Sachin
> 
> 
> 
> 
> On Sat, Apr 1, 2017 at 4:54 AM, Matthias J. Sax <ma...@confluent.io>
> wrote:
> 
>> That's a topic config you need to set at the broker side:
>> 
>> See config parameter `log.cleaner.*` in
>> http://kafka.apache.org/documentation/#brokerconfigs
>> 
>> 
>> -Matthias
>> 
>> On 3/31/17 11:49 AM, Sachin Mittal wrote:
>>> Hi,
>>> I have noticed that many times change log topics don't get compacted. The
>>> segment log file is always 1 GB.
>>> 
>>> So I would like to know how and when compaction comes into play.
>>> is there a way we can get the topic compacted say trigger compaction
>> after
>>> x seconds of a given message or a given segment file creation.
>>> 
>>> Also what could be the reason for change log topic not getting compacted.
>>> 
>>> Thanks
>>> Sachin
>>> 
>> 
>>