kafka log compaction

[Pushkar Deole <pd...@gmail.com>]

Hi All

I want some of my topics to retain data forever without any deletion since
those topics hold static data that is always required by application. Also,
for these topic I want to retain latest value for key.
I believe the cleanup policy of 'compact' would meet my needs. I have
following questions though:
1. Would setting the cleanup policy to compact (and No delete) would always
retain the latest value for a key?
2. Does parameters like segment.bytes, retention.ms also play any role in
compaction?

Re: kafka log compaction

[Ricardo Ferreira <ri...@riferrei.com>]

Pushkar,

"1. Would setting the cleanup policy to compact (and No delete) would always
retain the latest value for a key?" -- Yes. This is the purpose of this 
setting.

"2. Does parameters like segment.bytes, retention.ms also play any role in
compaction?" -- They don't play any role in compaction, they play a role 
in retention. Compaction has more to do with the behavior of keeping the 
very last mutation of a record -- whereas retention dictates how long 
the data needs to be retained. They can be used interchangeably.

Thanks

-- Ricardo

On 6/18/20 12:10 AM, Pushkar Deole wrote:
> Hi All
>
> I want some of my topics to retain data forever without any deletion since
> those topics hold static data that is always required by application. Also,
> for these topic I want to retain latest value for key.
> I believe the cleanup policy of 'compact' would meet my needs. I have
> following questions though:
> 1. Would setting the cleanup policy to compact (and No delete) would always
> retain the latest value for a key?
> 2. Does parameters like segment.bytes, retention.ms also play any role in
> compaction?
>