custom ProducerRecord timestamp and compact+delete cleanup

[Michal Hariš <mi...@gmail.com>]

Hi,

I've read on the message timestamp(KIP-32) that for time-rolling deletes
use LogAppendTime, but this only applies to delete policy.

What timestamp is used when the cleanup policy is _both_ compact+delete?

It would be very useful if it was the timestamp from the record metadata
rather than LogAppendTime or CreateTime as it would of course allow
kafka-backed state stores to simply use the underlying compaction to evict
records based on custom application timestamps and thus have rich ttl
behaviour, that could survive even a full replay of the input stream, etc.

Best,
Michal

Re: custom ProducerRecord timestamp and compact+delete cleanup

["Matthias J. Sax" <ma...@confluent.io>]

Both use the same timestamp that is record meta data timestamp.

KIP-32 behavior was changed later on.


-Matthias

On 11/27/17 8:00 AM, Michal Hariš wrote:
> Hi,
> 
> I've read on the message timestamp(KIP-32) that for time-rolling deletes
> use LogAppendTime, but this only applies to delete policy.
> 
> What timestamp is used when the cleanup policy is _both_ compact+delete?
> 
> It would be very useful if it was the timestamp from the record metadata
> rather than LogAppendTime or CreateTime as it would of course allow
> kafka-backed state stores to simply use the underlying compaction to evict
> records based on custom application timestamps and thus have rich ttl
> behaviour, that could survive even a full replay of the input stream, etc.
> 
> Best,
> Michal
>