You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by xiongqi wu <xi...@gmail.com> on 2018/08/09 23:18:56 UTC

[DISCUSS] KIP-354 Time-based log compaction policy

Hi Kafka,

This KIP tries to address GDPR concern to fulfill deletion request on time
through time-based log compaction on a compaction enabled topic:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-354%3A+Time-based+log+compaction+policy

Any feedback will be appreciated.


Xiongqi (Wesley) Wu

Re: [DISCUSS] KIP-354 Time-based log compaction policy

Posted by xiongqi wu <xi...@gmail.com>.
Yes.  we want to enforce a max time interval from a message arrival time to
the time the corresponding log segment needs to be compacted.

Today, if the message arriving rate is low for a log compacted topic, the
dirty ratio increases very slowly. As a result, a log segment might be
un-compacted for a long time.

Xiongqi (Welsey) Wu

On Mon, Aug 13, 2018 at 2:46 PM, Guozhang Wang <wa...@gmail.com> wrote:

> Guess I need to carefully read the wiki page before asking :) Thanks!
>
> Another qq after reading the proposal: is it a complimentary to KIP-58 (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 58+-+Make+Log+Compaction+Point+Configurable),
> just that KIP-58 is a "upper-bound" on what messages can be compacted, and
> this is for a "lower-bound" on what messages NEED to be compacted?
>
>
> Guozhang
>
> On Mon, Aug 13, 2018 at 2:31 PM, xiongqi wu <xi...@gmail.com> wrote:
>
> > HI Guozhang,
> >
> > As I mentioned in the motivation section, KIP-280 focuses on how to
> compact
> > the log segment to resolve the out of order messages compaction issue.
> > The issue we try to address in this KIP is different:  we want to
> introduce
> > a compaction policy so that a log segment can be pickup for compaction
> > after a specified time interval.  One use case is for GDPR to ensure
> timely
> > deletion of user record.
> >
> > There is no conflict and overlapping between this KIP and KIP-280.
> >
> > Thank you!
> >
> >
> > On Mon, Aug 13, 2018 at 1:33 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >
> > > Hello Xiongqi,
> > >
> > > I think this KIP is already been covered in KIP-280? Could you check
> out
> > > that one and see if it is the case.
> > >
> > >
> > > Guozhang
> > >
> > >
> > > On Mon, Aug 13, 2018 at 1:23 PM, xiongqi wu <xi...@gmail.com>
> wrote:
> > >
> > > > Hi Kafka,
> > > >
> > > > Just updated the confluence page to include the link to this KIP.
> > > >
> > > > Any comment will be appreciated:
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354%3A
> > > > +Time-based+log+compaction+policy
> > > >
> > > > Thank you.
> > > >
> > > > Xiongqi (Wesley) Wu
> > > >
> > > > On Thu, Aug 9, 2018 at 4:18 PM, xiongqi wu <xi...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Kafka,
> > > > >
> > > > > This KIP tries to address GDPR concern to fulfill deletion request
> on
> > > > time
> > > > > through time-based log compaction on a compaction enabled topic:
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 354%3A+Time-based+log+compaction+policy
> > > > >
> > > > > Any feedback will be appreciated.
> > > > >
> > > > >
> > > > > Xiongqi (Wesley) Wu
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
>
>
> --
> -- Guozhang
>

Re: [DISCUSS] KIP-354 Time-based log compaction policy

Posted by Guozhang Wang <wa...@gmail.com>.
Guess I need to carefully read the wiki page before asking :) Thanks!

Another qq after reading the proposal: is it a complimentary to KIP-58 (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-58+-+Make+Log+Compaction+Point+Configurable),
just that KIP-58 is a "upper-bound" on what messages can be compacted, and
this is for a "lower-bound" on what messages NEED to be compacted?


Guozhang

On Mon, Aug 13, 2018 at 2:31 PM, xiongqi wu <xi...@gmail.com> wrote:

> HI Guozhang,
>
> As I mentioned in the motivation section, KIP-280 focuses on how to compact
> the log segment to resolve the out of order messages compaction issue.
> The issue we try to address in this KIP is different:  we want to introduce
> a compaction policy so that a log segment can be pickup for compaction
> after a specified time interval.  One use case is for GDPR to ensure timely
> deletion of user record.
>
> There is no conflict and overlapping between this KIP and KIP-280.
>
> Thank you!
>
>
> On Mon, Aug 13, 2018 at 1:33 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > Hello Xiongqi,
> >
> > I think this KIP is already been covered in KIP-280? Could you check out
> > that one and see if it is the case.
> >
> >
> > Guozhang
> >
> >
> > On Mon, Aug 13, 2018 at 1:23 PM, xiongqi wu <xi...@gmail.com> wrote:
> >
> > > Hi Kafka,
> > >
> > > Just updated the confluence page to include the link to this KIP.
> > >
> > > Any comment will be appreciated:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354%3A
> > > +Time-based+log+compaction+policy
> > >
> > > Thank you.
> > >
> > > Xiongqi (Wesley) Wu
> > >
> > > On Thu, Aug 9, 2018 at 4:18 PM, xiongqi wu <xi...@gmail.com>
> wrote:
> > >
> > > > Hi Kafka,
> > > >
> > > > This KIP tries to address GDPR concern to fulfill deletion request on
> > > time
> > > > through time-based log compaction on a compaction enabled topic:
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 354%3A+Time-based+log+compaction+policy
> > > >
> > > > Any feedback will be appreciated.
> > > >
> > > >
> > > > Xiongqi (Wesley) Wu
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>



-- 
-- Guozhang

Re: [DISCUSS] KIP-354 Time-based log compaction policy

Posted by xiongqi wu <xi...@gmail.com>.
HI Guozhang,

As I mentioned in the motivation section, KIP-280 focuses on how to compact
the log segment to resolve the out of order messages compaction issue.
The issue we try to address in this KIP is different:  we want to introduce
a compaction policy so that a log segment can be pickup for compaction
after a specified time interval.  One use case is for GDPR to ensure timely
deletion of user record.

There is no conflict and overlapping between this KIP and KIP-280.

Thank you!


On Mon, Aug 13, 2018 at 1:33 PM, Guozhang Wang <wa...@gmail.com> wrote:

> Hello Xiongqi,
>
> I think this KIP is already been covered in KIP-280? Could you check out
> that one and see if it is the case.
>
>
> Guozhang
>
>
> On Mon, Aug 13, 2018 at 1:23 PM, xiongqi wu <xi...@gmail.com> wrote:
>
> > Hi Kafka,
> >
> > Just updated the confluence page to include the link to this KIP.
> >
> > Any comment will be appreciated:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354%3A
> > +Time-based+log+compaction+policy
> >
> > Thank you.
> >
> > Xiongqi (Wesley) Wu
> >
> > On Thu, Aug 9, 2018 at 4:18 PM, xiongqi wu <xi...@gmail.com> wrote:
> >
> > > Hi Kafka,
> > >
> > > This KIP tries to address GDPR concern to fulfill deletion request on
> > time
> > > through time-based log compaction on a compaction enabled topic:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 354%3A+Time-based+log+compaction+policy
> > >
> > > Any feedback will be appreciated.
> > >
> > >
> > > Xiongqi (Wesley) Wu
> > >
> >
>
>
>
> --
> -- Guozhang
>

Re: [DISCUSS] KIP-354 Time-based log compaction policy

Posted by Guozhang Wang <wa...@gmail.com>.
Hello Xiongqi,

I think this KIP is already been covered in KIP-280? Could you check out
that one and see if it is the case.


Guozhang


On Mon, Aug 13, 2018 at 1:23 PM, xiongqi wu <xi...@gmail.com> wrote:

> Hi Kafka,
>
> Just updated the confluence page to include the link to this KIP.
>
> Any comment will be appreciated:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354%3A
> +Time-based+log+compaction+policy
>
> Thank you.
>
> Xiongqi (Wesley) Wu
>
> On Thu, Aug 9, 2018 at 4:18 PM, xiongqi wu <xi...@gmail.com> wrote:
>
> > Hi Kafka,
> >
> > This KIP tries to address GDPR concern to fulfill deletion request on
> time
> > through time-based log compaction on a compaction enabled topic:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 354%3A+Time-based+log+compaction+policy
> >
> > Any feedback will be appreciated.
> >
> >
> > Xiongqi (Wesley) Wu
> >
>



-- 
-- Guozhang

Re: [DISCUSS] KIP-354 Time-based log compaction policy

Posted by xiongqi wu <xi...@gmail.com>.
Hi Kafka,

Just updated the confluence page to include the link to this KIP.

Any comment will be appreciated:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-354%3A+Time-based+log+compaction+policy

Thank you.

Xiongqi (Wesley) Wu

On Thu, Aug 9, 2018 at 4:18 PM, xiongqi wu <xi...@gmail.com> wrote:

> Hi Kafka,
>
> This KIP tries to address GDPR concern to fulfill deletion request on time
> through time-based log compaction on a compaction enabled topic:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 354%3A+Time-based+log+compaction+policy
>
> Any feedback will be appreciated.
>
>
> Xiongqi (Wesley) Wu
>

Re: [DISCUSS] KIP-354 Time-based log compaction policy

Posted by Brett Rann <br...@zendesk.com.INVALID>.
Eno,

For us as well the requirement is around compacted topics because they are
the topics that already facilitate selective deletes. Currently they allow
specifying a minimum life time, but lacks the ability to specify a maximum
life time.

For non compacted topics there's no ability to delete individual messages,
they're immutable logs. We treat those with hard rules: Max retention time
on the topic; accept the topic may get truncated; or to not store
information that may be subject to GDPR. (and i've read others use tricks
with encryption and forgetting the decryption key).

Enhancing compaction to support a max compaction time makes the compacted
topics more useful, especially in that it allows the dirty ratio to be used
for its intended purpose while allowing automatic cleaning based on a new
time config.

On Tue, Aug 14, 2018 at 9:00 PM Eno Thereska <en...@gmail.com> wrote:

> Adding to this, what about topics that are not log compacted? As Dong says,
> "one of the GDPR requirement is that we can not keep messages longer than
> e.g. 30 days in storage (e.g. Kafka)". The GDPR requirement must hold
> irrespective of the low level details, on whether the topic is compacted or
> not, right?
>
> Thanks
> Eno
>
>
> On Mon, Aug 13, 2018 at 6:58 PM, Dong Lin <li...@gmail.com> wrote:
>
> > Hey Xiongqi,
> >
> > Thanks for the KIP. I have two questions regarding the use-case for
> meeting
> > GDPR requirement.
> >
> > 1) If I recall correctly, one of the GDPR requirement is that we can not
> > keep messages longer than e.g. 30 days in storage (e.g. Kafka). Say there
> > exists a partition p0 which contains message1 with key1 and message2 with
> > key2. And then user keeps producing messages with key=key2 to this
> > partition. Since message1 with key1 is never overridden, sooner or later
> we
> > will want to delete message1 and keep the latest message with key=key2.
> But
> > currently it looks like log compact logic in Kafka will always put these
> > messages in the same segment. Will this be an issue?
> >
> > 2) The current KIP intends to provide the capability to delete a given
> > message in log compacted topic. Does such use-case also require Kafka to
> > keep the messages produced before the given message? If yes, then we can
> > probably just use AdminClient.deleteRecords() or time-based log retention
> > to meet the use-case requirement. If no, do you know what is the GDPR's
> > requirement on time-to-deletion after user explicitly requests the
> deletion
> > (e.g. 1 hour, 1 day, 7 day)?
> >
> > Thanks,
> > Dong
> >
> >
> > On Mon, Aug 13, 2018 at 3:44 PM, xiongqi wu <xi...@gmail.com> wrote:
> >
> > > Hi Eno,
> > >
> > > The GDPR request we are getting here at linkedin is if we get a request
> > to
> > > delete a record through a null key on a log compacted topic,
> > > we want to delete the record via compaction in a given time period
> like 2
> > > days (whatever is required by the policy).
> > >
> > > There might be other issues (such as orphan log segments under certain
> > > conditions) that lead to GDPR problem but they are more like something
> > we
> > > need to fix anyway regardless of GDPR.
> > >
> > >
> > > -- Xiongqi (Wesley) Wu
> > >
> > > On Mon, Aug 13, 2018 at 2:56 PM, Eno Thereska <en...@gmail.com>
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > Thanks for the KIP. I'd like to see a more precise definition of what
> > > part
> > > > of GDPR you are targeting as well as some sort of verification that
> > this
> > > > KIP actually addresses the problem. Right now I find this a bit
> vague:
> > > >
> > > > "Ability to delete a log message through compaction in a timely
> manner
> > > has
> > > > become an important requirement in some use cases (e.g., GDPR)"
> > > >
> > > >
> > > > Is there any guarantee that after this KIP the GDPR problem is solved
> > or
> > > do
> > > > we need to do something else as well, e.g., more KIPs?
> > > >
> > > >
> > > > Thanks
> > > >
> > > > Eno
> > > >
> > > >
> > > >
> > > > On Thu, Aug 9, 2018 at 4:18 PM, xiongqi wu <xi...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Kafka,
> > > > >
> > > > > This KIP tries to address GDPR concern to fulfill deletion request
> on
> > > > time
> > > > > through time-based log compaction on a compaction enabled topic:
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP->
> > > > > 354%3A+Time-based+log+compaction+policy
> > > > >
> > > > > Any feedback will be appreciated.
> > > > >
> > > > >
> > > > > Xiongqi (Wesley) Wu
> > > > >
> > > >
> > >
> >
>


-- 

Brett Rann

Senior DevOps Engineer


Zendesk International Ltd

395 Collins Street, Melbourne VIC 3000 Australia

Mobile: +61 (0) 418 826 017

Re: [DISCUSS] KIP-354 Time-based log compaction policy

Posted by Eno Thereska <en...@gmail.com>.
Adding to this, what about topics that are not log compacted? As Dong says,
"one of the GDPR requirement is that we can not keep messages longer than
e.g. 30 days in storage (e.g. Kafka)". The GDPR requirement must hold
irrespective of the low level details, on whether the topic is compacted or
not, right?

Thanks
Eno


On Mon, Aug 13, 2018 at 6:58 PM, Dong Lin <li...@gmail.com> wrote:

> Hey Xiongqi,
>
> Thanks for the KIP. I have two questions regarding the use-case for meeting
> GDPR requirement.
>
> 1) If I recall correctly, one of the GDPR requirement is that we can not
> keep messages longer than e.g. 30 days in storage (e.g. Kafka). Say there
> exists a partition p0 which contains message1 with key1 and message2 with
> key2. And then user keeps producing messages with key=key2 to this
> partition. Since message1 with key1 is never overridden, sooner or later we
> will want to delete message1 and keep the latest message with key=key2. But
> currently it looks like log compact logic in Kafka will always put these
> messages in the same segment. Will this be an issue?
>
> 2) The current KIP intends to provide the capability to delete a given
> message in log compacted topic. Does such use-case also require Kafka to
> keep the messages produced before the given message? If yes, then we can
> probably just use AdminClient.deleteRecords() or time-based log retention
> to meet the use-case requirement. If no, do you know what is the GDPR's
> requirement on time-to-deletion after user explicitly requests the deletion
> (e.g. 1 hour, 1 day, 7 day)?
>
> Thanks,
> Dong
>
>
> On Mon, Aug 13, 2018 at 3:44 PM, xiongqi wu <xi...@gmail.com> wrote:
>
> > Hi Eno,
> >
> > The GDPR request we are getting here at linkedin is if we get a request
> to
> > delete a record through a null key on a log compacted topic,
> > we want to delete the record via compaction in a given time period like 2
> > days (whatever is required by the policy).
> >
> > There might be other issues (such as orphan log segments under certain
> > conditions)  that lead to GDPR problem but they are more like something
> we
> > need to fix anyway regardless of GDPR.
> >
> >
> > -- Xiongqi (Wesley) Wu
> >
> > On Mon, Aug 13, 2018 at 2:56 PM, Eno Thereska <en...@gmail.com>
> > wrote:
> >
> > > Hello,
> > >
> > > Thanks for the KIP. I'd like to see a more precise definition of what
> > part
> > > of GDPR you are targeting as well as some sort of verification that
> this
> > > KIP actually addresses the problem. Right now I find this a bit vague:
> > >
> > > "Ability to delete a log message through compaction in a timely manner
> > has
> > > become an important requirement in some use cases (e.g., GDPR)"
> > >
> > >
> > > Is there any guarantee that after this KIP the GDPR problem is solved
> or
> > do
> > > we need to do something else as well, e.g., more KIPs?
> > >
> > >
> > > Thanks
> > >
> > > Eno
> > >
> > >
> > >
> > > On Thu, Aug 9, 2018 at 4:18 PM, xiongqi wu <xi...@gmail.com>
> wrote:
> > >
> > > > Hi Kafka,
> > > >
> > > > This KIP tries to address GDPR concern to fulfill deletion request on
> > > time
> > > > through time-based log compaction on a compaction enabled topic:
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 354%3A+Time-based+log+compaction+policy
> > > >
> > > > Any feedback will be appreciated.
> > > >
> > > >
> > > > Xiongqi (Wesley) Wu
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-354 Time-based log compaction policy

Posted by xiongqi wu <xi...@gmail.com>.
Hi Brett,

In my opinion, we don't need very accurate estimation of first record
timestamp because the compaction processing itself might take
some time. There is no strictly guaranteed that after the time interval
"T", the compaction is finished.  Rather than,
after time "T" + some compaction job interval,  the compaction will start
processing.
For this reason,  using the largest timestamp of previous segment to
estimate creation time of next segments looks sufficient to me.

One major principal is to not to introduce a lot IOs to those partitions
that doesn't require to be compacted.
I want to avoid looking into records or timestamp as much as possible.

Partition movement can alter the segment modification time.  However I see
this as the data is in a transit state.
After another time interval T when data is stabilized in new location, the
compaction will be launched.
We also use largestTimestamp in the time based retention.

On Wed, Aug 15, 2018 at 9:08 PM, Brett Rann <br...@zendesk.com.invalid>
wrote:

> > (segment.largestTimestamp is lastModified time of the log segment or max
> timestamp we see for the log segment. Due to the lack of record timestamp,
> segment.largestTimestamp might be earlier than the actual timestamp of
> latest record of that segment.).
>
> I'm curious about the mention of last modified time of the segment.  As
> noted back in here
> https://cwiki-test.apache.org/confluence/display/KAFKA/KIP-
> 33+-+Add+a+time+based+log+index
> using the creation/modified time of files is unreliable in cases of
> partitions being moved. It's why all those .timeindex files for each
> partition appeared in 0.10.* I wonder if there's a better way using that to
> get at a timestamp?
>
> On Thu, Aug 16, 2018 at 11:30 AM Brett Rann <br...@zendesk.com> wrote:
>
> > An API was suggested by Gwen and James when I discussed it with them. For
> > me I can think of it as a use case for scheduling compaction rather than
> > relying on an config based time trigger.  We're looking at creating some
> > potentially very large compacted topics for event sourcing and from an
> > operators perspective I'm concerned about potential impact of long
> running
> > compaction, especially if multiple topics run back to back.  Having the
> > ability to schedule them at my own time gives some peace of mind for that
> > concern.  Another use case might be a more urgent delete. That could be
> > done manually now by just modifying the max time and waiting for
> compaction
> > to run. But hitting an API end point is a bit cleaner.
> >
> > But in thinking about what that mechanism would be it started to feel
> like
> > it would be a complicated implementation so we've put it aside for now.
> But
> > maybe we just haven't seen the clean way yet.
> >
> >
> >
> > On Thu, Aug 16, 2018 at 11:22 AM xiongqi wu <xi...@gmail.com> wrote:
> >
> >> Brett,
> >>
> >> Thank you for your comments.
> >> I was thinking since we already has immediate compaction setting by
> >> setting
> >> min dirty ratio to 0, so I decide to use "0" as disabled state.
> >> I am ok to go with -1(disable), 0 (immediate) options.
> >>
> >> For the implementation, there are a few differences between mine and
> >> "Xiaohe Dong"'s :
> >> 1) I used the estimated creation time of a log segment instead of
> largest
> >> timestamp of a log to determine the compaction eligibility, because a
> log
> >> segment might stay as an active segment up to "max compaction lag". (see
> >> the KIP for detail).
> >> 2) I measure how much bytes that we must clean to follow the "max
> >> compaction lag" rule, and use that to determine the order of compaction.
> >> 3) force active segment to roll to follow the "max compaction lag"
> >>
> >> I can share my code so we can coordinate.
> >>
> >> I haven't think about a new API to force a compaction. what is the use
> >> case
> >> for this one?
> >>
> >>
> >> On Wed, Aug 15, 2018 at 5:33 PM, Brett Rann <br...@zendesk.com.invalid>
> >> wrote:
> >>
> >> > We've been looking into this too.
> >> >
> >> > Mailing list:
> >> > https://lists.apache.org/thread.html/ed7f6a6589f94e8c2a705553f364ef
> >> <https://lists.apache.org/thread.html/ed7f6a6589f94e8c2a705553f364ef>
> >> > 599cb6915e4c3ba9b561e610e4@%3Cdev.kafka.apache.org%3E
> >> > jira wish: https://issues.apache.org/jira/browse/KAFKA-7137
> >> <https://issues.apache.org/jira/browse/KAFKA-7137>
> >> > confluent slack discussion:
> >> >
> >> https://confluentcommunity.slack.com/archives/C49R61XMM/
> p1530760121000039
> >> <https://confluentcommunity.slack.com/archives/C49R61XMM/
> p1530760121000039>
> >> >
> >> > A person on my team has started on code so you might want to
> coordinate:
> >> > https://github.com/dongxiaohe/kafka/tree/dongxiaohe/log-
> >> <https://github.com/dongxiaohe/kafka/tree/dongxiaohe/log->
> >> > cleaner-compaction-max-lifetime-2.0
> >> >
> >> > He's been working with Jason Gustafson and James Chen around the
> >> changes.
> >> > You can ping him on confluent slack as Xiaohe Dong.
> >> >
> >> > It's great to know others are thinking on it as well.
> >> >
> >> > You've added the requirement to force a segment roll which we hadn't
> >> gotten
> >> > to yet, which is great. I was content with it not including the active
> >> > segment.
> >> >
> >> > > Adding topic level configuration "max.compaction.lag.ms", and
> >> > corresponding broker configuration "log.cleaner.max.compaction.lag.ms
> ",
> >> > which is set to 0 (disabled) by default.
> >> >
> >> > Glancing at some other settings convention seems to me to be -1 for
> >> > disabled (or infinite, which is more meaningful here). 0 to me implies
> >> > instant, a little quicker than 1.
> >> >
> >> > We've been trying to think about a way to trigger compaction as well
> >> > through an API call, which would need to be flagged somewhere (ZK
> admin/
> >> > space?) but we're struggling to think how that would be coordinated
> >> across
> >> > brokers and partitions. Have you given any thought to that?
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Thu, Aug 16, 2018 at 8:44 AM xiongqi wu <xi...@gmail.com>
> wrote:
> >> >
> >> > > Eno, Dong,
> >> > >
> >> > > I have updated the KIP. We decide not to address the issue that we
> >> might
> >> > > have for both compaction and time retention enabled topics (see the
> >> > > rejected alternative item 2). This KIP will only ensure log can be
> >> > > compacted after a specified time-interval.
> >> > >
> >> > > As suggested by Dong, we will also enforce "max.compaction.lag.ms"
> is
> >> > not
> >> > > less than "min.compaction.lag.ms".
> >> > >
> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354> Time-based
> >> > log
> >> > > compaction policy
> >> > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354> Time-based
> >> > log compaction policy>
> >> > >
> >> > >
> >> > > On Tue, Aug 14, 2018 at 5:01 PM, xiongqi wu <xi...@gmail.com>
> >> wrote:
> >> > >
> >> > > >
> >> > > > Per discussion with Dong, he made a very good point that if
> >> compaction
> >> > > > and time based retention are both enabled on a topic, the
> compaction
> >> > > might
> >> > > > prevent records from being deleted on time. The reason is when
> >> > compacting
> >> > > > multiple segments into one single segment, the newly created
> segment
> >> > will
> >> > > > have same lastmodified timestamp as latest original segment. We
> lose
> >> > the
> >> > > > timestamp of all original segments except the last one. As a
> result,
> >> > > > records might not be deleted as it should be through time based
> >> > > retention.
> >> > > >
> >> > > > With the current KIP proposal, if we want to ensure timely
> >> deletion, we
> >> > > > have the following configurations:
> >> > > > 1) enable time based log compaction only : deletion is done though
> >> > > > overriding the same key
> >> > > > 2) enable time based log retention only: deletion is done though
> >> > > > time-based retention
> >> > > > 3) enable both log compaction and time based retention: Deletion
> is
> >> not
> >> > > > guaranteed.
> >> > > >
> >> > > > Not sure if we have use case 3 and also want deletion to happen on
> >> > time.
> >> > > > There are several options to address deletion issue when enable
> both
> >> > > > compaction and retention:
> >> > > > A) During log compaction, looking into record timestamp to delete
> >> > expired
> >> > > > records. This can be done in compaction logic itself or use
> >> > > > AdminClient.deleteRecords() . But this assumes we have record
> >> > timestamp.
> >> > > > B) retain the lastModifed time of original segments during log
> >> > > compaction.
> >> > > > This requires extra meta data to record the information or not
> >> grouping
> >> > > > multiple segments into one during compaction.
> >> > > >
> >> > > > If we have use case 3 in general, I would prefer solution A and
> >> rely on
> >> > > > record timestamp.
> >> > > >
> >> > > >
> >> > > > Two questions:
> >> > > > Do we have use case 3? Is it nice to have or must have?
> >> > > > If we have use case 3 and want to go with solution A, should we
> >> > introduce
> >> > > > a new configuration to enforce deletion by timestamp?
> >> > > >
> >> > > >
> >> > > > On Tue, Aug 14, 2018 at 1:52 PM, xiongqi wu <xi...@gmail.com>
> >> > wrote:
> >> > > >
> >> > > >> Dong,
> >> > > >>
> >> > > >> Thanks for the comment.
> >> > > >>
> >> > > >> There are two retention policy: log compaction and time based
> >> > retention.
> >> > > >>
> >> > > >> Log compaction:
> >> > > >>
> >> > > >> we have use cases to keep infinite retention of a topic (only
> >> > > >> compaction). GDPR cares about deletion of PII (personal
> >> identifiable
> >> > > >> information) data.
> >> > > >> Since Kafka doesn't know what records contain PII, it relies on
> >> upper
> >> > > >> layer to delete those records.
> >> > > >> For those infinite retention uses uses, kafka needs to provide a
> >> way
> >> > to
> >> > > >> enforce compaction on time. This is what we try to address in
> this
> >> > KIP.
> >> > > >>
> >> > > >> Time based retention,
> >> > > >>
> >> > > >> There are also use cases that users of Kafka might want to expire
> >> all
> >> > > >> their data.
> >> > > >> In those cases, they can use time based retention of their
> topics.
> >> > > >>
> >> > > >>
> >> > > >> Regarding your first question, if a user wants to delete a key in
> >> the
> >> > > >> log compaction topic, the user has to send a deletion using the
> >> same
> >> > > key.
> >> > > >> Kafka only makes sure the deletion will happen under a certain
> time
> >> > > >> periods (like 2 days/7 days).
> >> > > >>
> >> > > >> Regarding your second question. In most cases, we might want to
> >> delete
> >> > > >> all duplicated keys at the same time.
> >> > > >> Compaction might be more efficient since we need to scan the log
> >> and
> >> > > find
> >> > > >> all duplicates. However, the expected use case is to set the time
> >> > based
> >> > > >> compaction interval on the order of days, and be larger than 'min
> >> > > >> compaction lag". We don't want log compaction to happen
> frequently
> >> > since
> >> > > >> it is expensive. The purpose is to help low production rate topic
> >> to
> >> > get
> >> > > >> compacted on time. For the topic with "normal" incoming message
> >> > message
> >> > > >> rate, the "min dirty ratio" might have triggered the compaction
> >> before
> >> > > this
> >> > > >> time based compaction policy takes effect.
> >> > > >>
> >> > > >>
> >> > > >> Eno,
> >> > > >>
> >> > > >> For your question, like I mentioned we have long time retention
> use
> >> > case
> >> > > >> for log compacted topic, but we want to provide ability to delete
> >> > > certain
> >> > > >> PII records on time.
> >> > > >> Kafka itself doesn't know whether a record contains sensitive
> >> > > information
> >> > > >> and relies on the user for deletion.
> >> > > >>
> >> > > >>
> >> > > >> On Mon, Aug 13, 2018 at 6:58 PM, Dong Lin <li...@gmail.com>
> >> > wrote:
> >> > > >>
> >> > > >>> Hey Xiongqi,
> >> > > >>>
> >> > > >>> Thanks for the KIP. I have two questions regarding the use-case
> >> for
> >> > > >>> meeting
> >> > > >>> GDPR requirement.
> >> > > >>>
> >> > > >>> 1) If I recall correctly, one of the GDPR requirement is that we
> >> can
> >> > > not
> >> > > >>> keep messages longer than e.g. 30 days in storage (e.g. Kafka).
> >> Say
> >> > > there
> >> > > >>> exists a partition p0 which contains message1 with key1 and
> >> message2
> >> > > with
> >> > > >>> key2. And then user keeps producing messages with key=key2 to
> this
> >> > > >>> partition. Since message1 with key1 is never overridden, sooner
> or
> >> > > later
> >> > > >>> we
> >> > > >>> will want to delete message1 and keep the latest message with
> >> > key=key2.
> >> > > >>> But
> >> > > >>> currently it looks like log compact logic in Kafka will always
> put
> >> > > these
> >> > > >>> messages in the same segment. Will this be an issue?
> >> > > >>>
> >> > > >>> 2) The current KIP intends to provide the capability to delete a
> >> > given
> >> > > >>> message in log compacted topic. Does such use-case also require
> >> Kafka
> >> > > to
> >> > > >>> keep the messages produced before the given message? If yes,
> then
> >> we
> >> > > can
> >> > > >>> probably just use AdminClient.deleteRecords() or time-based log
> >> > > retention
> >> > > >>> to meet the use-case requirement. If no, do you know what is the
> >> > GDPR's
> >> > > >>> requirement on time-to-deletion after user explicitly requests
> the
> >> > > >>> deletion
> >> > > >>> (e.g. 1 hour, 1 day, 7 day)?
> >> > > >>>
> >> > > >>> Thanks,
> >> > > >>> Dong
> >> > > >>>
> >> > > >>>
> >> > > >>> On Mon, Aug 13, 2018 at 3:44 PM, xiongqi wu <
> xiongqiwu@gmail.com>
> >> > > wrote:
> >> > > >>>
> >> > > >>> > Hi Eno,
> >> > > >>> >
> >> > > >>> > The GDPR request we are getting here at linkedin is if we get
> a
> >> > > >>> request to
> >> > > >>> > delete a record through a null key on a log compacted topic,
> >> > > >>> > we want to delete the record via compaction in a given time
> >> period
> >> > > >>> like 2
> >> > > >>> > days (whatever is required by the policy).
> >> > > >>> >
> >> > > >>> > There might be other issues (such as orphan log segments under
> >> > > certain
> >> > > >>> > conditions) that lead to GDPR problem but they are more like
> >> > > >>> something we
> >> > > >>> > need to fix anyway regardless of GDPR.
> >> > > >>> >
> >> > > >>> >
> >> > > >>> > -- Xiongqi (Wesley) Wu
> >> > > >>> >
> >> > > >>> > On Mon, Aug 13, 2018 at 2:56 PM, Eno Thereska <
> >> > > eno.thereska@gmail.com>
> >> > > >>> > wrote:
> >> > > >>> >
> >> > > >>> > > Hello,
> >> > > >>> > >
> >> > > >>> > > Thanks for the KIP. I'd like to see a more precise
> definition
> >> of
> >> > > what
> >> > > >>> > part
> >> > > >>> > > of GDPR you are targeting as well as some sort of
> verification
> >> > that
> >> > > >>> this
> >> > > >>> > > KIP actually addresses the problem. Right now I find this a
> >> bit
> >> > > >>> vague:
> >> > > >>> > >
> >> > > >>> > > "Ability to delete a log message through compaction in a
> >> timely
> >> > > >>> manner
> >> > > >>> > has
> >> > > >>> > > become an important requirement in some use cases (e.g.,
> >> GDPR)"
> >> > > >>> > >
> >> > > >>> > >
> >> > > >>> > > Is there any guarantee that after this KIP the GDPR problem
> is
> >> > > >>> solved or
> >> > > >>> > do
> >> > > >>> > > we need to do something else as well, e.g., more KIPs?
> >> > > >>> > >
> >> > > >>> > >
> >> > > >>> > > Thanks
> >> > > >>> > >
> >> > > >>> > > Eno
> >> > > >>> > >
> >> > > >>> > >
> >> > > >>> > >
> >> > > >>> > > On Thu, Aug 9, 2018 at 4:18 PM, xiongqi wu <
> >> xiongqiwu@gmail.com>
> >> > > >>> wrote:
> >> > > >>> > >
> >> > > >>> > > > Hi Kafka,
> >> > > >>> > > >
> >> > > >>> > > > This KIP tries to address GDPR concern to fulfill deletion
> >> > > request
> >> > > >>> on
> >> > > >>> > > time
> >> > > >>> > > > through time-based log compaction on a compaction enabled
> >> > topic:
> >> > > >>> > > >
> >> > > >>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> <https://cwiki.apache.org/confluence/display/KAFKA/KIP->
> >> > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> <https://cwiki.apache.org/confluence/display/KAFKA/KIP->>
> >> > > >>> > > > 354%3A+Time-based+log+compaction+policy
> >> > > >>> > > >
> >> > > >>> > > > Any feedback will be appreciated.
> >> > > >>> > > >
> >> > > >>> > > >
> >> > > >>> > > > Xiongqi (Wesley) Wu
> >> > > >>> > > >
> >> > > >>> > >
> >> > > >>> >
> >> > > >>>
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >> --
> >> > > >> Xiongqi (Wesley) Wu
> >> > > >>
> >> > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > Xiongqi (Wesley) Wu
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > Xiongqi (Wesley) Wu
> >> > >
> >> >
> >> >
> >> > --
> >> >
> >> > Brett Rann
> >> >
> >> > Senior DevOps Engineer
> >> >
> >> >
> >> > Zendesk International Ltd
> >> >
> >> > 395 Collins Street, Melbourne VIC 3000 Australia
> >> >
> >> > Mobile: +61 (0) 418 826 017
> >> >
> >>
> >>
> >>
> >> --
> >> Xiongqi (Wesley) Wu
> >>
> >
> >
> > --
> >
> > Brett Rann
> >
> > Senior DevOps Engineer
> >
> >
> > Zendesk International Ltd
> >
> > 395 Collins Street, Melbourne VIC 3000 Australia
> >
> > Mobile: +61 (0) 418 826 017
> >
>
>
> --
>
> Brett Rann
>
> Senior DevOps Engineer
>
>
> Zendesk International Ltd
>
> 395 Collins Street, Melbourne VIC 3000 Australia
>
> Mobile: +61 (0) 418 826 017
>



-- 
Xiongqi (Wesley) Wu

Re: [DISCUSS] KIP-354 Time-based log compaction policy

Posted by Brett Rann <br...@zendesk.com.INVALID>.
> (segment.largestTimestamp is lastModified time of the log segment or max
timestamp we see for the log segment. Due to the lack of record timestamp,
segment.largestTimestamp might be earlier than the actual timestamp of
latest record of that segment.).

I'm curious about the mention of last modified time of the segment.  As
noted back in here
https://cwiki-test.apache.org/confluence/display/KAFKA/KIP-33+-+Add+a+time+based+log+index
using the creation/modified time of files is unreliable in cases of
partitions being moved. It's why all those .timeindex files for each
partition appeared in 0.10.* I wonder if there's a better way using that to
get at a timestamp?

On Thu, Aug 16, 2018 at 11:30 AM Brett Rann <br...@zendesk.com> wrote:

> An API was suggested by Gwen and James when I discussed it with them. For
> me I can think of it as a use case for scheduling compaction rather than
> relying on an config based time trigger.  We're looking at creating some
> potentially very large compacted topics for event sourcing and from an
> operators perspective I'm concerned about potential impact of long running
> compaction, especially if multiple topics run back to back.  Having the
> ability to schedule them at my own time gives some peace of mind for that
> concern.  Another use case might be a more urgent delete. That could be
> done manually now by just modifying the max time and waiting for compaction
> to run. But hitting an API end point is a bit cleaner.
>
> But in thinking about what that mechanism would be it started to feel like
> it would be a complicated implementation so we've put it aside for now. But
> maybe we just haven't seen the clean way yet.
>
>
>
> On Thu, Aug 16, 2018 at 11:22 AM xiongqi wu <xi...@gmail.com> wrote:
>
>> Brett,
>>
>> Thank you for your comments.
>> I was thinking since we already has immediate compaction setting by
>> setting
>> min dirty ratio to 0, so I decide to use "0" as disabled state.
>> I am ok to go with -1(disable), 0 (immediate) options.
>>
>> For the implementation, there are a few differences between mine and
>> "Xiaohe Dong"'s :
>> 1) I used the estimated creation time of a log segment instead of largest
>> timestamp of a log to determine the compaction eligibility, because a log
>> segment might stay as an active segment up to "max compaction lag". (see
>> the KIP for detail).
>> 2) I measure how much bytes that we must clean to follow the "max
>> compaction lag" rule, and use that to determine the order of compaction.
>> 3) force active segment to roll to follow the "max compaction lag"
>>
>> I can share my code so we can coordinate.
>>
>> I haven't think about a new API to force a compaction. what is the use
>> case
>> for this one?
>>
>>
>> On Wed, Aug 15, 2018 at 5:33 PM, Brett Rann <br...@zendesk.com.invalid>
>> wrote:
>>
>> > We've been looking into this too.
>> >
>> > Mailing list:
>> > https://lists.apache.org/thread.html/ed7f6a6589f94e8c2a705553f364ef
>> <https://lists.apache.org/thread.html/ed7f6a6589f94e8c2a705553f364ef>
>> > 599cb6915e4c3ba9b561e610e4@%3Cdev.kafka.apache.org%3E
>> > jira wish: https://issues.apache.org/jira/browse/KAFKA-7137
>> <https://issues.apache.org/jira/browse/KAFKA-7137>
>> > confluent slack discussion:
>> >
>> https://confluentcommunity.slack.com/archives/C49R61XMM/p1530760121000039
>> <https://confluentcommunity.slack.com/archives/C49R61XMM/p1530760121000039>
>> >
>> > A person on my team has started on code so you might want to coordinate:
>> > https://github.com/dongxiaohe/kafka/tree/dongxiaohe/log-
>> <https://github.com/dongxiaohe/kafka/tree/dongxiaohe/log->
>> > cleaner-compaction-max-lifetime-2.0
>> >
>> > He's been working with Jason Gustafson and James Chen around the
>> changes.
>> > You can ping him on confluent slack as Xiaohe Dong.
>> >
>> > It's great to know others are thinking on it as well.
>> >
>> > You've added the requirement to force a segment roll which we hadn't
>> gotten
>> > to yet, which is great. I was content with it not including the active
>> > segment.
>> >
>> > > Adding topic level configuration "max.compaction.lag.ms", and
>> > corresponding broker configuration "log.cleaner.max.compaction.lag.ms",
>> > which is set to 0 (disabled) by default.
>> >
>> > Glancing at some other settings convention seems to me to be -1 for
>> > disabled (or infinite, which is more meaningful here). 0 to me implies
>> > instant, a little quicker than 1.
>> >
>> > We've been trying to think about a way to trigger compaction as well
>> > through an API call, which would need to be flagged somewhere (ZK admin/
>> > space?) but we're struggling to think how that would be coordinated
>> across
>> > brokers and partitions. Have you given any thought to that?
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Aug 16, 2018 at 8:44 AM xiongqi wu <xi...@gmail.com> wrote:
>> >
>> > > Eno, Dong,
>> > >
>> > > I have updated the KIP. We decide not to address the issue that we
>> might
>> > > have for both compaction and time retention enabled topics (see the
>> > > rejected alternative item 2). This KIP will only ensure log can be
>> > > compacted after a specified time-interval.
>> > >
>> > > As suggested by Dong, we will also enforce "max.compaction.lag.ms" is
>> > not
>> > > less than "min.compaction.lag.ms".
>> > >
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354> Time-based
>> > log
>> > > compaction policy
>> > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354> Time-based
>> > log compaction policy>
>> > >
>> > >
>> > > On Tue, Aug 14, 2018 at 5:01 PM, xiongqi wu <xi...@gmail.com>
>> wrote:
>> > >
>> > > >
>> > > > Per discussion with Dong, he made a very good point that if
>> compaction
>> > > > and time based retention are both enabled on a topic, the compaction
>> > > might
>> > > > prevent records from being deleted on time. The reason is when
>> > compacting
>> > > > multiple segments into one single segment, the newly created segment
>> > will
>> > > > have same lastmodified timestamp as latest original segment. We lose
>> > the
>> > > > timestamp of all original segments except the last one. As a result,
>> > > > records might not be deleted as it should be through time based
>> > > retention.
>> > > >
>> > > > With the current KIP proposal, if we want to ensure timely
>> deletion, we
>> > > > have the following configurations:
>> > > > 1) enable time based log compaction only : deletion is done though
>> > > > overriding the same key
>> > > > 2) enable time based log retention only: deletion is done though
>> > > > time-based retention
>> > > > 3) enable both log compaction and time based retention: Deletion is
>> not
>> > > > guaranteed.
>> > > >
>> > > > Not sure if we have use case 3 and also want deletion to happen on
>> > time.
>> > > > There are several options to address deletion issue when enable both
>> > > > compaction and retention:
>> > > > A) During log compaction, looking into record timestamp to delete
>> > expired
>> > > > records. This can be done in compaction logic itself or use
>> > > > AdminClient.deleteRecords() . But this assumes we have record
>> > timestamp.
>> > > > B) retain the lastModifed time of original segments during log
>> > > compaction.
>> > > > This requires extra meta data to record the information or not
>> grouping
>> > > > multiple segments into one during compaction.
>> > > >
>> > > > If we have use case 3 in general, I would prefer solution A and
>> rely on
>> > > > record timestamp.
>> > > >
>> > > >
>> > > > Two questions:
>> > > > Do we have use case 3? Is it nice to have or must have?
>> > > > If we have use case 3 and want to go with solution A, should we
>> > introduce
>> > > > a new configuration to enforce deletion by timestamp?
>> > > >
>> > > >
>> > > > On Tue, Aug 14, 2018 at 1:52 PM, xiongqi wu <xi...@gmail.com>
>> > wrote:
>> > > >
>> > > >> Dong,
>> > > >>
>> > > >> Thanks for the comment.
>> > > >>
>> > > >> There are two retention policy: log compaction and time based
>> > retention.
>> > > >>
>> > > >> Log compaction:
>> > > >>
>> > > >> we have use cases to keep infinite retention of a topic (only
>> > > >> compaction). GDPR cares about deletion of PII (personal
>> identifiable
>> > > >> information) data.
>> > > >> Since Kafka doesn't know what records contain PII, it relies on
>> upper
>> > > >> layer to delete those records.
>> > > >> For those infinite retention uses uses, kafka needs to provide a
>> way
>> > to
>> > > >> enforce compaction on time. This is what we try to address in this
>> > KIP.
>> > > >>
>> > > >> Time based retention,
>> > > >>
>> > > >> There are also use cases that users of Kafka might want to expire
>> all
>> > > >> their data.
>> > > >> In those cases, they can use time based retention of their topics.
>> > > >>
>> > > >>
>> > > >> Regarding your first question, if a user wants to delete a key in
>> the
>> > > >> log compaction topic, the user has to send a deletion using the
>> same
>> > > key.
>> > > >> Kafka only makes sure the deletion will happen under a certain time
>> > > >> periods (like 2 days/7 days).
>> > > >>
>> > > >> Regarding your second question. In most cases, we might want to
>> delete
>> > > >> all duplicated keys at the same time.
>> > > >> Compaction might be more efficient since we need to scan the log
>> and
>> > > find
>> > > >> all duplicates. However, the expected use case is to set the time
>> > based
>> > > >> compaction interval on the order of days, and be larger than 'min
>> > > >> compaction lag". We don't want log compaction to happen frequently
>> > since
>> > > >> it is expensive. The purpose is to help low production rate topic
>> to
>> > get
>> > > >> compacted on time. For the topic with "normal" incoming message
>> > message
>> > > >> rate, the "min dirty ratio" might have triggered the compaction
>> before
>> > > this
>> > > >> time based compaction policy takes effect.
>> > > >>
>> > > >>
>> > > >> Eno,
>> > > >>
>> > > >> For your question, like I mentioned we have long time retention use
>> > case
>> > > >> for log compacted topic, but we want to provide ability to delete
>> > > certain
>> > > >> PII records on time.
>> > > >> Kafka itself doesn't know whether a record contains sensitive
>> > > information
>> > > >> and relies on the user for deletion.
>> > > >>
>> > > >>
>> > > >> On Mon, Aug 13, 2018 at 6:58 PM, Dong Lin <li...@gmail.com>
>> > wrote:
>> > > >>
>> > > >>> Hey Xiongqi,
>> > > >>>
>> > > >>> Thanks for the KIP. I have two questions regarding the use-case
>> for
>> > > >>> meeting
>> > > >>> GDPR requirement.
>> > > >>>
>> > > >>> 1) If I recall correctly, one of the GDPR requirement is that we
>> can
>> > > not
>> > > >>> keep messages longer than e.g. 30 days in storage (e.g. Kafka).
>> Say
>> > > there
>> > > >>> exists a partition p0 which contains message1 with key1 and
>> message2
>> > > with
>> > > >>> key2. And then user keeps producing messages with key=key2 to this
>> > > >>> partition. Since message1 with key1 is never overridden, sooner or
>> > > later
>> > > >>> we
>> > > >>> will want to delete message1 and keep the latest message with
>> > key=key2.
>> > > >>> But
>> > > >>> currently it looks like log compact logic in Kafka will always put
>> > > these
>> > > >>> messages in the same segment. Will this be an issue?
>> > > >>>
>> > > >>> 2) The current KIP intends to provide the capability to delete a
>> > given
>> > > >>> message in log compacted topic. Does such use-case also require
>> Kafka
>> > > to
>> > > >>> keep the messages produced before the given message? If yes, then
>> we
>> > > can
>> > > >>> probably just use AdminClient.deleteRecords() or time-based log
>> > > retention
>> > > >>> to meet the use-case requirement. If no, do you know what is the
>> > GDPR's
>> > > >>> requirement on time-to-deletion after user explicitly requests the
>> > > >>> deletion
>> > > >>> (e.g. 1 hour, 1 day, 7 day)?
>> > > >>>
>> > > >>> Thanks,
>> > > >>> Dong
>> > > >>>
>> > > >>>
>> > > >>> On Mon, Aug 13, 2018 at 3:44 PM, xiongqi wu <xi...@gmail.com>
>> > > wrote:
>> > > >>>
>> > > >>> > Hi Eno,
>> > > >>> >
>> > > >>> > The GDPR request we are getting here at linkedin is if we get a
>> > > >>> request to
>> > > >>> > delete a record through a null key on a log compacted topic,
>> > > >>> > we want to delete the record via compaction in a given time
>> period
>> > > >>> like 2
>> > > >>> > days (whatever is required by the policy).
>> > > >>> >
>> > > >>> > There might be other issues (such as orphan log segments under
>> > > certain
>> > > >>> > conditions) that lead to GDPR problem but they are more like
>> > > >>> something we
>> > > >>> > need to fix anyway regardless of GDPR.
>> > > >>> >
>> > > >>> >
>> > > >>> > -- Xiongqi (Wesley) Wu
>> > > >>> >
>> > > >>> > On Mon, Aug 13, 2018 at 2:56 PM, Eno Thereska <
>> > > eno.thereska@gmail.com>
>> > > >>> > wrote:
>> > > >>> >
>> > > >>> > > Hello,
>> > > >>> > >
>> > > >>> > > Thanks for the KIP. I'd like to see a more precise definition
>> of
>> > > what
>> > > >>> > part
>> > > >>> > > of GDPR you are targeting as well as some sort of verification
>> > that
>> > > >>> this
>> > > >>> > > KIP actually addresses the problem. Right now I find this a
>> bit
>> > > >>> vague:
>> > > >>> > >
>> > > >>> > > "Ability to delete a log message through compaction in a
>> timely
>> > > >>> manner
>> > > >>> > has
>> > > >>> > > become an important requirement in some use cases (e.g.,
>> GDPR)"
>> > > >>> > >
>> > > >>> > >
>> > > >>> > > Is there any guarantee that after this KIP the GDPR problem is
>> > > >>> solved or
>> > > >>> > do
>> > > >>> > > we need to do something else as well, e.g., more KIPs?
>> > > >>> > >
>> > > >>> > >
>> > > >>> > > Thanks
>> > > >>> > >
>> > > >>> > > Eno
>> > > >>> > >
>> > > >>> > >
>> > > >>> > >
>> > > >>> > > On Thu, Aug 9, 2018 at 4:18 PM, xiongqi wu <
>> xiongqiwu@gmail.com>
>> > > >>> wrote:
>> > > >>> > >
>> > > >>> > > > Hi Kafka,
>> > > >>> > > >
>> > > >>> > > > This KIP tries to address GDPR concern to fulfill deletion
>> > > request
>> > > >>> on
>> > > >>> > > time
>> > > >>> > > > through time-based log compaction on a compaction enabled
>> > topic:
>> > > >>> > > >
>> > > >>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP->
>> > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP->>
>> > > >>> > > > 354%3A+Time-based+log+compaction+policy
>> > > >>> > > >
>> > > >>> > > > Any feedback will be appreciated.
>> > > >>> > > >
>> > > >>> > > >
>> > > >>> > > > Xiongqi (Wesley) Wu
>> > > >>> > > >
>> > > >>> > >
>> > > >>> >
>> > > >>>
>> > > >>
>> > > >>
>> > > >>
>> > > >> --
>> > > >> Xiongqi (Wesley) Wu
>> > > >>
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Xiongqi (Wesley) Wu
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Xiongqi (Wesley) Wu
>> > >
>> >
>> >
>> > --
>> >
>> > Brett Rann
>> >
>> > Senior DevOps Engineer
>> >
>> >
>> > Zendesk International Ltd
>> >
>> > 395 Collins Street, Melbourne VIC 3000 Australia
>> >
>> > Mobile: +61 (0) 418 826 017
>> >
>>
>>
>>
>> --
>> Xiongqi (Wesley) Wu
>>
>
>
> --
>
> Brett Rann
>
> Senior DevOps Engineer
>
>
> Zendesk International Ltd
>
> 395 Collins Street, Melbourne VIC 3000 Australia
>
> Mobile: +61 (0) 418 826 017
>


-- 

Brett Rann

Senior DevOps Engineer


Zendesk International Ltd

395 Collins Street, Melbourne VIC 3000 Australia

Mobile: +61 (0) 418 826 017

Re: [DISCUSS] KIP-354 Time-based log compaction policy

Posted by Brett Rann <br...@zendesk.com.INVALID>.
An API was suggested by Gwen and James when I discussed it with them. For
me I can think of it as a use case for scheduling compaction rather than
relying on an config based time trigger.  We're looking at creating some
potentially very large compacted topics for event sourcing and from an
operators perspective I'm concerned about potential impact of long running
compaction, especially if multiple topics run back to back.  Having the
ability to schedule them at my own time gives some peace of mind for that
concern.  Another use case might be a more urgent delete. That could be
done manually now by just modifying the max time and waiting for compaction
to run. But hitting an API end point is a bit cleaner.

But in thinking about what that mechanism would be it started to feel like
it would be a complicated implementation so we've put it aside for now. But
maybe we just haven't seen the clean way yet.



On Thu, Aug 16, 2018 at 11:22 AM xiongqi wu <xi...@gmail.com> wrote:

> Brett,
>
> Thank you for your comments.
> I was thinking since we already has immediate compaction setting by setting
> min dirty ratio to 0, so I decide to use "0" as disabled state.
> I am ok to go with -1(disable), 0 (immediate) options.
>
> For the implementation, there are a few differences between mine and
> "Xiaohe Dong"'s :
> 1) I used the estimated creation time of a log segment instead of largest
> timestamp of a log to determine the compaction eligibility, because a log
> segment might stay as an active segment up to "max compaction lag". (see
> the KIP for detail).
> 2) I measure how much bytes that we must clean to follow the "max
> compaction lag" rule, and use that to determine the order of compaction.
> 3) force active segment to roll to follow the "max compaction lag"
>
> I can share my code so we can coordinate.
>
> I haven't think about a new API to force a compaction. what is the use case
> for this one?
>
>
> On Wed, Aug 15, 2018 at 5:33 PM, Brett Rann <br...@zendesk.com.invalid>
> wrote:
>
> > We've been looking into this too.
> >
> > Mailing list:
> > https://lists.apache.org/thread.html/ed7f6a6589f94e8c2a705553f364ef
> <https://lists.apache.org/thread.html/ed7f6a6589f94e8c2a705553f364ef>
> > 599cb6915e4c3ba9b561e610e4@%3Cdev.kafka.apache.org%3E
> > jira wish: https://issues.apache.org/jira/browse/KAFKA-7137
> <https://issues.apache.org/jira/browse/KAFKA-7137>
> > confluent slack discussion:
> >
> https://confluentcommunity.slack.com/archives/C49R61XMM/p1530760121000039
> <https://confluentcommunity.slack.com/archives/C49R61XMM/p1530760121000039>
> >
> > A person on my team has started on code so you might want to coordinate:
> > https://github.com/dongxiaohe/kafka/tree/dongxiaohe/log-
> <https://github.com/dongxiaohe/kafka/tree/dongxiaohe/log->
> > cleaner-compaction-max-lifetime-2.0
> >
> > He's been working with Jason Gustafson and James Chen around the changes.
> > You can ping him on confluent slack as Xiaohe Dong.
> >
> > It's great to know others are thinking on it as well.
> >
> > You've added the requirement to force a segment roll which we hadn't
> gotten
> > to yet, which is great. I was content with it not including the active
> > segment.
> >
> > > Adding topic level configuration "max.compaction.lag.ms", and
> > corresponding broker configuration "log.cleaner.max.compaction.lag.ms",
> > which is set to 0 (disabled) by default.
> >
> > Glancing at some other settings convention seems to me to be -1 for
> > disabled (or infinite, which is more meaningful here). 0 to me implies
> > instant, a little quicker than 1.
> >
> > We've been trying to think about a way to trigger compaction as well
> > through an API call, which would need to be flagged somewhere (ZK admin/
> > space?) but we're struggling to think how that would be coordinated
> across
> > brokers and partitions. Have you given any thought to that?
> >
> >
> >
> >
> >
> >
> > On Thu, Aug 16, 2018 at 8:44 AM xiongqi wu <xi...@gmail.com> wrote:
> >
> > > Eno, Dong,
> > >
> > > I have updated the KIP. We decide not to address the issue that we
> might
> > > have for both compaction and time retention enabled topics (see the
> > > rejected alternative item 2). This KIP will only ensure log can be
> > > compacted after a specified time-interval.
> > >
> > > As suggested by Dong, we will also enforce "max.compaction.lag.ms" is
> > not
> > > less than "min.compaction.lag.ms".
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354> Time-based
> > log
> > > compaction policy
> > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354> Time-based
> > log compaction policy>
> > >
> > >
> > > On Tue, Aug 14, 2018 at 5:01 PM, xiongqi wu <xi...@gmail.com>
> wrote:
> > >
> > > >
> > > > Per discussion with Dong, he made a very good point that if
> compaction
> > > > and time based retention are both enabled on a topic, the compaction
> > > might
> > > > prevent records from being deleted on time. The reason is when
> > compacting
> > > > multiple segments into one single segment, the newly created segment
> > will
> > > > have same lastmodified timestamp as latest original segment. We lose
> > the
> > > > timestamp of all original segments except the last one. As a result,
> > > > records might not be deleted as it should be through time based
> > > retention.
> > > >
> > > > With the current KIP proposal, if we want to ensure timely deletion,
> we
> > > > have the following configurations:
> > > > 1) enable time based log compaction only : deletion is done though
> > > > overriding the same key
> > > > 2) enable time based log retention only: deletion is done though
> > > > time-based retention
> > > > 3) enable both log compaction and time based retention: Deletion is
> not
> > > > guaranteed.
> > > >
> > > > Not sure if we have use case 3 and also want deletion to happen on
> > time.
> > > > There are several options to address deletion issue when enable both
> > > > compaction and retention:
> > > > A) During log compaction, looking into record timestamp to delete
> > expired
> > > > records. This can be done in compaction logic itself or use
> > > > AdminClient.deleteRecords() . But this assumes we have record
> > timestamp.
> > > > B) retain the lastModifed time of original segments during log
> > > compaction.
> > > > This requires extra meta data to record the information or not
> grouping
> > > > multiple segments into one during compaction.
> > > >
> > > > If we have use case 3 in general, I would prefer solution A and rely
> on
> > > > record timestamp.
> > > >
> > > >
> > > > Two questions:
> > > > Do we have use case 3? Is it nice to have or must have?
> > > > If we have use case 3 and want to go with solution A, should we
> > introduce
> > > > a new configuration to enforce deletion by timestamp?
> > > >
> > > >
> > > > On Tue, Aug 14, 2018 at 1:52 PM, xiongqi wu <xi...@gmail.com>
> > wrote:
> > > >
> > > >> Dong,
> > > >>
> > > >> Thanks for the comment.
> > > >>
> > > >> There are two retention policy: log compaction and time based
> > retention.
> > > >>
> > > >> Log compaction:
> > > >>
> > > >> we have use cases to keep infinite retention of a topic (only
> > > >> compaction). GDPR cares about deletion of PII (personal identifiable
> > > >> information) data.
> > > >> Since Kafka doesn't know what records contain PII, it relies on
> upper
> > > >> layer to delete those records.
> > > >> For those infinite retention uses uses, kafka needs to provide a way
> > to
> > > >> enforce compaction on time. This is what we try to address in this
> > KIP.
> > > >>
> > > >> Time based retention,
> > > >>
> > > >> There are also use cases that users of Kafka might want to expire
> all
> > > >> their data.
> > > >> In those cases, they can use time based retention of their topics.
> > > >>
> > > >>
> > > >> Regarding your first question, if a user wants to delete a key in
> the
> > > >> log compaction topic, the user has to send a deletion using the same
> > > key.
> > > >> Kafka only makes sure the deletion will happen under a certain time
> > > >> periods (like 2 days/7 days).
> > > >>
> > > >> Regarding your second question. In most cases, we might want to
> delete
> > > >> all duplicated keys at the same time.
> > > >> Compaction might be more efficient since we need to scan the log and
> > > find
> > > >> all duplicates. However, the expected use case is to set the time
> > based
> > > >> compaction interval on the order of days, and be larger than 'min
> > > >> compaction lag". We don't want log compaction to happen frequently
> > since
> > > >> it is expensive. The purpose is to help low production rate topic to
> > get
> > > >> compacted on time. For the topic with "normal" incoming message
> > message
> > > >> rate, the "min dirty ratio" might have triggered the compaction
> before
> > > this
> > > >> time based compaction policy takes effect.
> > > >>
> > > >>
> > > >> Eno,
> > > >>
> > > >> For your question, like I mentioned we have long time retention use
> > case
> > > >> for log compacted topic, but we want to provide ability to delete
> > > certain
> > > >> PII records on time.
> > > >> Kafka itself doesn't know whether a record contains sensitive
> > > information
> > > >> and relies on the user for deletion.
> > > >>
> > > >>
> > > >> On Mon, Aug 13, 2018 at 6:58 PM, Dong Lin <li...@gmail.com>
> > wrote:
> > > >>
> > > >>> Hey Xiongqi,
> > > >>>
> > > >>> Thanks for the KIP. I have two questions regarding the use-case for
> > > >>> meeting
> > > >>> GDPR requirement.
> > > >>>
> > > >>> 1) If I recall correctly, one of the GDPR requirement is that we
> can
> > > not
> > > >>> keep messages longer than e.g. 30 days in storage (e.g. Kafka). Say
> > > there
> > > >>> exists a partition p0 which contains message1 with key1 and
> message2
> > > with
> > > >>> key2. And then user keeps producing messages with key=key2 to this
> > > >>> partition. Since message1 with key1 is never overridden, sooner or
> > > later
> > > >>> we
> > > >>> will want to delete message1 and keep the latest message with
> > key=key2.
> > > >>> But
> > > >>> currently it looks like log compact logic in Kafka will always put
> > > these
> > > >>> messages in the same segment. Will this be an issue?
> > > >>>
> > > >>> 2) The current KIP intends to provide the capability to delete a
> > given
> > > >>> message in log compacted topic. Does such use-case also require
> Kafka
> > > to
> > > >>> keep the messages produced before the given message? If yes, then
> we
> > > can
> > > >>> probably just use AdminClient.deleteRecords() or time-based log
> > > retention
> > > >>> to meet the use-case requirement. If no, do you know what is the
> > GDPR's
> > > >>> requirement on time-to-deletion after user explicitly requests the
> > > >>> deletion
> > > >>> (e.g. 1 hour, 1 day, 7 day)?
> > > >>>
> > > >>> Thanks,
> > > >>> Dong
> > > >>>
> > > >>>
> > > >>> On Mon, Aug 13, 2018 at 3:44 PM, xiongqi wu <xi...@gmail.com>
> > > wrote:
> > > >>>
> > > >>> > Hi Eno,
> > > >>> >
> > > >>> > The GDPR request we are getting here at linkedin is if we get a
> > > >>> request to
> > > >>> > delete a record through a null key on a log compacted topic,
> > > >>> > we want to delete the record via compaction in a given time
> period
> > > >>> like 2
> > > >>> > days (whatever is required by the policy).
> > > >>> >
> > > >>> > There might be other issues (such as orphan log segments under
> > > certain
> > > >>> > conditions) that lead to GDPR problem but they are more like
> > > >>> something we
> > > >>> > need to fix anyway regardless of GDPR.
> > > >>> >
> > > >>> >
> > > >>> > -- Xiongqi (Wesley) Wu
> > > >>> >
> > > >>> > On Mon, Aug 13, 2018 at 2:56 PM, Eno Thereska <
> > > eno.thereska@gmail.com>
> > > >>> > wrote:
> > > >>> >
> > > >>> > > Hello,
> > > >>> > >
> > > >>> > > Thanks for the KIP. I'd like to see a more precise definition
> of
> > > what
> > > >>> > part
> > > >>> > > of GDPR you are targeting as well as some sort of verification
> > that
> > > >>> this
> > > >>> > > KIP actually addresses the problem. Right now I find this a bit
> > > >>> vague:
> > > >>> > >
> > > >>> > > "Ability to delete a log message through compaction in a timely
> > > >>> manner
> > > >>> > has
> > > >>> > > become an important requirement in some use cases (e.g., GDPR)"
> > > >>> > >
> > > >>> > >
> > > >>> > > Is there any guarantee that after this KIP the GDPR problem is
> > > >>> solved or
> > > >>> > do
> > > >>> > > we need to do something else as well, e.g., more KIPs?
> > > >>> > >
> > > >>> > >
> > > >>> > > Thanks
> > > >>> > >
> > > >>> > > Eno
> > > >>> > >
> > > >>> > >
> > > >>> > >
> > > >>> > > On Thu, Aug 9, 2018 at 4:18 PM, xiongqi wu <
> xiongqiwu@gmail.com>
> > > >>> wrote:
> > > >>> > >
> > > >>> > > > Hi Kafka,
> > > >>> > > >
> > > >>> > > > This KIP tries to address GDPR concern to fulfill deletion
> > > request
> > > >>> on
> > > >>> > > time
> > > >>> > > > through time-based log compaction on a compaction enabled
> > topic:
> > > >>> > > >
> > > >>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP->
> > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP->>
> > > >>> > > > 354%3A+Time-based+log+compaction+policy
> > > >>> > > >
> > > >>> > > > Any feedback will be appreciated.
> > > >>> > > >
> > > >>> > > >
> > > >>> > > > Xiongqi (Wesley) Wu
> > > >>> > > >
> > > >>> > >
> > > >>> >
> > > >>>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Xiongqi (Wesley) Wu
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Xiongqi (Wesley) Wu
> > > >
> > >
> > >
> > >
> > > --
> > > Xiongqi (Wesley) Wu
> > >
> >
> >
> > --
> >
> > Brett Rann
> >
> > Senior DevOps Engineer
> >
> >
> > Zendesk International Ltd
> >
> > 395 Collins Street, Melbourne VIC 3000 Australia
> >
> > Mobile: +61 (0) 418 826 017
> >
>
>
>
> --
> Xiongqi (Wesley) Wu
>


-- 

Brett Rann

Senior DevOps Engineer


Zendesk International Ltd

395 Collins Street, Melbourne VIC 3000 Australia

Mobile: +61 (0) 418 826 017

Re: [DISCUSS] KIP-354 Time-based log compaction policy

Posted by xiongqi wu <xi...@gmail.com>.
Brett,

Thank you for your comments.
I was thinking since we already has immediate compaction setting by setting
min dirty ratio to 0, so I decide to use "0" as disabled state.
I am ok to go with -1(disable), 0 (immediate) options.

For the implementation, there are a few differences between mine and
"Xiaohe Dong"'s :
1) I used the estimated creation time of a log segment instead of largest
timestamp of a log to determine the compaction eligibility, because a log
segment might stay as an active segment up to "max compaction lag". (see
the KIP for detail).
2) I measure how much bytes that we must clean to follow the "max
compaction lag" rule, and use that to determine the order of compaction.
3) force active segment to roll to follow the "max compaction lag"

I can share my code so we can coordinate.

I haven't think about a new API to force a compaction. what is the use case
for this one?


On Wed, Aug 15, 2018 at 5:33 PM, Brett Rann <br...@zendesk.com.invalid>
wrote:

> We've been looking into this too.
>
> Mailing list:
> https://lists.apache.org/thread.html/ed7f6a6589f94e8c2a705553f364ef
> 599cb6915e4c3ba9b561e610e4@%3Cdev.kafka.apache.org%3E
> jira wish: https://issues.apache.org/jira/browse/KAFKA-7137
> confluent slack discussion:
> https://confluentcommunity.slack.com/archives/C49R61XMM/p1530760121000039
>
> A person on my team has started on code so you might want to coordinate:
> https://github.com/dongxiaohe/kafka/tree/dongxiaohe/log-
> cleaner-compaction-max-lifetime-2.0
>
>  He's been working with Jason Gustafson and James Chen around the changes.
> You can ping him on confluent slack as Xiaohe Dong.
>
> It's great to know others are thinking on it as well.
>
> You've added the requirement to force a segment roll which we hadn't gotten
> to yet, which is great. I was content with it not including the active
> segment.
>
> > Adding topic level configuration "max.compaction.lag.ms",  and
> corresponding broker configuration "log.cleaner.max.compaction.lag.ms",
> which is set to 0 (disabled) by default.
>
> Glancing at some other settings convention seems to me to be -1 for
> disabled (or infinite, which is more meaningful here).  0 to me implies
> instant, a little quicker than 1.
>
> We've been trying to think about a way to trigger compaction as well
> through an API call, which would need to be flagged somewhere (ZK admin/
> space?) but we're struggling to think how that would be coordinated across
> brokers and partitions.  Have you given any thought to that?
>
>
>
>
>
>
> On Thu, Aug 16, 2018 at 8:44 AM xiongqi wu <xi...@gmail.com> wrote:
>
> > Eno, Dong,
> >
> > I have updated the KIP. We decide not to address the issue that we might
> > have for both compaction and time retention enabled topics (see the
> > rejected alternative item 2). This KIP will only ensure log can be
> > compacted after a specified time-interval.
> >
> > As suggested by Dong, we will also enforce "max.compaction.lag.ms" is
> not
> > less than "min.compaction.lag.ms".
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354: Time-based
> log
> > compaction policy
> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354: Time-based
> log compaction policy>
> >
> >
> > On Tue, Aug 14, 2018 at 5:01 PM, xiongqi wu <xi...@gmail.com> wrote:
> >
> > >
> > > Per discussion with Dong, he made a very good point that if compaction
> > > and time based retention are both enabled on a topic, the compaction
> > might
> > > prevent records from being deleted on time. The reason is when
> compacting
> > > multiple segments into one single segment, the newly created segment
> will
> > > have same lastmodified timestamp as latest original segment. We lose
> the
> > > timestamp of all original segments except the last one. As a result,
> > > records might not be deleted as it should be through time based
> > retention.
> > >
> > > With the current KIP proposal, if we want to ensure timely deletion, we
> > > have the following configurations:
> > > 1) enable time based log compaction only : deletion is done though
> > > overriding the same key
> > > 2) enable time based log retention only: deletion is done though
> > > time-based retention
> > > 3) enable both log compaction and time based retention: Deletion is not
> > > guaranteed.
> > >
> > > Not sure if we have use case 3 and also want deletion to happen on
> time.
> > > There are several options to address deletion issue when enable both
> > > compaction and retention:
> > > A) During log compaction, looking into record timestamp to delete
> expired
> > > records. This can be done in compaction logic itself or use
> > > AdminClient.deleteRecords() . But this assumes we have record
> timestamp.
> > > B) retain the lastModifed time of original segments during log
> > compaction.
> > > This requires extra meta data to record the information or not grouping
> > > multiple segments into one during compaction.
> > >
> > > If we have use case 3 in general, I would prefer solution A and rely on
> > > record timestamp.
> > >
> > >
> > > Two questions:
> > > Do we have use case 3? Is it nice to have or must have?
> > > If we have use case 3 and want to go with solution A, should we
> introduce
> > > a new configuration to enforce deletion by timestamp?
> > >
> > >
> > > On Tue, Aug 14, 2018 at 1:52 PM, xiongqi wu <xi...@gmail.com>
> wrote:
> > >
> > >> Dong,
> > >>
> > >> Thanks for the comment.
> > >>
> > >> There are two retention policy: log compaction and time based
> retention.
> > >>
> > >> Log compaction:
> > >>
> > >> we have use cases to keep infinite retention of a topic (only
> > >> compaction). GDPR cares about deletion of PII (personal identifiable
> > >> information) data.
> > >> Since Kafka doesn't know what records contain PII, it relies on upper
> > >> layer to delete those records.
> > >> For those infinite retention uses uses, kafka needs to provide a way
> to
> > >> enforce compaction on time. This is what we try to address in this
> KIP.
> > >>
> > >> Time based retention,
> > >>
> > >> There are also use cases that users of Kafka might want to expire all
> > >> their data.
> > >> In those cases, they can use time based retention of their topics.
> > >>
> > >>
> > >> Regarding your first question, if a user wants to delete a key in the
> > >> log compaction topic, the user has to send a deletion using the same
> > key.
> > >> Kafka only makes sure the deletion will happen under a certain time
> > >> periods (like 2 days/7 days).
> > >>
> > >> Regarding your second question. In most cases, we might want to delete
> > >> all duplicated keys at the same time.
> > >> Compaction might be more efficient since we need to scan the log and
> > find
> > >> all duplicates. However, the expected use case is to set the time
> based
> > >> compaction interval on the order of days, and be larger than 'min
> > >> compaction lag". We don't want log compaction to happen frequently
> since
> > >> it is expensive. The purpose is to help low production rate topic to
> get
> > >> compacted on time. For the topic with "normal" incoming message
> message
> > >> rate, the "min dirty ratio" might have triggered the compaction before
> > this
> > >> time based compaction policy takes effect.
> > >>
> > >>
> > >> Eno,
> > >>
> > >> For your question, like I mentioned we have long time retention use
> case
> > >> for log compacted topic, but we want to provide ability to delete
> > certain
> > >> PII records on time.
> > >> Kafka itself doesn't know whether a record contains sensitive
> > information
> > >> and relies on the user for deletion.
> > >>
> > >>
> > >> On Mon, Aug 13, 2018 at 6:58 PM, Dong Lin <li...@gmail.com>
> wrote:
> > >>
> > >>> Hey Xiongqi,
> > >>>
> > >>> Thanks for the KIP. I have two questions regarding the use-case for
> > >>> meeting
> > >>> GDPR requirement.
> > >>>
> > >>> 1) If I recall correctly, one of the GDPR requirement is that we can
> > not
> > >>> keep messages longer than e.g. 30 days in storage (e.g. Kafka). Say
> > there
> > >>> exists a partition p0 which contains message1 with key1 and message2
> > with
> > >>> key2. And then user keeps producing messages with key=key2 to this
> > >>> partition. Since message1 with key1 is never overridden, sooner or
> > later
> > >>> we
> > >>> will want to delete message1 and keep the latest message with
> key=key2.
> > >>> But
> > >>> currently it looks like log compact logic in Kafka will always put
> > these
> > >>> messages in the same segment. Will this be an issue?
> > >>>
> > >>> 2) The current KIP intends to provide the capability to delete a
> given
> > >>> message in log compacted topic. Does such use-case also require Kafka
> > to
> > >>> keep the messages produced before the given message? If yes, then we
> > can
> > >>> probably just use AdminClient.deleteRecords() or time-based log
> > retention
> > >>> to meet the use-case requirement. If no, do you know what is the
> GDPR's
> > >>> requirement on time-to-deletion after user explicitly requests the
> > >>> deletion
> > >>> (e.g. 1 hour, 1 day, 7 day)?
> > >>>
> > >>> Thanks,
> > >>> Dong
> > >>>
> > >>>
> > >>> On Mon, Aug 13, 2018 at 3:44 PM, xiongqi wu <xi...@gmail.com>
> > wrote:
> > >>>
> > >>> > Hi Eno,
> > >>> >
> > >>> > The GDPR request we are getting here at linkedin is if we get a
> > >>> request to
> > >>> > delete a record through a null key on a log compacted topic,
> > >>> > we want to delete the record via compaction in a given time period
> > >>> like 2
> > >>> > days (whatever is required by the policy).
> > >>> >
> > >>> > There might be other issues (such as orphan log segments under
> > certain
> > >>> > conditions) that lead to GDPR problem but they are more like
> > >>> something we
> > >>> > need to fix anyway regardless of GDPR.
> > >>> >
> > >>> >
> > >>> > -- Xiongqi (Wesley) Wu
> > >>> >
> > >>> > On Mon, Aug 13, 2018 at 2:56 PM, Eno Thereska <
> > eno.thereska@gmail.com>
> > >>> > wrote:
> > >>> >
> > >>> > > Hello,
> > >>> > >
> > >>> > > Thanks for the KIP. I'd like to see a more precise definition of
> > what
> > >>> > part
> > >>> > > of GDPR you are targeting as well as some sort of verification
> that
> > >>> this
> > >>> > > KIP actually addresses the problem. Right now I find this a bit
> > >>> vague:
> > >>> > >
> > >>> > > "Ability to delete a log message through compaction in a timely
> > >>> manner
> > >>> > has
> > >>> > > become an important requirement in some use cases (e.g., GDPR)"
> > >>> > >
> > >>> > >
> > >>> > > Is there any guarantee that after this KIP the GDPR problem is
> > >>> solved or
> > >>> > do
> > >>> > > we need to do something else as well, e.g., more KIPs?
> > >>> > >
> > >>> > >
> > >>> > > Thanks
> > >>> > >
> > >>> > > Eno
> > >>> > >
> > >>> > >
> > >>> > >
> > >>> > > On Thu, Aug 9, 2018 at 4:18 PM, xiongqi wu <xi...@gmail.com>
> > >>> wrote:
> > >>> > >
> > >>> > > > Hi Kafka,
> > >>> > > >
> > >>> > > > This KIP tries to address GDPR concern to fulfill deletion
> > request
> > >>> on
> > >>> > > time
> > >>> > > > through time-based log compaction on a compaction enabled
> topic:
> > >>> > > >
> > >>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP->
> > >>> > > > 354%3A+Time-based+log+compaction+policy
> > >>> > > >
> > >>> > > > Any feedback will be appreciated.
> > >>> > > >
> > >>> > > >
> > >>> > > > Xiongqi (Wesley) Wu
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Xiongqi (Wesley) Wu
> > >>
> > >
> > >
> > >
> > > --
> > > Xiongqi (Wesley) Wu
> > >
> >
> >
> >
> > --
> > Xiongqi (Wesley) Wu
> >
>
>
> --
>
> Brett Rann
>
> Senior DevOps Engineer
>
>
> Zendesk International Ltd
>
> 395 Collins Street, Melbourne VIC 3000 Australia
>
> Mobile: +61 (0) 418 826 017
>



-- 
Xiongqi (Wesley) Wu

Re: [DISCUSS] KIP-354 Time-based log compaction policy

Posted by Brett Rann <br...@zendesk.com.INVALID>.
We've been looking into this too.

Mailing list:
https://lists.apache.org/thread.html/ed7f6a6589f94e8c2a705553f364ef599cb6915e4c3ba9b561e610e4@%3Cdev.kafka.apache.org%3E
jira wish: https://issues.apache.org/jira/browse/KAFKA-7137
confluent slack discussion:
https://confluentcommunity.slack.com/archives/C49R61XMM/p1530760121000039

A person on my team has started on code so you might want to coordinate:
https://github.com/dongxiaohe/kafka/tree/dongxiaohe/log-cleaner-compaction-max-lifetime-2.0

 He's been working with Jason Gustafson and James Chen around the changes.
You can ping him on confluent slack as Xiaohe Dong.

It's great to know others are thinking on it as well.

You've added the requirement to force a segment roll which we hadn't gotten
to yet, which is great. I was content with it not including the active
segment.

> Adding topic level configuration "max.compaction.lag.ms",  and
corresponding broker configuration "log.cleaner.max.compaction.lag.ms",
which is set to 0 (disabled) by default.

Glancing at some other settings convention seems to me to be -1 for
disabled (or infinite, which is more meaningful here).  0 to me implies
instant, a little quicker than 1.

We've been trying to think about a way to trigger compaction as well
through an API call, which would need to be flagged somewhere (ZK admin/
space?) but we're struggling to think how that would be coordinated across
brokers and partitions.  Have you given any thought to that?






On Thu, Aug 16, 2018 at 8:44 AM xiongqi wu <xi...@gmail.com> wrote:

> Eno, Dong,
>
> I have updated the KIP. We decide not to address the issue that we might
> have for both compaction and time retention enabled topics (see the
> rejected alternative item 2). This KIP will only ensure log can be
> compacted after a specified time-interval.
>
> As suggested by Dong, we will also enforce "max.compaction.lag.ms" is not
> less than "min.compaction.lag.ms".
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354: Time-based log
> compaction policy
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354: Time-based log compaction policy>
>
>
> On Tue, Aug 14, 2018 at 5:01 PM, xiongqi wu <xi...@gmail.com> wrote:
>
> >
> > Per discussion with Dong, he made a very good point that if compaction
> > and time based retention are both enabled on a topic, the compaction
> might
> > prevent records from being deleted on time. The reason is when compacting
> > multiple segments into one single segment, the newly created segment will
> > have same lastmodified timestamp as latest original segment. We lose the
> > timestamp of all original segments except the last one. As a result,
> > records might not be deleted as it should be through time based
> retention.
> >
> > With the current KIP proposal, if we want to ensure timely deletion, we
> > have the following configurations:
> > 1) enable time based log compaction only : deletion is done though
> > overriding the same key
> > 2) enable time based log retention only: deletion is done though
> > time-based retention
> > 3) enable both log compaction and time based retention: Deletion is not
> > guaranteed.
> >
> > Not sure if we have use case 3 and also want deletion to happen on time.
> > There are several options to address deletion issue when enable both
> > compaction and retention:
> > A) During log compaction, looking into record timestamp to delete expired
> > records. This can be done in compaction logic itself or use
> > AdminClient.deleteRecords() . But this assumes we have record timestamp.
> > B) retain the lastModifed time of original segments during log
> compaction.
> > This requires extra meta data to record the information or not grouping
> > multiple segments into one during compaction.
> >
> > If we have use case 3 in general, I would prefer solution A and rely on
> > record timestamp.
> >
> >
> > Two questions:
> > Do we have use case 3? Is it nice to have or must have?
> > If we have use case 3 and want to go with solution A, should we introduce
> > a new configuration to enforce deletion by timestamp?
> >
> >
> > On Tue, Aug 14, 2018 at 1:52 PM, xiongqi wu <xi...@gmail.com> wrote:
> >
> >> Dong,
> >>
> >> Thanks for the comment.
> >>
> >> There are two retention policy: log compaction and time based retention.
> >>
> >> Log compaction:
> >>
> >> we have use cases to keep infinite retention of a topic (only
> >> compaction). GDPR cares about deletion of PII (personal identifiable
> >> information) data.
> >> Since Kafka doesn't know what records contain PII, it relies on upper
> >> layer to delete those records.
> >> For those infinite retention uses uses, kafka needs to provide a way to
> >> enforce compaction on time. This is what we try to address in this KIP.
> >>
> >> Time based retention,
> >>
> >> There are also use cases that users of Kafka might want to expire all
> >> their data.
> >> In those cases, they can use time based retention of their topics.
> >>
> >>
> >> Regarding your first question, if a user wants to delete a key in the
> >> log compaction topic, the user has to send a deletion using the same
> key.
> >> Kafka only makes sure the deletion will happen under a certain time
> >> periods (like 2 days/7 days).
> >>
> >> Regarding your second question. In most cases, we might want to delete
> >> all duplicated keys at the same time.
> >> Compaction might be more efficient since we need to scan the log and
> find
> >> all duplicates. However, the expected use case is to set the time based
> >> compaction interval on the order of days, and be larger than 'min
> >> compaction lag". We don't want log compaction to happen frequently since
> >> it is expensive. The purpose is to help low production rate topic to get
> >> compacted on time. For the topic with "normal" incoming message message
> >> rate, the "min dirty ratio" might have triggered the compaction before
> this
> >> time based compaction policy takes effect.
> >>
> >>
> >> Eno,
> >>
> >> For your question, like I mentioned we have long time retention use case
> >> for log compacted topic, but we want to provide ability to delete
> certain
> >> PII records on time.
> >> Kafka itself doesn't know whether a record contains sensitive
> information
> >> and relies on the user for deletion.
> >>
> >>
> >> On Mon, Aug 13, 2018 at 6:58 PM, Dong Lin <li...@gmail.com> wrote:
> >>
> >>> Hey Xiongqi,
> >>>
> >>> Thanks for the KIP. I have two questions regarding the use-case for
> >>> meeting
> >>> GDPR requirement.
> >>>
> >>> 1) If I recall correctly, one of the GDPR requirement is that we can
> not
> >>> keep messages longer than e.g. 30 days in storage (e.g. Kafka). Say
> there
> >>> exists a partition p0 which contains message1 with key1 and message2
> with
> >>> key2. And then user keeps producing messages with key=key2 to this
> >>> partition. Since message1 with key1 is never overridden, sooner or
> later
> >>> we
> >>> will want to delete message1 and keep the latest message with key=key2.
> >>> But
> >>> currently it looks like log compact logic in Kafka will always put
> these
> >>> messages in the same segment. Will this be an issue?
> >>>
> >>> 2) The current KIP intends to provide the capability to delete a given
> >>> message in log compacted topic. Does such use-case also require Kafka
> to
> >>> keep the messages produced before the given message? If yes, then we
> can
> >>> probably just use AdminClient.deleteRecords() or time-based log
> retention
> >>> to meet the use-case requirement. If no, do you know what is the GDPR's
> >>> requirement on time-to-deletion after user explicitly requests the
> >>> deletion
> >>> (e.g. 1 hour, 1 day, 7 day)?
> >>>
> >>> Thanks,
> >>> Dong
> >>>
> >>>
> >>> On Mon, Aug 13, 2018 at 3:44 PM, xiongqi wu <xi...@gmail.com>
> wrote:
> >>>
> >>> > Hi Eno,
> >>> >
> >>> > The GDPR request we are getting here at linkedin is if we get a
> >>> request to
> >>> > delete a record through a null key on a log compacted topic,
> >>> > we want to delete the record via compaction in a given time period
> >>> like 2
> >>> > days (whatever is required by the policy).
> >>> >
> >>> > There might be other issues (such as orphan log segments under
> certain
> >>> > conditions) that lead to GDPR problem but they are more like
> >>> something we
> >>> > need to fix anyway regardless of GDPR.
> >>> >
> >>> >
> >>> > -- Xiongqi (Wesley) Wu
> >>> >
> >>> > On Mon, Aug 13, 2018 at 2:56 PM, Eno Thereska <
> eno.thereska@gmail.com>
> >>> > wrote:
> >>> >
> >>> > > Hello,
> >>> > >
> >>> > > Thanks for the KIP. I'd like to see a more precise definition of
> what
> >>> > part
> >>> > > of GDPR you are targeting as well as some sort of verification that
> >>> this
> >>> > > KIP actually addresses the problem. Right now I find this a bit
> >>> vague:
> >>> > >
> >>> > > "Ability to delete a log message through compaction in a timely
> >>> manner
> >>> > has
> >>> > > become an important requirement in some use cases (e.g., GDPR)"
> >>> > >
> >>> > >
> >>> > > Is there any guarantee that after this KIP the GDPR problem is
> >>> solved or
> >>> > do
> >>> > > we need to do something else as well, e.g., more KIPs?
> >>> > >
> >>> > >
> >>> > > Thanks
> >>> > >
> >>> > > Eno
> >>> > >
> >>> > >
> >>> > >
> >>> > > On Thu, Aug 9, 2018 at 4:18 PM, xiongqi wu <xi...@gmail.com>
> >>> wrote:
> >>> > >
> >>> > > > Hi Kafka,
> >>> > > >
> >>> > > > This KIP tries to address GDPR concern to fulfill deletion
> request
> >>> on
> >>> > > time
> >>> > > > through time-based log compaction on a compaction enabled topic:
> >>> > > >
> >>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP->
> >>> > > > 354%3A+Time-based+log+compaction+policy
> >>> > > >
> >>> > > > Any feedback will be appreciated.
> >>> > > >
> >>> > > >
> >>> > > > Xiongqi (Wesley) Wu
> >>> > > >
> >>> > >
> >>> >
> >>>
> >>
> >>
> >>
> >> --
> >> Xiongqi (Wesley) Wu
> >>
> >
> >
> >
> > --
> > Xiongqi (Wesley) Wu
> >
>
>
>
> --
> Xiongqi (Wesley) Wu
>


-- 

Brett Rann

Senior DevOps Engineer


Zendesk International Ltd

395 Collins Street, Melbourne VIC 3000 Australia

Mobile: +61 (0) 418 826 017

Re: [DISCUSS] KIP-354 Time-based log compaction policy

Posted by xiongqi wu <xi...@gmail.com>.
Eno, Dong,

I have updated the KIP.  We decide not to address the issue that we might
have for both compaction and time retention enabled topics (see the
rejected alternative item 2).  This KIP will only ensure log can be
compacted after a specified time-interval.

As suggested by Dong,  we will also enforce "max.compaction.lag.ms" is not
less than "min.compaction.lag.ms".

https://cwiki.apache.org/confluence/display/KAFKA/KIP-354%3A+Time-based+log+compaction+policy


On Tue, Aug 14, 2018 at 5:01 PM, xiongqi wu <xi...@gmail.com> wrote:

>
> Per discussion with Dong,  he made a very good point that if compaction
> and time based retention are both enabled on a topic, the compaction might
> prevent records from being deleted on time.  The reason is when compacting
> multiple segments into one single segment, the newly created segment will
> have same lastmodified timestamp as latest original segment. We lose the
> timestamp of all original segments except the last one. As a result,
> records might not be deleted as it should be through time based retention.
>
> With the current KIP proposal,   if we want to ensure timely deletion,  we
> have the following configurations:
> 1) enable time based log compaction only :  deletion is done though
> overriding the same key
> 2) enable time based log retention only: deletion is done though
> time-based retention
> 3) enable both log compaction and time based retention:    Deletion is not
> guaranteed.
>
> Not sure if we have use case 3 and also want deletion to happen on time.
> There are several options to address deletion issue when enable both
> compaction and retention:
> A) During log compaction, looking into record timestamp to delete expired
> records. This can be done in compaction logic itself or use
> AdminClient.deleteRecords() . But this assumes we have record timestamp.
> B) retain the lastModifed time of original segments during log compaction.
> This requires extra meta data to record the information or not grouping
> multiple segments into one during compaction.
>
> If we have use case 3 in general,  I would prefer solution A and rely on
> record timestamp.
>
>
> Two questions:
> Do we have use case 3? Is it nice to have or must have?
> If we have use case 3 and want to go with solution A,  should we introduce
> a new configuration to enforce deletion by timestamp?
>
>
> On Tue, Aug 14, 2018 at 1:52 PM, xiongqi wu <xi...@gmail.com> wrote:
>
>> Dong,
>>
>> Thanks for the comment.
>>
>> There are two retention policy: log compaction and time based retention.
>>
>> Log compaction:
>>
>> we have use cases to keep infinite retention of a topic (only
>> compaction).  GDPR cares about deletion of PII  (personal identifiable
>> information) data.
>> Since Kafka doesn't know what records contain PII, it relies on upper
>> layer to delete those records.
>> For those infinite retention uses uses,  kafka needs to provide a way to
>> enforce compaction on time. This is what we try to address in this KIP.
>>
>> Time based retention,
>>
>> There are also use cases that users of Kafka might want to expire all
>> their data.
>> In those cases, they can use time based retention of their topics.
>>
>>
>> Regarding your first question,  if a user wants to delete a key in the
>> log compaction topic,  the user has to send a deletion using the same key.
>> Kafka only makes sure the deletion will happen under a certain time
>> periods (like 2 days/7 days).
>>
>> Regarding your second question.  In most cases, we might want to delete
>> all duplicated keys at the same time.
>> Compaction might be more efficient since we need to scan the log and find
>> all duplicates.  However,  the expected use case is to set the time based
>> compaction interval on the order of days,  and be larger than 'min
>> compaction lag".  We don't want log compaction to happen frequently since
>> it is expensive.  The purpose is to help low production rate topic to get
>> compacted on time.  For the topic with "normal" incoming message message
>> rate, the "min dirty ratio" might have triggered the compaction before this
>> time based compaction policy takes effect.
>>
>>
>> Eno,
>>
>> For your question,  like I mentioned we have long time retention use case
>> for log compacted topic, but we want to provide ability to delete certain
>> PII records on time.
>> Kafka itself doesn't know whether a record contains sensitive information
>> and relies on the user for deletion.
>>
>>
>> On Mon, Aug 13, 2018 at 6:58 PM, Dong Lin <li...@gmail.com> wrote:
>>
>>> Hey Xiongqi,
>>>
>>> Thanks for the KIP. I have two questions regarding the use-case for
>>> meeting
>>> GDPR requirement.
>>>
>>> 1) If I recall correctly, one of the GDPR requirement is that we can not
>>> keep messages longer than e.g. 30 days in storage (e.g. Kafka). Say there
>>> exists a partition p0 which contains message1 with key1 and message2 with
>>> key2. And then user keeps producing messages with key=key2 to this
>>> partition. Since message1 with key1 is never overridden, sooner or later
>>> we
>>> will want to delete message1 and keep the latest message with key=key2.
>>> But
>>> currently it looks like log compact logic in Kafka will always put these
>>> messages in the same segment. Will this be an issue?
>>>
>>> 2) The current KIP intends to provide the capability to delete a given
>>> message in log compacted topic. Does such use-case also require Kafka to
>>> keep the messages produced before the given message? If yes, then we can
>>> probably just use AdminClient.deleteRecords() or time-based log retention
>>> to meet the use-case requirement. If no, do you know what is the GDPR's
>>> requirement on time-to-deletion after user explicitly requests the
>>> deletion
>>> (e.g. 1 hour, 1 day, 7 day)?
>>>
>>> Thanks,
>>> Dong
>>>
>>>
>>> On Mon, Aug 13, 2018 at 3:44 PM, xiongqi wu <xi...@gmail.com> wrote:
>>>
>>> > Hi Eno,
>>> >
>>> > The GDPR request we are getting here at linkedin is if we get a
>>> request to
>>> > delete a record through a null key on a log compacted topic,
>>> > we want to delete the record via compaction in a given time period
>>> like 2
>>> > days (whatever is required by the policy).
>>> >
>>> > There might be other issues (such as orphan log segments under certain
>>> > conditions)  that lead to GDPR problem but they are more like
>>> something we
>>> > need to fix anyway regardless of GDPR.
>>> >
>>> >
>>> > -- Xiongqi (Wesley) Wu
>>> >
>>> > On Mon, Aug 13, 2018 at 2:56 PM, Eno Thereska <en...@gmail.com>
>>> > wrote:
>>> >
>>> > > Hello,
>>> > >
>>> > > Thanks for the KIP. I'd like to see a more precise definition of what
>>> > part
>>> > > of GDPR you are targeting as well as some sort of verification that
>>> this
>>> > > KIP actually addresses the problem. Right now I find this a bit
>>> vague:
>>> > >
>>> > > "Ability to delete a log message through compaction in a timely
>>> manner
>>> > has
>>> > > become an important requirement in some use cases (e.g., GDPR)"
>>> > >
>>> > >
>>> > > Is there any guarantee that after this KIP the GDPR problem is
>>> solved or
>>> > do
>>> > > we need to do something else as well, e.g., more KIPs?
>>> > >
>>> > >
>>> > > Thanks
>>> > >
>>> > > Eno
>>> > >
>>> > >
>>> > >
>>> > > On Thu, Aug 9, 2018 at 4:18 PM, xiongqi wu <xi...@gmail.com>
>>> wrote:
>>> > >
>>> > > > Hi Kafka,
>>> > > >
>>> > > > This KIP tries to address GDPR concern to fulfill deletion request
>>> on
>>> > > time
>>> > > > through time-based log compaction on a compaction enabled topic:
>>> > > >
>>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>> > > > 354%3A+Time-based+log+compaction+policy
>>> > > >
>>> > > > Any feedback will be appreciated.
>>> > > >
>>> > > >
>>> > > > Xiongqi (Wesley) Wu
>>> > > >
>>> > >
>>> >
>>>
>>
>>
>>
>> --
>> Xiongqi (Wesley) Wu
>>
>
>
>
> --
> Xiongqi (Wesley) Wu
>



-- 
Xiongqi (Wesley) Wu

Re: [DISCUSS] KIP-354 Time-based log compaction policy

Posted by xiongqi wu <xi...@gmail.com>.
Per discussion with Dong,  he made a very good point that if compaction and
time based retention are both enabled on a topic, the compaction might
prevent records from being deleted on time.  The reason is when compacting
multiple segments into one single segment, the newly created segment will
have same lastmodified timestamp as latest original segment. We lose the
timestamp of all original segments except the last one. As a result,
records might not be deleted as it should be through time based retention.

With the current KIP proposal,   if we want to ensure timely deletion,  we
have the following configurations:
1) enable time based log compaction only :  deletion is done though
overriding the same key
2) enable time based log retention only: deletion is done though time-based
retention
3) enable both log compaction and time based retention:    Deletion is not
guaranteed.

Not sure if we have use case 3 and also want deletion to happen on time.
There are several options to address deletion issue when enable both
compaction and retention:
A) During log compaction, looking into record timestamp to delete expired
records. This can be done in compaction logic itself or use
AdminClient.deleteRecords() . But this assumes we have record timestamp.
B) retain the lastModifed time of original segments during log compaction.
This requires extra meta data to record the information or not grouping
multiple segments into one during compaction.

If we have use case 3 in general,  I would prefer solution A and rely on
record timestamp.


Two questions:
Do we have use case 3? Is it nice to have or must have?
If we have use case 3 and want to go with solution A,  should we introduce
a new configuration to enforce deletion by timestamp?


On Tue, Aug 14, 2018 at 1:52 PM, xiongqi wu <xi...@gmail.com> wrote:

> Dong,
>
> Thanks for the comment.
>
> There are two retention policy: log compaction and time based retention.
>
> Log compaction:
>
> we have use cases to keep infinite retention of a topic (only
> compaction).  GDPR cares about deletion of PII  (personal identifiable
> information) data.
> Since Kafka doesn't know what records contain PII, it relies on upper
> layer to delete those records.
> For those infinite retention uses uses,  kafka needs to provide a way to
> enforce compaction on time. This is what we try to address in this KIP.
>
> Time based retention,
>
> There are also use cases that users of Kafka might want to expire all
> their data.
> In those cases, they can use time based retention of their topics.
>
>
> Regarding your first question,  if a user wants to delete a key in the log
> compaction topic,  the user has to send a deletion using the same key.
> Kafka only makes sure the deletion will happen under a certain time
> periods (like 2 days/7 days).
>
> Regarding your second question.  In most cases, we might want to delete
> all duplicated keys at the same time.
> Compaction might be more efficient since we need to scan the log and find
> all duplicates.  However,  the expected use case is to set the time based
> compaction interval on the order of days,  and be larger than 'min
> compaction lag".  We don't want log compaction to happen frequently since
> it is expensive.  The purpose is to help low production rate topic to get
> compacted on time.  For the topic with "normal" incoming message message
> rate, the "min dirty ratio" might have triggered the compaction before this
> time based compaction policy takes effect.
>
>
> Eno,
>
> For your question,  like I mentioned we have long time retention use case
> for log compacted topic, but we want to provide ability to delete certain
> PII records on time.
> Kafka itself doesn't know whether a record contains sensitive information
> and relies on the user for deletion.
>
>
> On Mon, Aug 13, 2018 at 6:58 PM, Dong Lin <li...@gmail.com> wrote:
>
>> Hey Xiongqi,
>>
>> Thanks for the KIP. I have two questions regarding the use-case for
>> meeting
>> GDPR requirement.
>>
>> 1) If I recall correctly, one of the GDPR requirement is that we can not
>> keep messages longer than e.g. 30 days in storage (e.g. Kafka). Say there
>> exists a partition p0 which contains message1 with key1 and message2 with
>> key2. And then user keeps producing messages with key=key2 to this
>> partition. Since message1 with key1 is never overridden, sooner or later
>> we
>> will want to delete message1 and keep the latest message with key=key2.
>> But
>> currently it looks like log compact logic in Kafka will always put these
>> messages in the same segment. Will this be an issue?
>>
>> 2) The current KIP intends to provide the capability to delete a given
>> message in log compacted topic. Does such use-case also require Kafka to
>> keep the messages produced before the given message? If yes, then we can
>> probably just use AdminClient.deleteRecords() or time-based log retention
>> to meet the use-case requirement. If no, do you know what is the GDPR's
>> requirement on time-to-deletion after user explicitly requests the
>> deletion
>> (e.g. 1 hour, 1 day, 7 day)?
>>
>> Thanks,
>> Dong
>>
>>
>> On Mon, Aug 13, 2018 at 3:44 PM, xiongqi wu <xi...@gmail.com> wrote:
>>
>> > Hi Eno,
>> >
>> > The GDPR request we are getting here at linkedin is if we get a request
>> to
>> > delete a record through a null key on a log compacted topic,
>> > we want to delete the record via compaction in a given time period like
>> 2
>> > days (whatever is required by the policy).
>> >
>> > There might be other issues (such as orphan log segments under certain
>> > conditions)  that lead to GDPR problem but they are more like something
>> we
>> > need to fix anyway regardless of GDPR.
>> >
>> >
>> > -- Xiongqi (Wesley) Wu
>> >
>> > On Mon, Aug 13, 2018 at 2:56 PM, Eno Thereska <en...@gmail.com>
>> > wrote:
>> >
>> > > Hello,
>> > >
>> > > Thanks for the KIP. I'd like to see a more precise definition of what
>> > part
>> > > of GDPR you are targeting as well as some sort of verification that
>> this
>> > > KIP actually addresses the problem. Right now I find this a bit vague:
>> > >
>> > > "Ability to delete a log message through compaction in a timely manner
>> > has
>> > > become an important requirement in some use cases (e.g., GDPR)"
>> > >
>> > >
>> > > Is there any guarantee that after this KIP the GDPR problem is solved
>> or
>> > do
>> > > we need to do something else as well, e.g., more KIPs?
>> > >
>> > >
>> > > Thanks
>> > >
>> > > Eno
>> > >
>> > >
>> > >
>> > > On Thu, Aug 9, 2018 at 4:18 PM, xiongqi wu <xi...@gmail.com>
>> wrote:
>> > >
>> > > > Hi Kafka,
>> > > >
>> > > > This KIP tries to address GDPR concern to fulfill deletion request
>> on
>> > > time
>> > > > through time-based log compaction on a compaction enabled topic:
>> > > >
>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > > > 354%3A+Time-based+log+compaction+policy
>> > > >
>> > > > Any feedback will be appreciated.
>> > > >
>> > > >
>> > > > Xiongqi (Wesley) Wu
>> > > >
>> > >
>> >
>>
>
>
>
> --
> Xiongqi (Wesley) Wu
>



-- 
Xiongqi (Wesley) Wu

Re: [DISCUSS] KIP-354 Time-based log compaction policy

Posted by xiongqi wu <xi...@gmail.com>.
Dong,

Thanks for the comment.

There are two retention policy: log compaction and time based retention.

Log compaction:

we have use cases to keep infinite retention of a topic (only compaction).
GDPR cares about deletion of PII  (personal identifiable information) data.
Since Kafka doesn't know what records contain PII, it relies on upper layer
to delete those records.
For those infinite retention uses uses,  kafka needs to provide a way to
enforce compaction on time. This is what we try to address in this KIP.

Time based retention,

There are also use cases that users of Kafka might want to expire all their
data.
In those cases, they can use time based retention of their topics.


Regarding your first question,  if a user wants to delete a key in the log
compaction topic,  the user has to send a deletion using the same key.
Kafka only makes sure the deletion will happen under a certain time periods
(like 2 days/7 days).

Regarding your second question.  In most cases, we might want to delete all
duplicated keys at the same time.
Compaction might be more efficient since we need to scan the log and find
all duplicates.  However,  the expected use case is to set the time based
compaction interval on the order of days,  and be larger than 'min
compaction lag".  We don't want log compaction to happen frequently since
it is expensive.  The purpose is to help low production rate topic to get
compacted on time.  For the topic with "normal" incoming message message
rate, the "min dirty ratio" might have triggered the compaction before this
time based compaction policy takes effect.


Eno,

For your question,  like I mentioned we have long time retention use case
for log compacted topic, but we want to provide ability to delete certain
PII records on time.
Kafka itself doesn't know whether a record contains sensitive information
and relies on the user for deletion.


On Mon, Aug 13, 2018 at 6:58 PM, Dong Lin <li...@gmail.com> wrote:

> Hey Xiongqi,
>
> Thanks for the KIP. I have two questions regarding the use-case for meeting
> GDPR requirement.
>
> 1) If I recall correctly, one of the GDPR requirement is that we can not
> keep messages longer than e.g. 30 days in storage (e.g. Kafka). Say there
> exists a partition p0 which contains message1 with key1 and message2 with
> key2. And then user keeps producing messages with key=key2 to this
> partition. Since message1 with key1 is never overridden, sooner or later we
> will want to delete message1 and keep the latest message with key=key2. But
> currently it looks like log compact logic in Kafka will always put these
> messages in the same segment. Will this be an issue?
>
> 2) The current KIP intends to provide the capability to delete a given
> message in log compacted topic. Does such use-case also require Kafka to
> keep the messages produced before the given message? If yes, then we can
> probably just use AdminClient.deleteRecords() or time-based log retention
> to meet the use-case requirement. If no, do you know what is the GDPR's
> requirement on time-to-deletion after user explicitly requests the deletion
> (e.g. 1 hour, 1 day, 7 day)?
>
> Thanks,
> Dong
>
>
> On Mon, Aug 13, 2018 at 3:44 PM, xiongqi wu <xi...@gmail.com> wrote:
>
> > Hi Eno,
> >
> > The GDPR request we are getting here at linkedin is if we get a request
> to
> > delete a record through a null key on a log compacted topic,
> > we want to delete the record via compaction in a given time period like 2
> > days (whatever is required by the policy).
> >
> > There might be other issues (such as orphan log segments under certain
> > conditions)  that lead to GDPR problem but they are more like something
> we
> > need to fix anyway regardless of GDPR.
> >
> >
> > -- Xiongqi (Wesley) Wu
> >
> > On Mon, Aug 13, 2018 at 2:56 PM, Eno Thereska <en...@gmail.com>
> > wrote:
> >
> > > Hello,
> > >
> > > Thanks for the KIP. I'd like to see a more precise definition of what
> > part
> > > of GDPR you are targeting as well as some sort of verification that
> this
> > > KIP actually addresses the problem. Right now I find this a bit vague:
> > >
> > > "Ability to delete a log message through compaction in a timely manner
> > has
> > > become an important requirement in some use cases (e.g., GDPR)"
> > >
> > >
> > > Is there any guarantee that after this KIP the GDPR problem is solved
> or
> > do
> > > we need to do something else as well, e.g., more KIPs?
> > >
> > >
> > > Thanks
> > >
> > > Eno
> > >
> > >
> > >
> > > On Thu, Aug 9, 2018 at 4:18 PM, xiongqi wu <xi...@gmail.com>
> wrote:
> > >
> > > > Hi Kafka,
> > > >
> > > > This KIP tries to address GDPR concern to fulfill deletion request on
> > > time
> > > > through time-based log compaction on a compaction enabled topic:
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 354%3A+Time-based+log+compaction+policy
> > > >
> > > > Any feedback will be appreciated.
> > > >
> > > >
> > > > Xiongqi (Wesley) Wu
> > > >
> > >
> >
>



-- 
Xiongqi (Wesley) Wu

Re: [DISCUSS] KIP-354 Time-based log compaction policy

Posted by Dong Lin <li...@gmail.com>.
Hey Xiongqi,

Thanks for the KIP. I have two questions regarding the use-case for meeting
GDPR requirement.

1) If I recall correctly, one of the GDPR requirement is that we can not
keep messages longer than e.g. 30 days in storage (e.g. Kafka). Say there
exists a partition p0 which contains message1 with key1 and message2 with
key2. And then user keeps producing messages with key=key2 to this
partition. Since message1 with key1 is never overridden, sooner or later we
will want to delete message1 and keep the latest message with key=key2. But
currently it looks like log compact logic in Kafka will always put these
messages in the same segment. Will this be an issue?

2) The current KIP intends to provide the capability to delete a given
message in log compacted topic. Does such use-case also require Kafka to
keep the messages produced before the given message? If yes, then we can
probably just use AdminClient.deleteRecords() or time-based log retention
to meet the use-case requirement. If no, do you know what is the GDPR's
requirement on time-to-deletion after user explicitly requests the deletion
(e.g. 1 hour, 1 day, 7 day)?

Thanks,
Dong


On Mon, Aug 13, 2018 at 3:44 PM, xiongqi wu <xi...@gmail.com> wrote:

> Hi Eno,
>
> The GDPR request we are getting here at linkedin is if we get a request to
> delete a record through a null key on a log compacted topic,
> we want to delete the record via compaction in a given time period like 2
> days (whatever is required by the policy).
>
> There might be other issues (such as orphan log segments under certain
> conditions)  that lead to GDPR problem but they are more like something we
> need to fix anyway regardless of GDPR.
>
>
> -- Xiongqi (Wesley) Wu
>
> On Mon, Aug 13, 2018 at 2:56 PM, Eno Thereska <en...@gmail.com>
> wrote:
>
> > Hello,
> >
> > Thanks for the KIP. I'd like to see a more precise definition of what
> part
> > of GDPR you are targeting as well as some sort of verification that this
> > KIP actually addresses the problem. Right now I find this a bit vague:
> >
> > "Ability to delete a log message through compaction in a timely manner
> has
> > become an important requirement in some use cases (e.g., GDPR)"
> >
> >
> > Is there any guarantee that after this KIP the GDPR problem is solved or
> do
> > we need to do something else as well, e.g., more KIPs?
> >
> >
> > Thanks
> >
> > Eno
> >
> >
> >
> > On Thu, Aug 9, 2018 at 4:18 PM, xiongqi wu <xi...@gmail.com> wrote:
> >
> > > Hi Kafka,
> > >
> > > This KIP tries to address GDPR concern to fulfill deletion request on
> > time
> > > through time-based log compaction on a compaction enabled topic:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 354%3A+Time-based+log+compaction+policy
> > >
> > > Any feedback will be appreciated.
> > >
> > >
> > > Xiongqi (Wesley) Wu
> > >
> >
>

Re: [DISCUSS] KIP-354 Time-based log compaction policy

Posted by xiongqi wu <xi...@gmail.com>.
Hi Eno,

The GDPR request we are getting here at linkedin is if we get a request to
delete a record through a null key on a log compacted topic,
we want to delete the record via compaction in a given time period like 2
days (whatever is required by the policy).

There might be other issues (such as orphan log segments under certain
conditions)  that lead to GDPR problem but they are more like something we
need to fix anyway regardless of GDPR.


-- Xiongqi (Wesley) Wu

On Mon, Aug 13, 2018 at 2:56 PM, Eno Thereska <en...@gmail.com>
wrote:

> Hello,
>
> Thanks for the KIP. I'd like to see a more precise definition of what part
> of GDPR you are targeting as well as some sort of verification that this
> KIP actually addresses the problem. Right now I find this a bit vague:
>
> "Ability to delete a log message through compaction in a timely manner has
> become an important requirement in some use cases (e.g., GDPR)"
>
>
> Is there any guarantee that after this KIP the GDPR problem is solved or do
> we need to do something else as well, e.g., more KIPs?
>
>
> Thanks
>
> Eno
>
>
>
> On Thu, Aug 9, 2018 at 4:18 PM, xiongqi wu <xi...@gmail.com> wrote:
>
> > Hi Kafka,
> >
> > This KIP tries to address GDPR concern to fulfill deletion request on
> time
> > through time-based log compaction on a compaction enabled topic:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 354%3A+Time-based+log+compaction+policy
> >
> > Any feedback will be appreciated.
> >
> >
> > Xiongqi (Wesley) Wu
> >
>

Re: [DISCUSS] KIP-354 Time-based log compaction policy

Posted by Eno Thereska <en...@gmail.com>.
Hello,

Thanks for the KIP. I'd like to see a more precise definition of what part
of GDPR you are targeting as well as some sort of verification that this
KIP actually addresses the problem. Right now I find this a bit vague:

"Ability to delete a log message through compaction in a timely manner has
become an important requirement in some use cases (e.g., GDPR)"


Is there any guarantee that after this KIP the GDPR problem is solved or do
we need to do something else as well, e.g., more KIPs?


Thanks

Eno



On Thu, Aug 9, 2018 at 4:18 PM, xiongqi wu <xi...@gmail.com> wrote:

> Hi Kafka,
>
> This KIP tries to address GDPR concern to fulfill deletion request on time
> through time-based log compaction on a compaction enabled topic:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 354%3A+Time-based+log+compaction+policy
>
> Any feedback will be appreciated.
>
>
> Xiongqi (Wesley) Wu
>