You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@kafka.apache.org by Eric Wasserman <er...@gmail.com> on 2016/05/24 20:37:24 UTC

[VOTE] KIP-58 - Make Log Compaction Point Configurable

Hi,

I would like to begin voting on KIP-58 - Make Log Compaction Point
Configurable

KIP-58 is here:  <
https://cwiki.apache.org/confluence/display/KAFKA/KIP-58+-+Make+Log+Compaction+Point+Configurable
>

The Jira ticket KAFKA-1981 Make log compaction point configurable
is here: <https://issues.apache.org/jira/browse/KAFKA-1981>

The original pull request is here: <
https://github.com/apache/kafka/pull/1168>
(this includes configurations for size and message count lags that will be
removed per discussion of KIP-58).

The vote will run for 72 hours.

Re: [VOTE] KIP-58 - Make Log Compaction Point Configurable

Posted by Ismael Juma <is...@juma.me.uk>.

Sounds good.
On 25 May 2016 17:26, "Gwen Shapira" <gw...@confluent.io> wrote:

> All topic level names are inconsistent. We can have a separate discussion /
> KIP on getting out of that mess.
>
> Gwen
>
> On Wed, May 25, 2016 at 6:07 AM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > +1 (binding)
> >
> > I also think `log.cleaner.compaction.delay.ms` is clearer. As an aside,
> I
> > did notice that the topic level config for `log.segment.delete.delay.ms`
> > (mentioned by Ewen) is `file.delete.delay.ms`, which seems a bit
> > inconsistent.
> >
> > Ismael
> >
> > On Wed, May 25, 2016 at 4:43 AM, Ewen Cheslack-Postava <
> ewen@confluent.io>
> > wrote:
> >
> > > +1 (binding)
> > >
> > > Agreed that the log.cleaner.compaction.delay.ms is probably a better
> > name,
> > > and consistent with log.segment.delete.delay.ms. Checked configs for
> > other
> > > suffixes that seemed reasonable and despite only appearing in that one
> > > broker config, it seems the best match.
> > >
> > > -Ewen
> > >
> > > On Tue, May 24, 2016 at 8:16 PM, Jay Kreps <ja...@confluent.io> wrote:
> > >
> > > > I'm +1 on the concept.
> > > >
> > > > As with others I think the core challenge is to express this in an
> > > > intuitive way, and carry the same terminology across the docs, the
> > > configs,
> > > > and docstrings for the configs. Pictures would help.
> > > >
> > > > -Jay
> > > >
> > > > On Tue, May 24, 2016 at 6:54 PM, James Cheng <wu...@gmail.com>
> > > wrote:
> > > >
> > > > > I'm not sure what are the rules for who is allowed to vote, but
> I'm:
> > > > >
> > > > > +1 (non-binding) on the proposal
> > > > >
> > > > > I agree that the "log.cleaner.min.compaction.lag.ms" name is a
> > little
> > > > > confusing.
> > > > >
> > > > > I like Becket's "log.cleaner.compaction.delay.ms", or something
> > > similar.
> > > > >
> > > > > The KIP describes it as the portion of the topic "that will remain
> > > > > uncompacted", so if you're open to alternate names:
> > > > >
> > > > > "log.cleaner.uncompacted.range.ms"
> > > > > "log.cleaner.uncompacted.head.ms" (Except that I always get "log
> > tail"
> > > > > and "log head" mixed up...)
> > > > > "log.cleaner.uncompacted.retention.ms" (Will it be confusing to
> have
> > > the
> > > > > word "retention" in non-time-based topics?)
> > > > >
> > > > > I just thought of something: what happens to the value of "
> > > > > log.cleaner.delete.retention.ms"? Does it still have the same
> > meaning
> > > as
> > > > > before? Does the timer start when log compaction happens (as it
> > > currently
> > > > > does), so in reality, tombstones will only be removed from the log
> > some
> > > > > time after (log.cleaner.min.compaction.lag.ms +
> > > > > log.cleaner.delete.retention.ms)?
> > > > >
> > > > > -James
> > > > >
> > > > > > On May 24, 2016, at 5:46 PM, Becket Qin <be...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > +1 (non-binding) on the proposal. Just a minor suggestion.
> > > > > >
> > > > > > I am wondering should we change the config name to "
> > > > > > log.cleaner.compaction.delay.ms"? The first glance at the
> > > > configuration
> > > > > > name is a little confusing. I was thinking do we have a "max"
> lag?
> > > And
> > > > is
> > > > > > this "lag" a bad thing?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jiangjie (Becket) Qin
> > > > > >
> > > > > >
> > > > > > On Tue, May 24, 2016 at 4:21 PM, Gwen Shapira <gwen@confluent.io
> >
> > > > wrote:
> > > > > >
> > > > > >> +1 (binding)
> > > > > >>
> > > > > >> Thanks for responding to all my original concerns in the
> > discussion
> > > > > thread.
> > > > > >>
> > > > > >> On Tue, May 24, 2016 at 1:37 PM, Eric Wasserman <
> > > > > eric.wasserman@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Hi,
> > > > > >>>
> > > > > >>> I would like to begin voting on KIP-58 - Make Log Compaction
> > Point
> > > > > >>> Configurable
> > > > > >>>
> > > > > >>> KIP-58 is here:  <
> > > > > >>>
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-58+-+Make+Log+Compaction+Point+Configurable
> > > > > >>>>
> > > > > >>>
> > > > > >>> The Jira ticket KAFKA-1981 Make log compaction point
> configurable
> > > > > >>> is here: <https://issues.apache.org/jira/browse/KAFKA-1981>
> > > > > >>>
> > > > > >>> The original pull request is here: <
> > > > > >>> https://github.com/apache/kafka/pull/1168>
> > > > > >>> (this includes configurations for size and message count lags
> > that
> > > > will
> > > > > >> be
> > > > > >>> removed per discussion of KIP-58).
> > > > > >>>
> > > > > >>> The vote will run for 72 hours.
> > > > > >>>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks,
> > > Ewen
> > >
> >
>

Re: [VOTE] KIP-58 - Make Log Compaction Point Configurable

Posted by Joel Koshy <jj...@gmail.com>.

+1 on the proposal

Re: inconsistent names: KAFKA-3234
<https://issues.apache.org/jira/browse/KAFKA-3234> has a patch and
discussion in the PR that should help address the inconsistencies and
various other issues but we decided it would need a small KIP. (If someone
else wishes to take over that feel free to)

On Wed, May 25, 2016 at 9:26 AM, Gwen Shapira <gw...@confluent.io> wrote:

> All topic level names are inconsistent. We can have a separate discussion /
> KIP on getting out of that mess.
>
> Gwen
>
> On Wed, May 25, 2016 at 6:07 AM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > +1 (binding)
> >
> > I also think `log.cleaner.compaction.delay.ms` is clearer. As an aside,
> I
> > did notice that the topic level config for `log.segment.delete.delay.ms`
> > (mentioned by Ewen) is `file.delete.delay.ms`, which seems a bit
> > inconsistent.
> >
> > Ismael
> >
> > On Wed, May 25, 2016 at 4:43 AM, Ewen Cheslack-Postava <
> ewen@confluent.io>
> > wrote:
> >
> > > +1 (binding)
> > >
> > > Agreed that the log.cleaner.compaction.delay.ms is probably a better
> > name,
> > > and consistent with log.segment.delete.delay.ms. Checked configs for
> > other
> > > suffixes that seemed reasonable and despite only appearing in that one
> > > broker config, it seems the best match.
> > >
> > > -Ewen
> > >
> > > On Tue, May 24, 2016 at 8:16 PM, Jay Kreps <ja...@confluent.io> wrote:
> > >
> > > > I'm +1 on the concept.
> > > >
> > > > As with others I think the core challenge is to express this in an
> > > > intuitive way, and carry the same terminology across the docs, the
> > > configs,
> > > > and docstrings for the configs. Pictures would help.
> > > >
> > > > -Jay
> > > >
> > > > On Tue, May 24, 2016 at 6:54 PM, James Cheng <wu...@gmail.com>
> > > wrote:
> > > >
> > > > > I'm not sure what are the rules for who is allowed to vote, but
> I'm:
> > > > >
> > > > > +1 (non-binding) on the proposal
> > > > >
> > > > > I agree that the "log.cleaner.min.compaction.lag.ms" name is a
> > little
> > > > > confusing.
> > > > >
> > > > > I like Becket's "log.cleaner.compaction.delay.ms", or something
> > > similar.
> > > > >
> > > > > The KIP describes it as the portion of the topic "that will remain
> > > > > uncompacted", so if you're open to alternate names:
> > > > >
> > > > > "log.cleaner.uncompacted.range.ms"
> > > > > "log.cleaner.uncompacted.head.ms" (Except that I always get "log
> > tail"
> > > > > and "log head" mixed up...)
> > > > > "log.cleaner.uncompacted.retention.ms" (Will it be confusing to
> have
> > > the
> > > > > word "retention" in non-time-based topics?)
> > > > >
> > > > > I just thought of something: what happens to the value of "
> > > > > log.cleaner.delete.retention.ms"? Does it still have the same
> > meaning
> > > as
> > > > > before? Does the timer start when log compaction happens (as it
> > > currently
> > > > > does), so in reality, tombstones will only be removed from the log
> > some
> > > > > time after (log.cleaner.min.compaction.lag.ms +
> > > > > log.cleaner.delete.retention.ms)?
> > > > >
> > > > > -James
> > > > >
> > > > > > On May 24, 2016, at 5:46 PM, Becket Qin <be...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > +1 (non-binding) on the proposal. Just a minor suggestion.
> > > > > >
> > > > > > I am wondering should we change the config name to "
> > > > > > log.cleaner.compaction.delay.ms"? The first glance at the
> > > > configuration
> > > > > > name is a little confusing. I was thinking do we have a "max"
> lag?
> > > And
> > > > is
> > > > > > this "lag" a bad thing?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jiangjie (Becket) Qin
> > > > > >
> > > > > >
> > > > > > On Tue, May 24, 2016 at 4:21 PM, Gwen Shapira <gwen@confluent.io
> >
> > > > wrote:
> > > > > >
> > > > > >> +1 (binding)
> > > > > >>
> > > > > >> Thanks for responding to all my original concerns in the
> > discussion
> > > > > thread.
> > > > > >>
> > > > > >> On Tue, May 24, 2016 at 1:37 PM, Eric Wasserman <
> > > > > eric.wasserman@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Hi,
> > > > > >>>
> > > > > >>> I would like to begin voting on KIP-58 - Make Log Compaction
> > Point
> > > > > >>> Configurable
> > > > > >>>
> > > > > >>> KIP-58 is here:  <
> > > > > >>>
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-58+-+Make+Log+Compaction+Point+Configurable
> > > > > >>>>
> > > > > >>>
> > > > > >>> The Jira ticket KAFKA-1981 Make log compaction point
> configurable
> > > > > >>> is here: <https://issues.apache.org/jira/browse/KAFKA-1981>
> > > > > >>>
> > > > > >>> The original pull request is here: <
> > > > > >>> https://github.com/apache/kafka/pull/1168>
> > > > > >>> (this includes configurations for size and message count lags
> > that
> > > > will
> > > > > >> be
> > > > > >>> removed per discussion of KIP-58).
> > > > > >>>
> > > > > >>> The vote will run for 72 hours.
> > > > > >>>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks,
> > > Ewen
> > >
> >
>

Re: [VOTE] KIP-58 - Make Log Compaction Point Configurable

Posted by Gwen Shapira <gw...@confluent.io>.

All topic level names are inconsistent. We can have a separate discussion /
KIP on getting out of that mess.

Gwen

On Wed, May 25, 2016 at 6:07 AM, Ismael Juma <is...@juma.me.uk> wrote:

> +1 (binding)
>
> I also think `log.cleaner.compaction.delay.ms` is clearer. As an aside, I
> did notice that the topic level config for `log.segment.delete.delay.ms`
> (mentioned by Ewen) is `file.delete.delay.ms`, which seems a bit
> inconsistent.
>
> Ismael
>
> On Wed, May 25, 2016 at 4:43 AM, Ewen Cheslack-Postava <ew...@confluent.io>
> wrote:
>
> > +1 (binding)
> >
> > Agreed that the log.cleaner.compaction.delay.ms is probably a better
> name,
> > and consistent with log.segment.delete.delay.ms. Checked configs for
> other
> > suffixes that seemed reasonable and despite only appearing in that one
> > broker config, it seems the best match.
> >
> > -Ewen
> >
> > On Tue, May 24, 2016 at 8:16 PM, Jay Kreps <ja...@confluent.io> wrote:
> >
> > > I'm +1 on the concept.
> > >
> > > As with others I think the core challenge is to express this in an
> > > intuitive way, and carry the same terminology across the docs, the
> > configs,
> > > and docstrings for the configs. Pictures would help.
> > >
> > > -Jay
> > >
> > > On Tue, May 24, 2016 at 6:54 PM, James Cheng <wu...@gmail.com>
> > wrote:
> > >
> > > > I'm not sure what are the rules for who is allowed to vote, but I'm:
> > > >
> > > > +1 (non-binding) on the proposal
> > > >
> > > > I agree that the "log.cleaner.min.compaction.lag.ms" name is a
> little
> > > > confusing.
> > > >
> > > > I like Becket's "log.cleaner.compaction.delay.ms", or something
> > similar.
> > > >
> > > > The KIP describes it as the portion of the topic "that will remain
> > > > uncompacted", so if you're open to alternate names:
> > > >
> > > > "log.cleaner.uncompacted.range.ms"
> > > > "log.cleaner.uncompacted.head.ms" (Except that I always get "log
> tail"
> > > > and "log head" mixed up...)
> > > > "log.cleaner.uncompacted.retention.ms" (Will it be confusing to have
> > the
> > > > word "retention" in non-time-based topics?)
> > > >
> > > > I just thought of something: what happens to the value of "
> > > > log.cleaner.delete.retention.ms"? Does it still have the same
> meaning
> > as
> > > > before? Does the timer start when log compaction happens (as it
> > currently
> > > > does), so in reality, tombstones will only be removed from the log
> some
> > > > time after (log.cleaner.min.compaction.lag.ms +
> > > > log.cleaner.delete.retention.ms)?
> > > >
> > > > -James
> > > >
> > > > > On May 24, 2016, at 5:46 PM, Becket Qin <be...@gmail.com>
> > wrote:
> > > > >
> > > > > +1 (non-binding) on the proposal. Just a minor suggestion.
> > > > >
> > > > > I am wondering should we change the config name to "
> > > > > log.cleaner.compaction.delay.ms"? The first glance at the
> > > configuration
> > > > > name is a little confusing. I was thinking do we have a "max" lag?
> > And
> > > is
> > > > > this "lag" a bad thing?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jiangjie (Becket) Qin
> > > > >
> > > > >
> > > > > On Tue, May 24, 2016 at 4:21 PM, Gwen Shapira <gw...@confluent.io>
> > > wrote:
> > > > >
> > > > >> +1 (binding)
> > > > >>
> > > > >> Thanks for responding to all my original concerns in the
> discussion
> > > > thread.
> > > > >>
> > > > >> On Tue, May 24, 2016 at 1:37 PM, Eric Wasserman <
> > > > eric.wasserman@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >>> Hi,
> > > > >>>
> > > > >>> I would like to begin voting on KIP-58 - Make Log Compaction
> Point
> > > > >>> Configurable
> > > > >>>
> > > > >>> KIP-58 is here:  <
> > > > >>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-58+-+Make+Log+Compaction+Point+Configurable
> > > > >>>>
> > > > >>>
> > > > >>> The Jira ticket KAFKA-1981 Make log compaction point configurable
> > > > >>> is here: <https://issues.apache.org/jira/browse/KAFKA-1981>
> > > > >>>
> > > > >>> The original pull request is here: <
> > > > >>> https://github.com/apache/kafka/pull/1168>
> > > > >>> (this includes configurations for size and message count lags
> that
> > > will
> > > > >> be
> > > > >>> removed per discussion of KIP-58).
> > > > >>>
> > > > >>> The vote will run for 72 hours.
> > > > >>>
> > > > >>
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > Thanks,
> > Ewen
> >
>

Re: [VOTE] KIP-58 - Make Log Compaction Point Configurable

Posted by Grant Henke <gh...@cloudera.com>.

+1 (non-binding)

On Wed, May 25, 2016 at 8:20 AM, Ben Stopford <be...@confluent.io> wrote:

> +1 (non-binding)
>
> > On 25 May 2016, at 14:07, Ismael Juma <is...@juma.me.uk> wrote:
> >
> > +1 (binding)
> >
> > I also think `log.cleaner.compaction.delay.ms` is clearer. As an aside,
> I
> > did notice that the topic level config for `log.segment.delete.delay.ms`
> > (mentioned by Ewen) is `file.delete.delay.ms`, which seems a bit
> > inconsistent.
> >
> > Ismael
> >
> > On Wed, May 25, 2016 at 4:43 AM, Ewen Cheslack-Postava <
> ewen@confluent.io>
> > wrote:
> >
> >> +1 (binding)
> >>
> >> Agreed that the log.cleaner.compaction.delay.ms is probably a better
> name,
> >> and consistent with log.segment.delete.delay.ms. Checked configs for
> other
> >> suffixes that seemed reasonable and despite only appearing in that one
> >> broker config, it seems the best match.
> >>
> >> -Ewen
> >>
> >> On Tue, May 24, 2016 at 8:16 PM, Jay Kreps <ja...@confluent.io> wrote:
> >>
> >>> I'm +1 on the concept.
> >>>
> >>> As with others I think the core challenge is to express this in an
> >>> intuitive way, and carry the same terminology across the docs, the
> >> configs,
> >>> and docstrings for the configs. Pictures would help.
> >>>
> >>> -Jay
> >>>
> >>> On Tue, May 24, 2016 at 6:54 PM, James Cheng <wu...@gmail.com>
> >> wrote:
> >>>
> >>>> I'm not sure what are the rules for who is allowed to vote, but I'm:
> >>>>
> >>>> +1 (non-binding) on the proposal
> >>>>
> >>>> I agree that the "log.cleaner.min.compaction.lag.ms" name is a little
> >>>> confusing.
> >>>>
> >>>> I like Becket's "log.cleaner.compaction.delay.ms", or something
> >> similar.
> >>>>
> >>>> The KIP describes it as the portion of the topic "that will remain
> >>>> uncompacted", so if you're open to alternate names:
> >>>>
> >>>> "log.cleaner.uncompacted.range.ms"
> >>>> "log.cleaner.uncompacted.head.ms" (Except that I always get "log
> tail"
> >>>> and "log head" mixed up...)
> >>>> "log.cleaner.uncompacted.retention.ms" (Will it be confusing to have
> >> the
> >>>> word "retention" in non-time-based topics?)
> >>>>
> >>>> I just thought of something: what happens to the value of "
> >>>> log.cleaner.delete.retention.ms"? Does it still have the same meaning
> >> as
> >>>> before? Does the timer start when log compaction happens (as it
> >> currently
> >>>> does), so in reality, tombstones will only be removed from the log
> some
> >>>> time after (log.cleaner.min.compaction.lag.ms +
> >>>> log.cleaner.delete.retention.ms)?
> >>>>
> >>>> -James
> >>>>
> >>>>> On May 24, 2016, at 5:46 PM, Becket Qin <be...@gmail.com>
> >> wrote:
> >>>>>
> >>>>> +1 (non-binding) on the proposal. Just a minor suggestion.
> >>>>>
> >>>>> I am wondering should we change the config name to "
> >>>>> log.cleaner.compaction.delay.ms"? The first glance at the
> >>> configuration
> >>>>> name is a little confusing. I was thinking do we have a "max" lag?
> >> And
> >>> is
> >>>>> this "lag" a bad thing?
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Jiangjie (Becket) Qin
> >>>>>
> >>>>>
> >>>>> On Tue, May 24, 2016 at 4:21 PM, Gwen Shapira <gw...@confluent.io>
> >>> wrote:
> >>>>>
> >>>>>> +1 (binding)
> >>>>>>
> >>>>>> Thanks for responding to all my original concerns in the discussion
> >>>> thread.
> >>>>>>
> >>>>>> On Tue, May 24, 2016 at 1:37 PM, Eric Wasserman <
> >>>> eric.wasserman@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> I would like to begin voting on KIP-58 - Make Log Compaction Point
> >>>>>>> Configurable
> >>>>>>>
> >>>>>>> KIP-58 is here:  <
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-58+-+Make+Log+Compaction+Point+Configurable
> >>>>>>>>
> >>>>>>>
> >>>>>>> The Jira ticket KAFKA-1981 Make log compaction point configurable
> >>>>>>> is here: <https://issues.apache.org/jira/browse/KAFKA-1981>
> >>>>>>>
> >>>>>>> The original pull request is here: <
> >>>>>>> https://github.com/apache/kafka/pull/1168>
> >>>>>>> (this includes configurations for size and message count lags that
> >>> will
> >>>>>> be
> >>>>>>> removed per discussion of KIP-58).
> >>>>>>>
> >>>>>>> The vote will run for 72 hours.
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >>
> >> --
> >> Thanks,
> >> Ewen
> >>
>
>


-- 
Grant Henke
Software Engineer | Cloudera
grant@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke

Re: [VOTE] KIP-58 - Make Log Compaction Point Configurable

Posted by Ben Stopford <be...@confluent.io>.

+1 (non-binding)

> On 25 May 2016, at 14:07, Ismael Juma <is...@juma.me.uk> wrote:
> 
> +1 (binding)
> 
> I also think `log.cleaner.compaction.delay.ms` is clearer. As an aside, I
> did notice that the topic level config for `log.segment.delete.delay.ms`
> (mentioned by Ewen) is `file.delete.delay.ms`, which seems a bit
> inconsistent.
> 
> Ismael
> 
> On Wed, May 25, 2016 at 4:43 AM, Ewen Cheslack-Postava <ew...@confluent.io>
> wrote:
> 
>> +1 (binding)
>> 
>> Agreed that the log.cleaner.compaction.delay.ms is probably a better name,
>> and consistent with log.segment.delete.delay.ms. Checked configs for other
>> suffixes that seemed reasonable and despite only appearing in that one
>> broker config, it seems the best match.
>> 
>> -Ewen
>> 
>> On Tue, May 24, 2016 at 8:16 PM, Jay Kreps <ja...@confluent.io> wrote:
>> 
>>> I'm +1 on the concept.
>>> 
>>> As with others I think the core challenge is to express this in an
>>> intuitive way, and carry the same terminology across the docs, the
>> configs,
>>> and docstrings for the configs. Pictures would help.
>>> 
>>> -Jay
>>> 
>>> On Tue, May 24, 2016 at 6:54 PM, James Cheng <wu...@gmail.com>
>> wrote:
>>> 
>>>> I'm not sure what are the rules for who is allowed to vote, but I'm:
>>>> 
>>>> +1 (non-binding) on the proposal
>>>> 
>>>> I agree that the "log.cleaner.min.compaction.lag.ms" name is a little
>>>> confusing.
>>>> 
>>>> I like Becket's "log.cleaner.compaction.delay.ms", or something
>> similar.
>>>> 
>>>> The KIP describes it as the portion of the topic "that will remain
>>>> uncompacted", so if you're open to alternate names:
>>>> 
>>>> "log.cleaner.uncompacted.range.ms"
>>>> "log.cleaner.uncompacted.head.ms" (Except that I always get "log tail"
>>>> and "log head" mixed up...)
>>>> "log.cleaner.uncompacted.retention.ms" (Will it be confusing to have
>> the
>>>> word "retention" in non-time-based topics?)
>>>> 
>>>> I just thought of something: what happens to the value of "
>>>> log.cleaner.delete.retention.ms"? Does it still have the same meaning
>> as
>>>> before? Does the timer start when log compaction happens (as it
>> currently
>>>> does), so in reality, tombstones will only be removed from the log some
>>>> time after (log.cleaner.min.compaction.lag.ms +
>>>> log.cleaner.delete.retention.ms)?
>>>> 
>>>> -James
>>>> 
>>>>> On May 24, 2016, at 5:46 PM, Becket Qin <be...@gmail.com>
>> wrote:
>>>>> 
>>>>> +1 (non-binding) on the proposal. Just a minor suggestion.
>>>>> 
>>>>> I am wondering should we change the config name to "
>>>>> log.cleaner.compaction.delay.ms"? The first glance at the
>>> configuration
>>>>> name is a little confusing. I was thinking do we have a "max" lag?
>> And
>>> is
>>>>> this "lag" a bad thing?
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Jiangjie (Becket) Qin
>>>>> 
>>>>> 
>>>>> On Tue, May 24, 2016 at 4:21 PM, Gwen Shapira <gw...@confluent.io>
>>> wrote:
>>>>> 
>>>>>> +1 (binding)
>>>>>> 
>>>>>> Thanks for responding to all my original concerns in the discussion
>>>> thread.
>>>>>> 
>>>>>> On Tue, May 24, 2016 at 1:37 PM, Eric Wasserman <
>>>> eric.wasserman@gmail.com>
>>>>>> wrote:
>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> I would like to begin voting on KIP-58 - Make Log Compaction Point
>>>>>>> Configurable
>>>>>>> 
>>>>>>> KIP-58 is here:  <
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-58+-+Make+Log+Compaction+Point+Configurable
>>>>>>>> 
>>>>>>> 
>>>>>>> The Jira ticket KAFKA-1981 Make log compaction point configurable
>>>>>>> is here: <https://issues.apache.org/jira/browse/KAFKA-1981>
>>>>>>> 
>>>>>>> The original pull request is here: <
>>>>>>> https://github.com/apache/kafka/pull/1168>
>>>>>>> (this includes configurations for size and message count lags that
>>> will
>>>>>> be
>>>>>>> removed per discussion of KIP-58).
>>>>>>> 
>>>>>>> The vote will run for 72 hours.
>>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>> 
>> 
>> 
>> 
>> --
>> Thanks,
>> Ewen
>>

Re: [VOTE] KIP-58 - Make Log Compaction Point Configurable

Posted by Ismael Juma <is...@juma.me.uk>.

+1 (binding)

I also think `log.cleaner.compaction.delay.ms` is clearer. As an aside, I
did notice that the topic level config for `log.segment.delete.delay.ms`
(mentioned by Ewen) is `file.delete.delay.ms`, which seems a bit
inconsistent.

Ismael

On Wed, May 25, 2016 at 4:43 AM, Ewen Cheslack-Postava <ew...@confluent.io>
wrote:

> +1 (binding)
>
> Agreed that the log.cleaner.compaction.delay.ms is probably a better name,
> and consistent with log.segment.delete.delay.ms. Checked configs for other
> suffixes that seemed reasonable and despite only appearing in that one
> broker config, it seems the best match.
>
> -Ewen
>
> On Tue, May 24, 2016 at 8:16 PM, Jay Kreps <ja...@confluent.io> wrote:
>
> > I'm +1 on the concept.
> >
> > As with others I think the core challenge is to express this in an
> > intuitive way, and carry the same terminology across the docs, the
> configs,
> > and docstrings for the configs. Pictures would help.
> >
> > -Jay
> >
> > On Tue, May 24, 2016 at 6:54 PM, James Cheng <wu...@gmail.com>
> wrote:
> >
> > > I'm not sure what are the rules for who is allowed to vote, but I'm:
> > >
> > > +1 (non-binding) on the proposal
> > >
> > > I agree that the "log.cleaner.min.compaction.lag.ms" name is a little
> > > confusing.
> > >
> > > I like Becket's "log.cleaner.compaction.delay.ms", or something
> similar.
> > >
> > > The KIP describes it as the portion of the topic "that will remain
> > > uncompacted", so if you're open to alternate names:
> > >
> > > "log.cleaner.uncompacted.range.ms"
> > > "log.cleaner.uncompacted.head.ms" (Except that I always get "log tail"
> > > and "log head" mixed up...)
> > > "log.cleaner.uncompacted.retention.ms" (Will it be confusing to have
> the
> > > word "retention" in non-time-based topics?)
> > >
> > > I just thought of something: what happens to the value of "
> > > log.cleaner.delete.retention.ms"? Does it still have the same meaning
> as
> > > before? Does the timer start when log compaction happens (as it
> currently
> > > does), so in reality, tombstones will only be removed from the log some
> > > time after (log.cleaner.min.compaction.lag.ms +
> > > log.cleaner.delete.retention.ms)?
> > >
> > > -James
> > >
> > > > On May 24, 2016, at 5:46 PM, Becket Qin <be...@gmail.com>
> wrote:
> > > >
> > > > +1 (non-binding) on the proposal. Just a minor suggestion.
> > > >
> > > > I am wondering should we change the config name to "
> > > > log.cleaner.compaction.delay.ms"? The first glance at the
> > configuration
> > > > name is a little confusing. I was thinking do we have a "max" lag?
> And
> > is
> > > > this "lag" a bad thing?
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > >
> > > > On Tue, May 24, 2016 at 4:21 PM, Gwen Shapira <gw...@confluent.io>
> > wrote:
> > > >
> > > >> +1 (binding)
> > > >>
> > > >> Thanks for responding to all my original concerns in the discussion
> > > thread.
> > > >>
> > > >> On Tue, May 24, 2016 at 1:37 PM, Eric Wasserman <
> > > eric.wasserman@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> Hi,
> > > >>>
> > > >>> I would like to begin voting on KIP-58 - Make Log Compaction Point
> > > >>> Configurable
> > > >>>
> > > >>> KIP-58 is here:  <
> > > >>>
> > > >>>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-58+-+Make+Log+Compaction+Point+Configurable
> > > >>>>
> > > >>>
> > > >>> The Jira ticket KAFKA-1981 Make log compaction point configurable
> > > >>> is here: <https://issues.apache.org/jira/browse/KAFKA-1981>
> > > >>>
> > > >>> The original pull request is here: <
> > > >>> https://github.com/apache/kafka/pull/1168>
> > > >>> (this includes configurations for size and message count lags that
> > will
> > > >> be
> > > >>> removed per discussion of KIP-58).
> > > >>>
> > > >>> The vote will run for 72 hours.
> > > >>>
> > > >>
> > >
> > >
> >
>
>
>
> --
> Thanks,
> Ewen
>

Re: [VOTE] KIP-58 - Make Log Compaction Point Configurable

Posted by Manikumar Reddy <ma...@gmail.com>.

+1 (non binding)

On Wed, May 25, 2016 at 4:03 PM, Tom Crayford <tc...@heroku.com> wrote:

> +1 (non binding)
>
> Agree on log.cleaner.compaction.delay.ms being the better name.
>
> I think this setting is going to be extremely hard to tune for users, and
> worry about adding yet more configuration - Kafka already has a huge number
> of tunables though, so we're in well trod ground with "just add more
> tuning". I can't however, come up with any better mechanism (that doesn't
> require tuning at all) without gross interactions with consumer offset
> storage, so I remain a +1 here.
>
> Thanks
>
> Tom Crayford
> Heroku Kafka
>
> On Wednesday, 25 May 2016, Ewen Cheslack-Postava <ewen@confluent.io
> <javascript:_e(%7B%7D,'cvml','ewen@confluent.io');>> wrote:
>
> > +1 (binding)
> >
> > Agreed that the log.cleaner.compaction.delay.ms is probably a better
> name,
> > and consistent with log.segment.delete.delay.ms. Checked configs for
> other
> > suffixes that seemed reasonable and despite only appearing in that one
> > broker config, it seems the best match.
> >
> > -Ewen
> >
> > On Tue, May 24, 2016 at 8:16 PM, Jay Kreps <ja...@confluent.io> wrote:
> >
> > > I'm +1 on the concept.
> > >
> > > As with others I think the core challenge is to express this in an
> > > intuitive way, and carry the same terminology across the docs, the
> > configs,
> > > and docstrings for the configs. Pictures would help.
> > >
> > > -Jay
> > >
> > > On Tue, May 24, 2016 at 6:54 PM, James Cheng <wu...@gmail.com>
> > wrote:
> > >
> > > > I'm not sure what are the rules for who is allowed to vote, but I'm:
> > > >
> > > > +1 (non-binding) on the proposal
> > > >
> > > > I agree that the "log.cleaner.min.compaction.lag.ms" name is a
> little
> > > > confusing.
> > > >
> > > > I like Becket's "log.cleaner.compaction.delay.ms", or something
> > similar.
> > > >
> > > > The KIP describes it as the portion of the topic "that will remain
> > > > uncompacted", so if you're open to alternate names:
> > > >
> > > > "log.cleaner.uncompacted.range.ms"
> > > > "log.cleaner.uncompacted.head.ms" (Except that I always get "log
> tail"
> > > > and "log head" mixed up...)
> > > > "log.cleaner.uncompacted.retention.ms" (Will it be confusing to have
> > the
> > > > word "retention" in non-time-based topics?)
> > > >
> > > > I just thought of something: what happens to the value of "
> > > > log.cleaner.delete.retention.ms"? Does it still have the same
> meaning
> > as
> > > > before? Does the timer start when log compaction happens (as it
> > currently
> > > > does), so in reality, tombstones will only be removed from the log
> some
> > > > time after (log.cleaner.min.compaction.lag.ms +
> > > > log.cleaner.delete.retention.ms)?
> > > >
> > > > -James
> > > >
> > > > > On May 24, 2016, at 5:46 PM, Becket Qin <be...@gmail.com>
> > wrote:
> > > > >
> > > > > +1 (non-binding) on the proposal. Just a minor suggestion.
> > > > >
> > > > > I am wondering should we change the config name to "
> > > > > log.cleaner.compaction.delay.ms"? The first glance at the
> > > configuration
> > > > > name is a little confusing. I was thinking do we have a "max" lag?
> > And
> > > is
> > > > > this "lag" a bad thing?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jiangjie (Becket) Qin
> > > > >
> > > > >
> > > > > On Tue, May 24, 2016 at 4:21 PM, Gwen Shapira <gw...@confluent.io>
> > > wrote:
> > > > >
> > > > >> +1 (binding)
> > > > >>
> > > > >> Thanks for responding to all my original concerns in the
> discussion
> > > > thread.
> > > > >>
> > > > >> On Tue, May 24, 2016 at 1:37 PM, Eric Wasserman <
> > > > eric.wasserman@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >>> Hi,
> > > > >>>
> > > > >>> I would like to begin voting on KIP-58 - Make Log Compaction
> Point
> > > > >>> Configurable
> > > > >>>
> > > > >>> KIP-58 is here:  <
> > > > >>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-58+-+Make+Log+Compaction+Point+Configurable
> > > > >>>>
> > > > >>>
> > > > >>> The Jira ticket KAFKA-1981 Make log compaction point configurable
> > > > >>> is here: <https://issues.apache.org/jira/browse/KAFKA-1981>
> > > > >>>
> > > > >>> The original pull request is here: <
> > > > >>> https://github.com/apache/kafka/pull/1168>
> > > > >>> (this includes configurations for size and message count lags
> that
> > > will
> > > > >> be
> > > > >>> removed per discussion of KIP-58).
> > > > >>>
> > > > >>> The vote will run for 72 hours.
> > > > >>>
> > > > >>
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > Thanks,
> > Ewen
> >
>