You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Manikumar <ma...@gmail.com> on 2017/10/06 17:53:23 UTC
Re: [DISCUSS] KIP-186: Increase offsets retention default to 7 days
looks like VOTE thread is started for this KIP.
On Wed, Aug 16, 2017 at 5:39 PM, Stevo Slavić <ss...@gmail.com> wrote:
> +1 for making consistent default log and offsets retention time.
> I like Stephane's suggestion too, log retention override should override
> offset retention too if not explicitly configured.
>
> Please consider additionally:
> - introducing offsets.retention.hours config property
> - syncing log and offsets retention.check.interval.ms, if there's no real
> reason for the two to differ
> -- consider making retention check interval by default (if not explicitly
> configured) a fraction of retention time
> - name all "offsets" configs with "offsets" prefix (now it's a mix of
> singular/"offset" and plural/"offsets")
>
>
> On Fri, Aug 11, 2017 at 2:01 AM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > +1 from me
> >
> > On Wed, Aug 9, 2017 at 9:40 AM, Jason Gustafson <ja...@confluent.io>
> > wrote:
> >
> > > +1 on the bump to 7 days. Wanted to mention one minor point. The
> > > OffsetCommit RPC still provides the ability to set the retention time
> > from
> > > the client, but we do not use it in the consumer. Should we consider
> > adding
> > > a consumer config to set this? Given the problems people had with the
> old
> > > default, such a config would probably have gotten a fair bit of use.
> > Maybe
> > > it's less necessary with the new default, but there may be situations
> > where
> > > you don't want to keep the offsets for too long. For example, the
> console
> > > consumer commits offsets with a generated group id. We might want to
> set
> > a
> > > low retention time to keep it from filling the offset cache with
> garbage
> > > from such groups.
> > >
> >
> > I agree with Jason here, but maybe itself deserves a separate KIP
> > discussion.
> >
> >
> > >
> > > -Jason
> > >
> > > On Wed, Aug 9, 2017 at 5:24 AM, Sönke Liebau <
> > > soenke.liebau@opencore.com.invalid> wrote:
> > >
> > > > Just had this create issues at a customer as well, +1
> > > >
> > > > On Wed, Aug 9, 2017 at 11:46 AM, Mickael Maison <
> > > mickael.maison@gmail.com>
> > > > wrote:
> > > >
> > > > > Yes the current default is too short, +1
> > > > >
> > > > > On Wed, Aug 9, 2017 at 8:56 AM, Ismael Juma <is...@juma.me.uk>
> > wrote:
> > > > > > Thanks for the KIP, +1 from me.
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Wed, Aug 9, 2017 at 1:24 AM, Ewen Cheslack-Postava <
> > > > ewen@confluent.io
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Hi all,
> > > > > >>
> > > > > >> I posted a simple new KIP for a problem we see with a lot of
> > users:
> > > > > >> KIP-186: Increase offsets retention default to 7 days
> > > > > >>
> > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > >> 186%3A+Increase+offsets+retention+default+to+7+days
> > > > > >>
> > > > > >> Note that in addition to the KIP text itself, the linked JIRA
> > > already
> > > > > >> existed and has a bunch of discussion on the subject.
> > > > > >>
> > > > > >> -Ewen
> > > > > >>
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Sönke Liebau
> > > > Partner
> > > > Tel. +49 179 7940878
> > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
Re: [DISCUSS] KIP-186: Increase offsets retention default to 7 days
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
Reviving this thread.
@Stevo I think there are more refinements we can make, but I'd like to get
at least this pushed through because this keeps biting people unexpectedly.
How about we bump the default now just to address the immediate issue and
we can follow up with additional refinements in future KIPs?
It's also worth pointing out we also have
https://cwiki.apache.org/confluence/display/KAFKA/KIP-211%3A+Revise+Expiration+Semantics+of+Consumer+Group+Offsets.
That KIP dovetails nicely with this one -- this addresses the immediate
problem but can still expire offsets while the group is still active (which
is probably unintuitive). However, even if/when KIP-211 makes it in, this
will simply extend the period of the group being empty before removing
offsets to 7 days by default. This is still a significant improvement since
it helps with, e.g., cases where you have an outage and need to debug and
it takes longer than 24h. 1w is still a much better default even in the
case we wait for the group to be inactive before starting the expiration
timer.
I've also updated the KIP with the trivial PR:
https://github.com/apache/kafka/pull/4648
I'm going to kick off the vote thread so we can get this fixed for the next
version.
-Ewen
On Fri, Oct 6, 2017 at 11:00 AM, Manikumar <ma...@gmail.com>
wrote:
> looks like VOTE thread is *NOT* started for this KIP.
>
> On Fri, Oct 6, 2017 at 11:23 PM, Manikumar <ma...@gmail.com>
> wrote:
>
> > looks like VOTE thread is started for this KIP.
> >
> >
> > On Wed, Aug 16, 2017 at 5:39 PM, Stevo Slavić <ss...@gmail.com> wrote:
> >
> >> +1 for making consistent default log and offsets retention time.
> >> I like Stephane's suggestion too, log retention override should override
> >> offset retention too if not explicitly configured.
> >>
> >> Please consider additionally:
> >> - introducing offsets.retention.hours config property
> >> - syncing log and offsets retention.check.interval.ms, if there's no
> real
> >> reason for the two to differ
> >> -- consider making retention check interval by default (if not
> explicitly
> >> configured) a fraction of retention time
> >> - name all "offsets" configs with "offsets" prefix (now it's a mix of
> >> singular/"offset" and plural/"offsets")
> >>
> >>
> >> On Fri, Aug 11, 2017 at 2:01 AM, Guozhang Wang <wa...@gmail.com>
> >> wrote:
> >>
> >> > +1 from me
> >> >
> >> > On Wed, Aug 9, 2017 at 9:40 AM, Jason Gustafson <ja...@confluent.io>
> >> > wrote:
> >> >
> >> > > +1 on the bump to 7 days. Wanted to mention one minor point. The
> >> > > OffsetCommit RPC still provides the ability to set the retention
> time
> >> > from
> >> > > the client, but we do not use it in the consumer. Should we consider
> >> > adding
> >> > > a consumer config to set this? Given the problems people had with
> the
> >> old
> >> > > default, such a config would probably have gotten a fair bit of use.
> >> > Maybe
> >> > > it's less necessary with the new default, but there may be
> situations
> >> > where
> >> > > you don't want to keep the offsets for too long. For example, the
> >> console
> >> > > consumer commits offsets with a generated group id. We might want to
> >> set
> >> > a
> >> > > low retention time to keep it from filling the offset cache with
> >> garbage
> >> > > from such groups.
> >> > >
> >> >
> >> > I agree with Jason here, but maybe itself deserves a separate KIP
> >> > discussion.
> >> >
> >> >
> >> > >
> >> > > -Jason
> >> > >
> >> > > On Wed, Aug 9, 2017 at 5:24 AM, Sönke Liebau <
> >> > > soenke.liebau@opencore.com.invalid> wrote:
> >> > >
> >> > > > Just had this create issues at a customer as well, +1
> >> > > >
> >> > > > On Wed, Aug 9, 2017 at 11:46 AM, Mickael Maison <
> >> > > mickael.maison@gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > > > Yes the current default is too short, +1
> >> > > > >
> >> > > > > On Wed, Aug 9, 2017 at 8:56 AM, Ismael Juma <is...@juma.me.uk>
> >> > wrote:
> >> > > > > > Thanks for the KIP, +1 from me.
> >> > > > > >
> >> > > > > > Ismael
> >> > > > > >
> >> > > > > > On Wed, Aug 9, 2017 at 1:24 AM, Ewen Cheslack-Postava <
> >> > > > ewen@confluent.io
> >> > > > > >
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > >> Hi all,
> >> > > > > >>
> >> > > > > >> I posted a simple new KIP for a problem we see with a lot of
> >> > users:
> >> > > > > >> KIP-186: Increase offsets retention default to 7 days
> >> > > > > >>
> >> > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> > > > > >> 186%3A+Increase+offsets+retention+default+to+7+days
> >> > > > > >>
> >> > > > > >> Note that in addition to the KIP text itself, the linked JIRA
> >> > > already
> >> > > > > >> existed and has a bunch of discussion on the subject.
> >> > > > > >>
> >> > > > > >> -Ewen
> >> > > > > >>
> >> > > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > Sönke Liebau
> >> > > > Partner
> >> > > > Tel. +49 179 7940878
> >> > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel -
> >> Germany
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > -- Guozhang
> >> >
> >>
> >
> >
>
Re: [DISCUSS] KIP-186: Increase offsets retention default to 7 days
Posted by Manikumar <ma...@gmail.com>.
looks like VOTE thread is *NOT* started for this KIP.
On Fri, Oct 6, 2017 at 11:23 PM, Manikumar <ma...@gmail.com>
wrote:
> looks like VOTE thread is started for this KIP.
>
>
> On Wed, Aug 16, 2017 at 5:39 PM, Stevo Slavić <ss...@gmail.com> wrote:
>
>> +1 for making consistent default log and offsets retention time.
>> I like Stephane's suggestion too, log retention override should override
>> offset retention too if not explicitly configured.
>>
>> Please consider additionally:
>> - introducing offsets.retention.hours config property
>> - syncing log and offsets retention.check.interval.ms, if there's no real
>> reason for the two to differ
>> -- consider making retention check interval by default (if not explicitly
>> configured) a fraction of retention time
>> - name all "offsets" configs with "offsets" prefix (now it's a mix of
>> singular/"offset" and plural/"offsets")
>>
>>
>> On Fri, Aug 11, 2017 at 2:01 AM, Guozhang Wang <wa...@gmail.com>
>> wrote:
>>
>> > +1 from me
>> >
>> > On Wed, Aug 9, 2017 at 9:40 AM, Jason Gustafson <ja...@confluent.io>
>> > wrote:
>> >
>> > > +1 on the bump to 7 days. Wanted to mention one minor point. The
>> > > OffsetCommit RPC still provides the ability to set the retention time
>> > from
>> > > the client, but we do not use it in the consumer. Should we consider
>> > adding
>> > > a consumer config to set this? Given the problems people had with the
>> old
>> > > default, such a config would probably have gotten a fair bit of use.
>> > Maybe
>> > > it's less necessary with the new default, but there may be situations
>> > where
>> > > you don't want to keep the offsets for too long. For example, the
>> console
>> > > consumer commits offsets with a generated group id. We might want to
>> set
>> > a
>> > > low retention time to keep it from filling the offset cache with
>> garbage
>> > > from such groups.
>> > >
>> >
>> > I agree with Jason here, but maybe itself deserves a separate KIP
>> > discussion.
>> >
>> >
>> > >
>> > > -Jason
>> > >
>> > > On Wed, Aug 9, 2017 at 5:24 AM, Sönke Liebau <
>> > > soenke.liebau@opencore.com.invalid> wrote:
>> > >
>> > > > Just had this create issues at a customer as well, +1
>> > > >
>> > > > On Wed, Aug 9, 2017 at 11:46 AM, Mickael Maison <
>> > > mickael.maison@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Yes the current default is too short, +1
>> > > > >
>> > > > > On Wed, Aug 9, 2017 at 8:56 AM, Ismael Juma <is...@juma.me.uk>
>> > wrote:
>> > > > > > Thanks for the KIP, +1 from me.
>> > > > > >
>> > > > > > Ismael
>> > > > > >
>> > > > > > On Wed, Aug 9, 2017 at 1:24 AM, Ewen Cheslack-Postava <
>> > > > ewen@confluent.io
>> > > > > >
>> > > > > > wrote:
>> > > > > >
>> > > > > >> Hi all,
>> > > > > >>
>> > > > > >> I posted a simple new KIP for a problem we see with a lot of
>> > users:
>> > > > > >> KIP-186: Increase offsets retention default to 7 days
>> > > > > >>
>> > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > > > > >> 186%3A+Increase+offsets+retention+default+to+7+days
>> > > > > >>
>> > > > > >> Note that in addition to the KIP text itself, the linked JIRA
>> > > already
>> > > > > >> existed and has a bunch of discussion on the subject.
>> > > > > >>
>> > > > > >> -Ewen
>> > > > > >>
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Sönke Liebau
>> > > > Partner
>> > > > Tel. +49 179 7940878
>> > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel -
>> Germany
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > -- Guozhang
>> >
>>
>
>
Re: [DISCUSS] KIP-186: Increase offsets retention default to 7 days
Posted by Ted Yu <yu...@gmail.com>.
+1 on the KIP.
bq. introducing offsets.retention.hours config property
Probably the introduction would cause confusion among users due to the
existing minutes config.
On Fri, Oct 6, 2017 at 10:53 AM, Manikumar <ma...@gmail.com>
wrote:
> looks like VOTE thread is started for this KIP.
>
> On Wed, Aug 16, 2017 at 5:39 PM, Stevo Slavić <ss...@gmail.com> wrote:
>
> > +1 for making consistent default log and offsets retention time.
> > I like Stephane's suggestion too, log retention override should override
> > offset retention too if not explicitly configured.
> >
> > Please consider additionally:
> > - introducing offsets.retention.hours config property
> > - syncing log and offsets retention.check.interval.ms, if there's no
> real
> > reason for the two to differ
> > -- consider making retention check interval by default (if not explicitly
> > configured) a fraction of retention time
> > - name all "offsets" configs with "offsets" prefix (now it's a mix of
> > singular/"offset" and plural/"offsets")
> >
> >
> > On Fri, Aug 11, 2017 at 2:01 AM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >
> > > +1 from me
> > >
> > > On Wed, Aug 9, 2017 at 9:40 AM, Jason Gustafson <ja...@confluent.io>
> > > wrote:
> > >
> > > > +1 on the bump to 7 days. Wanted to mention one minor point. The
> > > > OffsetCommit RPC still provides the ability to set the retention time
> > > from
> > > > the client, but we do not use it in the consumer. Should we consider
> > > adding
> > > > a consumer config to set this? Given the problems people had with the
> > old
> > > > default, such a config would probably have gotten a fair bit of use.
> > > Maybe
> > > > it's less necessary with the new default, but there may be situations
> > > where
> > > > you don't want to keep the offsets for too long. For example, the
> > console
> > > > consumer commits offsets with a generated group id. We might want to
> > set
> > > a
> > > > low retention time to keep it from filling the offset cache with
> > garbage
> > > > from such groups.
> > > >
> > >
> > > I agree with Jason here, but maybe itself deserves a separate KIP
> > > discussion.
> > >
> > >
> > > >
> > > > -Jason
> > > >
> > > > On Wed, Aug 9, 2017 at 5:24 AM, Sönke Liebau <
> > > > soenke.liebau@opencore.com.invalid> wrote:
> > > >
> > > > > Just had this create issues at a customer as well, +1
> > > > >
> > > > > On Wed, Aug 9, 2017 at 11:46 AM, Mickael Maison <
> > > > mickael.maison@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Yes the current default is too short, +1
> > > > > >
> > > > > > On Wed, Aug 9, 2017 at 8:56 AM, Ismael Juma <is...@juma.me.uk>
> > > wrote:
> > > > > > > Thanks for the KIP, +1 from me.
> > > > > > >
> > > > > > > Ismael
> > > > > > >
> > > > > > > On Wed, Aug 9, 2017 at 1:24 AM, Ewen Cheslack-Postava <
> > > > > ewen@confluent.io
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi all,
> > > > > > >>
> > > > > > >> I posted a simple new KIP for a problem we see with a lot of
> > > users:
> > > > > > >> KIP-186: Increase offsets retention default to 7 days
> > > > > > >>
> > > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > >> 186%3A+Increase+offsets+retention+default+to+7+days
> > > > > > >>
> > > > > > >> Note that in addition to the KIP text itself, the linked JIRA
> > > > already
> > > > > > >> existed and has a bunch of discussion on the subject.
> > > > > > >>
> > > > > > >> -Ewen
> > > > > > >>
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Sönke Liebau
> > > > > Partner
> > > > > Tel. +49 179 7940878
> > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel -
> Germany
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>