You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Ewen Cheslack-Postava <ew...@confluent.io> on 2017/08/09 00:24:58 UTC
[DISCUSS] KIP-186: Increase offsets retention default to 7 days
Hi all,
I posted a simple new KIP for a problem we see with a lot of users:
KIP-186: Increase offsets retention default to 7 days
https://cwiki.apache.org/confluence/display/KAFKA/KIP-186%3A+Increase+offsets+retention+default+to+7+days
Note that in addition to the KIP text itself, the linked JIRA already
existed and has a bunch of discussion on the subject.
-Ewen
Re: [DISCUSS] KIP-186: Increase offsets retention default to 7 days
Posted by Vahid S Hashemian <va...@us.ibm.com>.
+1 on both the KIP and Jason's suggestion of adding the consumer config.
Thanks.
--Vahid
From: Jason Gustafson <ja...@confluent.io>
To: dev@kafka.apache.org
Date: 08/09/2017 09:40 AM
Subject: Re: [DISCUSS] KIP-186: Increase offsets retention default
to 7 days
+1 on the bump to 7 days. Wanted to mention one minor point. The
OffsetCommit RPC still provides the ability to set the retention time from
the client, but we do not use it in the consumer. Should we consider
adding
a consumer config to set this? Given the problems people had with the old
default, such a config would probably have gotten a fair bit of use. Maybe
it's less necessary with the new default, but there may be situations
where
you don't want to keep the offsets for too long. For example, the console
consumer commits offsets with a generated group id. We might want to set a
low retention time to keep it from filling the offset cache with garbage
from such groups.
-Jason
On Wed, Aug 9, 2017 at 5:24 AM, Sönke Liebau <
soenke.liebau@opencore.com.invalid> wrote:
> Just had this create issues at a customer as well, +1
>
> On Wed, Aug 9, 2017 at 11:46 AM, Mickael Maison
<mi...@gmail.com>
> wrote:
>
> > Yes the current default is too short, +1
> >
> > On Wed, Aug 9, 2017 at 8:56 AM, Ismael Juma <is...@juma.me.uk> wrote:
> > > Thanks for the KIP, +1 from me.
> > >
> > > Ismael
> > >
> > > On Wed, Aug 9, 2017 at 1:24 AM, Ewen Cheslack-Postava <
> ewen@confluent.io
> > >
> > > wrote:
> > >
> > >> Hi all,
> > >>
> > >> I posted a simple new KIP for a problem we see with a lot of users:
> > >> KIP-186: Increase offsets retention default to 7 days
> > >>
> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >> 186%3A+Increase+offsets+retention+default+to+7+days
> > >>
> > >> Note that in addition to the KIP text itself, the linked JIRA
already
> > >> existed and has a bunch of discussion on the subject.
> > >>
> > >> -Ewen
> > >>
> >
>
>
>
> --
> Sönke Liebau
> Partner
> Tel. +49 179 7940878
> OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
>
Re: [DISCUSS] KIP-186: Increase offsets retention default to 7 days
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
Reviving this thread.
@Stevo I think there are more refinements we can make, but I'd like to get
at least this pushed through because this keeps biting people unexpectedly.
How about we bump the default now just to address the immediate issue and
we can follow up with additional refinements in future KIPs?
It's also worth pointing out we also have
https://cwiki.apache.org/confluence/display/KAFKA/KIP-211%3A+Revise+Expiration+Semantics+of+Consumer+Group+Offsets.
That KIP dovetails nicely with this one -- this addresses the immediate
problem but can still expire offsets while the group is still active (which
is probably unintuitive). However, even if/when KIP-211 makes it in, this
will simply extend the period of the group being empty before removing
offsets to 7 days by default. This is still a significant improvement since
it helps with, e.g., cases where you have an outage and need to debug and
it takes longer than 24h. 1w is still a much better default even in the
case we wait for the group to be inactive before starting the expiration
timer.
I've also updated the KIP with the trivial PR:
https://github.com/apache/kafka/pull/4648
I'm going to kick off the vote thread so we can get this fixed for the next
version.
-Ewen
On Fri, Oct 6, 2017 at 11:00 AM, Manikumar <ma...@gmail.com>
wrote:
> looks like VOTE thread is *NOT* started for this KIP.
>
> On Fri, Oct 6, 2017 at 11:23 PM, Manikumar <ma...@gmail.com>
> wrote:
>
> > looks like VOTE thread is started for this KIP.
> >
> >
> > On Wed, Aug 16, 2017 at 5:39 PM, Stevo Slavić <ss...@gmail.com> wrote:
> >
> >> +1 for making consistent default log and offsets retention time.
> >> I like Stephane's suggestion too, log retention override should override
> >> offset retention too if not explicitly configured.
> >>
> >> Please consider additionally:
> >> - introducing offsets.retention.hours config property
> >> - syncing log and offsets retention.check.interval.ms, if there's no
> real
> >> reason for the two to differ
> >> -- consider making retention check interval by default (if not
> explicitly
> >> configured) a fraction of retention time
> >> - name all "offsets" configs with "offsets" prefix (now it's a mix of
> >> singular/"offset" and plural/"offsets")
> >>
> >>
> >> On Fri, Aug 11, 2017 at 2:01 AM, Guozhang Wang <wa...@gmail.com>
> >> wrote:
> >>
> >> > +1 from me
> >> >
> >> > On Wed, Aug 9, 2017 at 9:40 AM, Jason Gustafson <ja...@confluent.io>
> >> > wrote:
> >> >
> >> > > +1 on the bump to 7 days. Wanted to mention one minor point. The
> >> > > OffsetCommit RPC still provides the ability to set the retention
> time
> >> > from
> >> > > the client, but we do not use it in the consumer. Should we consider
> >> > adding
> >> > > a consumer config to set this? Given the problems people had with
> the
> >> old
> >> > > default, such a config would probably have gotten a fair bit of use.
> >> > Maybe
> >> > > it's less necessary with the new default, but there may be
> situations
> >> > where
> >> > > you don't want to keep the offsets for too long. For example, the
> >> console
> >> > > consumer commits offsets with a generated group id. We might want to
> >> set
> >> > a
> >> > > low retention time to keep it from filling the offset cache with
> >> garbage
> >> > > from such groups.
> >> > >
> >> >
> >> > I agree with Jason here, but maybe itself deserves a separate KIP
> >> > discussion.
> >> >
> >> >
> >> > >
> >> > > -Jason
> >> > >
> >> > > On Wed, Aug 9, 2017 at 5:24 AM, Sönke Liebau <
> >> > > soenke.liebau@opencore.com.invalid> wrote:
> >> > >
> >> > > > Just had this create issues at a customer as well, +1
> >> > > >
> >> > > > On Wed, Aug 9, 2017 at 11:46 AM, Mickael Maison <
> >> > > mickael.maison@gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > > > Yes the current default is too short, +1
> >> > > > >
> >> > > > > On Wed, Aug 9, 2017 at 8:56 AM, Ismael Juma <is...@juma.me.uk>
> >> > wrote:
> >> > > > > > Thanks for the KIP, +1 from me.
> >> > > > > >
> >> > > > > > Ismael
> >> > > > > >
> >> > > > > > On Wed, Aug 9, 2017 at 1:24 AM, Ewen Cheslack-Postava <
> >> > > > ewen@confluent.io
> >> > > > > >
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > >> Hi all,
> >> > > > > >>
> >> > > > > >> I posted a simple new KIP for a problem we see with a lot of
> >> > users:
> >> > > > > >> KIP-186: Increase offsets retention default to 7 days
> >> > > > > >>
> >> > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> > > > > >> 186%3A+Increase+offsets+retention+default+to+7+days
> >> > > > > >>
> >> > > > > >> Note that in addition to the KIP text itself, the linked JIRA
> >> > > already
> >> > > > > >> existed and has a bunch of discussion on the subject.
> >> > > > > >>
> >> > > > > >> -Ewen
> >> > > > > >>
> >> > > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > Sönke Liebau
> >> > > > Partner
> >> > > > Tel. +49 179 7940878
> >> > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel -
> >> Germany
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > -- Guozhang
> >> >
> >>
> >
> >
>
Re: [DISCUSS] KIP-186: Increase offsets retention default to 7 days
Posted by Manikumar <ma...@gmail.com>.
looks like VOTE thread is *NOT* started for this KIP.
On Fri, Oct 6, 2017 at 11:23 PM, Manikumar <ma...@gmail.com>
wrote:
> looks like VOTE thread is started for this KIP.
>
>
> On Wed, Aug 16, 2017 at 5:39 PM, Stevo Slavić <ss...@gmail.com> wrote:
>
>> +1 for making consistent default log and offsets retention time.
>> I like Stephane's suggestion too, log retention override should override
>> offset retention too if not explicitly configured.
>>
>> Please consider additionally:
>> - introducing offsets.retention.hours config property
>> - syncing log and offsets retention.check.interval.ms, if there's no real
>> reason for the two to differ
>> -- consider making retention check interval by default (if not explicitly
>> configured) a fraction of retention time
>> - name all "offsets" configs with "offsets" prefix (now it's a mix of
>> singular/"offset" and plural/"offsets")
>>
>>
>> On Fri, Aug 11, 2017 at 2:01 AM, Guozhang Wang <wa...@gmail.com>
>> wrote:
>>
>> > +1 from me
>> >
>> > On Wed, Aug 9, 2017 at 9:40 AM, Jason Gustafson <ja...@confluent.io>
>> > wrote:
>> >
>> > > +1 on the bump to 7 days. Wanted to mention one minor point. The
>> > > OffsetCommit RPC still provides the ability to set the retention time
>> > from
>> > > the client, but we do not use it in the consumer. Should we consider
>> > adding
>> > > a consumer config to set this? Given the problems people had with the
>> old
>> > > default, such a config would probably have gotten a fair bit of use.
>> > Maybe
>> > > it's less necessary with the new default, but there may be situations
>> > where
>> > > you don't want to keep the offsets for too long. For example, the
>> console
>> > > consumer commits offsets with a generated group id. We might want to
>> set
>> > a
>> > > low retention time to keep it from filling the offset cache with
>> garbage
>> > > from such groups.
>> > >
>> >
>> > I agree with Jason here, but maybe itself deserves a separate KIP
>> > discussion.
>> >
>> >
>> > >
>> > > -Jason
>> > >
>> > > On Wed, Aug 9, 2017 at 5:24 AM, Sönke Liebau <
>> > > soenke.liebau@opencore.com.invalid> wrote:
>> > >
>> > > > Just had this create issues at a customer as well, +1
>> > > >
>> > > > On Wed, Aug 9, 2017 at 11:46 AM, Mickael Maison <
>> > > mickael.maison@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Yes the current default is too short, +1
>> > > > >
>> > > > > On Wed, Aug 9, 2017 at 8:56 AM, Ismael Juma <is...@juma.me.uk>
>> > wrote:
>> > > > > > Thanks for the KIP, +1 from me.
>> > > > > >
>> > > > > > Ismael
>> > > > > >
>> > > > > > On Wed, Aug 9, 2017 at 1:24 AM, Ewen Cheslack-Postava <
>> > > > ewen@confluent.io
>> > > > > >
>> > > > > > wrote:
>> > > > > >
>> > > > > >> Hi all,
>> > > > > >>
>> > > > > >> I posted a simple new KIP for a problem we see with a lot of
>> > users:
>> > > > > >> KIP-186: Increase offsets retention default to 7 days
>> > > > > >>
>> > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > > > > >> 186%3A+Increase+offsets+retention+default+to+7+days
>> > > > > >>
>> > > > > >> Note that in addition to the KIP text itself, the linked JIRA
>> > > already
>> > > > > >> existed and has a bunch of discussion on the subject.
>> > > > > >>
>> > > > > >> -Ewen
>> > > > > >>
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Sönke Liebau
>> > > > Partner
>> > > > Tel. +49 179 7940878
>> > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel -
>> Germany
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > -- Guozhang
>> >
>>
>
>
Re: [DISCUSS] KIP-186: Increase offsets retention default to 7 days
Posted by Ted Yu <yu...@gmail.com>.
+1 on the KIP.
bq. introducing offsets.retention.hours config property
Probably the introduction would cause confusion among users due to the
existing minutes config.
On Fri, Oct 6, 2017 at 10:53 AM, Manikumar <ma...@gmail.com>
wrote:
> looks like VOTE thread is started for this KIP.
>
> On Wed, Aug 16, 2017 at 5:39 PM, Stevo Slavić <ss...@gmail.com> wrote:
>
> > +1 for making consistent default log and offsets retention time.
> > I like Stephane's suggestion too, log retention override should override
> > offset retention too if not explicitly configured.
> >
> > Please consider additionally:
> > - introducing offsets.retention.hours config property
> > - syncing log and offsets retention.check.interval.ms, if there's no
> real
> > reason for the two to differ
> > -- consider making retention check interval by default (if not explicitly
> > configured) a fraction of retention time
> > - name all "offsets" configs with "offsets" prefix (now it's a mix of
> > singular/"offset" and plural/"offsets")
> >
> >
> > On Fri, Aug 11, 2017 at 2:01 AM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >
> > > +1 from me
> > >
> > > On Wed, Aug 9, 2017 at 9:40 AM, Jason Gustafson <ja...@confluent.io>
> > > wrote:
> > >
> > > > +1 on the bump to 7 days. Wanted to mention one minor point. The
> > > > OffsetCommit RPC still provides the ability to set the retention time
> > > from
> > > > the client, but we do not use it in the consumer. Should we consider
> > > adding
> > > > a consumer config to set this? Given the problems people had with the
> > old
> > > > default, such a config would probably have gotten a fair bit of use.
> > > Maybe
> > > > it's less necessary with the new default, but there may be situations
> > > where
> > > > you don't want to keep the offsets for too long. For example, the
> > console
> > > > consumer commits offsets with a generated group id. We might want to
> > set
> > > a
> > > > low retention time to keep it from filling the offset cache with
> > garbage
> > > > from such groups.
> > > >
> > >
> > > I agree with Jason here, but maybe itself deserves a separate KIP
> > > discussion.
> > >
> > >
> > > >
> > > > -Jason
> > > >
> > > > On Wed, Aug 9, 2017 at 5:24 AM, Sönke Liebau <
> > > > soenke.liebau@opencore.com.invalid> wrote:
> > > >
> > > > > Just had this create issues at a customer as well, +1
> > > > >
> > > > > On Wed, Aug 9, 2017 at 11:46 AM, Mickael Maison <
> > > > mickael.maison@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Yes the current default is too short, +1
> > > > > >
> > > > > > On Wed, Aug 9, 2017 at 8:56 AM, Ismael Juma <is...@juma.me.uk>
> > > wrote:
> > > > > > > Thanks for the KIP, +1 from me.
> > > > > > >
> > > > > > > Ismael
> > > > > > >
> > > > > > > On Wed, Aug 9, 2017 at 1:24 AM, Ewen Cheslack-Postava <
> > > > > ewen@confluent.io
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi all,
> > > > > > >>
> > > > > > >> I posted a simple new KIP for a problem we see with a lot of
> > > users:
> > > > > > >> KIP-186: Increase offsets retention default to 7 days
> > > > > > >>
> > > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > >> 186%3A+Increase+offsets+retention+default+to+7+days
> > > > > > >>
> > > > > > >> Note that in addition to the KIP text itself, the linked JIRA
> > > > already
> > > > > > >> existed and has a bunch of discussion on the subject.
> > > > > > >>
> > > > > > >> -Ewen
> > > > > > >>
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Sönke Liebau
> > > > > Partner
> > > > > Tel. +49 179 7940878
> > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel -
> Germany
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
Re: [DISCUSS] KIP-186: Increase offsets retention default to 7 days
Posted by Manikumar <ma...@gmail.com>.
looks like VOTE thread is started for this KIP.
On Wed, Aug 16, 2017 at 5:39 PM, Stevo Slavić <ss...@gmail.com> wrote:
> +1 for making consistent default log and offsets retention time.
> I like Stephane's suggestion too, log retention override should override
> offset retention too if not explicitly configured.
>
> Please consider additionally:
> - introducing offsets.retention.hours config property
> - syncing log and offsets retention.check.interval.ms, if there's no real
> reason for the two to differ
> -- consider making retention check interval by default (if not explicitly
> configured) a fraction of retention time
> - name all "offsets" configs with "offsets" prefix (now it's a mix of
> singular/"offset" and plural/"offsets")
>
>
> On Fri, Aug 11, 2017 at 2:01 AM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > +1 from me
> >
> > On Wed, Aug 9, 2017 at 9:40 AM, Jason Gustafson <ja...@confluent.io>
> > wrote:
> >
> > > +1 on the bump to 7 days. Wanted to mention one minor point. The
> > > OffsetCommit RPC still provides the ability to set the retention time
> > from
> > > the client, but we do not use it in the consumer. Should we consider
> > adding
> > > a consumer config to set this? Given the problems people had with the
> old
> > > default, such a config would probably have gotten a fair bit of use.
> > Maybe
> > > it's less necessary with the new default, but there may be situations
> > where
> > > you don't want to keep the offsets for too long. For example, the
> console
> > > consumer commits offsets with a generated group id. We might want to
> set
> > a
> > > low retention time to keep it from filling the offset cache with
> garbage
> > > from such groups.
> > >
> >
> > I agree with Jason here, but maybe itself deserves a separate KIP
> > discussion.
> >
> >
> > >
> > > -Jason
> > >
> > > On Wed, Aug 9, 2017 at 5:24 AM, Sönke Liebau <
> > > soenke.liebau@opencore.com.invalid> wrote:
> > >
> > > > Just had this create issues at a customer as well, +1
> > > >
> > > > On Wed, Aug 9, 2017 at 11:46 AM, Mickael Maison <
> > > mickael.maison@gmail.com>
> > > > wrote:
> > > >
> > > > > Yes the current default is too short, +1
> > > > >
> > > > > On Wed, Aug 9, 2017 at 8:56 AM, Ismael Juma <is...@juma.me.uk>
> > wrote:
> > > > > > Thanks for the KIP, +1 from me.
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Wed, Aug 9, 2017 at 1:24 AM, Ewen Cheslack-Postava <
> > > > ewen@confluent.io
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Hi all,
> > > > > >>
> > > > > >> I posted a simple new KIP for a problem we see with a lot of
> > users:
> > > > > >> KIP-186: Increase offsets retention default to 7 days
> > > > > >>
> > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > >> 186%3A+Increase+offsets+retention+default+to+7+days
> > > > > >>
> > > > > >> Note that in addition to the KIP text itself, the linked JIRA
> > > already
> > > > > >> existed and has a bunch of discussion on the subject.
> > > > > >>
> > > > > >> -Ewen
> > > > > >>
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Sönke Liebau
> > > > Partner
> > > > Tel. +49 179 7940878
> > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
Re: [DISCUSS] KIP-186: Increase offsets retention default to 7 days
Posted by Stevo Slavić <ss...@gmail.com>.
+1 for making consistent default log and offsets retention time.
I like Stephane's suggestion too, log retention override should override
offset retention too if not explicitly configured.
Please consider additionally:
- introducing offsets.retention.hours config property
- syncing log and offsets retention.check.interval.ms, if there's no real
reason for the two to differ
-- consider making retention check interval by default (if not explicitly
configured) a fraction of retention time
- name all "offsets" configs with "offsets" prefix (now it's a mix of
singular/"offset" and plural/"offsets")
On Fri, Aug 11, 2017 at 2:01 AM, Guozhang Wang <wa...@gmail.com> wrote:
> +1 from me
>
> On Wed, Aug 9, 2017 at 9:40 AM, Jason Gustafson <ja...@confluent.io>
> wrote:
>
> > +1 on the bump to 7 days. Wanted to mention one minor point. The
> > OffsetCommit RPC still provides the ability to set the retention time
> from
> > the client, but we do not use it in the consumer. Should we consider
> adding
> > a consumer config to set this? Given the problems people had with the old
> > default, such a config would probably have gotten a fair bit of use.
> Maybe
> > it's less necessary with the new default, but there may be situations
> where
> > you don't want to keep the offsets for too long. For example, the console
> > consumer commits offsets with a generated group id. We might want to set
> a
> > low retention time to keep it from filling the offset cache with garbage
> > from such groups.
> >
>
> I agree with Jason here, but maybe itself deserves a separate KIP
> discussion.
>
>
> >
> > -Jason
> >
> > On Wed, Aug 9, 2017 at 5:24 AM, Sönke Liebau <
> > soenke.liebau@opencore.com.invalid> wrote:
> >
> > > Just had this create issues at a customer as well, +1
> > >
> > > On Wed, Aug 9, 2017 at 11:46 AM, Mickael Maison <
> > mickael.maison@gmail.com>
> > > wrote:
> > >
> > > > Yes the current default is too short, +1
> > > >
> > > > On Wed, Aug 9, 2017 at 8:56 AM, Ismael Juma <is...@juma.me.uk>
> wrote:
> > > > > Thanks for the KIP, +1 from me.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Wed, Aug 9, 2017 at 1:24 AM, Ewen Cheslack-Postava <
> > > ewen@confluent.io
> > > > >
> > > > > wrote:
> > > > >
> > > > >> Hi all,
> > > > >>
> > > > >> I posted a simple new KIP for a problem we see with a lot of
> users:
> > > > >> KIP-186: Increase offsets retention default to 7 days
> > > > >>
> > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > >> 186%3A+Increase+offsets+retention+default+to+7+days
> > > > >>
> > > > >> Note that in addition to the KIP text itself, the linked JIRA
> > already
> > > > >> existed and has a bunch of discussion on the subject.
> > > > >>
> > > > >> -Ewen
> > > > >>
> > > >
> > >
> > >
> > >
> > > --
> > > Sönke Liebau
> > > Partner
> > > Tel. +49 179 7940878
> > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
> > >
> >
>
>
>
> --
> -- Guozhang
>
Re: [DISCUSS] KIP-186: Increase offsets retention default to 7 days
Posted by Guozhang Wang <wa...@gmail.com>.
+1 from me
On Wed, Aug 9, 2017 at 9:40 AM, Jason Gustafson <ja...@confluent.io> wrote:
> +1 on the bump to 7 days. Wanted to mention one minor point. The
> OffsetCommit RPC still provides the ability to set the retention time from
> the client, but we do not use it in the consumer. Should we consider adding
> a consumer config to set this? Given the problems people had with the old
> default, such a config would probably have gotten a fair bit of use. Maybe
> it's less necessary with the new default, but there may be situations where
> you don't want to keep the offsets for too long. For example, the console
> consumer commits offsets with a generated group id. We might want to set a
> low retention time to keep it from filling the offset cache with garbage
> from such groups.
>
I agree with Jason here, but maybe itself deserves a separate KIP
discussion.
>
> -Jason
>
> On Wed, Aug 9, 2017 at 5:24 AM, Sönke Liebau <
> soenke.liebau@opencore.com.invalid> wrote:
>
> > Just had this create issues at a customer as well, +1
> >
> > On Wed, Aug 9, 2017 at 11:46 AM, Mickael Maison <
> mickael.maison@gmail.com>
> > wrote:
> >
> > > Yes the current default is too short, +1
> > >
> > > On Wed, Aug 9, 2017 at 8:56 AM, Ismael Juma <is...@juma.me.uk> wrote:
> > > > Thanks for the KIP, +1 from me.
> > > >
> > > > Ismael
> > > >
> > > > On Wed, Aug 9, 2017 at 1:24 AM, Ewen Cheslack-Postava <
> > ewen@confluent.io
> > > >
> > > > wrote:
> > > >
> > > >> Hi all,
> > > >>
> > > >> I posted a simple new KIP for a problem we see with a lot of users:
> > > >> KIP-186: Increase offsets retention default to 7 days
> > > >>
> > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > >> 186%3A+Increase+offsets+retention+default+to+7+days
> > > >>
> > > >> Note that in addition to the KIP text itself, the linked JIRA
> already
> > > >> existed and has a bunch of discussion on the subject.
> > > >>
> > > >> -Ewen
> > > >>
> > >
> >
> >
> >
> > --
> > Sönke Liebau
> > Partner
> > Tel. +49 179 7940878
> > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
> >
>
--
-- Guozhang
Re: [DISCUSS] KIP-186: Increase offsets retention default to 7 days
Posted by Jason Gustafson <ja...@confluent.io>.
+1 on the bump to 7 days. Wanted to mention one minor point. The
OffsetCommit RPC still provides the ability to set the retention time from
the client, but we do not use it in the consumer. Should we consider adding
a consumer config to set this? Given the problems people had with the old
default, such a config would probably have gotten a fair bit of use. Maybe
it's less necessary with the new default, but there may be situations where
you don't want to keep the offsets for too long. For example, the console
consumer commits offsets with a generated group id. We might want to set a
low retention time to keep it from filling the offset cache with garbage
from such groups.
-Jason
On Wed, Aug 9, 2017 at 5:24 AM, Sönke Liebau <
soenke.liebau@opencore.com.invalid> wrote:
> Just had this create issues at a customer as well, +1
>
> On Wed, Aug 9, 2017 at 11:46 AM, Mickael Maison <mi...@gmail.com>
> wrote:
>
> > Yes the current default is too short, +1
> >
> > On Wed, Aug 9, 2017 at 8:56 AM, Ismael Juma <is...@juma.me.uk> wrote:
> > > Thanks for the KIP, +1 from me.
> > >
> > > Ismael
> > >
> > > On Wed, Aug 9, 2017 at 1:24 AM, Ewen Cheslack-Postava <
> ewen@confluent.io
> > >
> > > wrote:
> > >
> > >> Hi all,
> > >>
> > >> I posted a simple new KIP for a problem we see with a lot of users:
> > >> KIP-186: Increase offsets retention default to 7 days
> > >>
> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >> 186%3A+Increase+offsets+retention+default+to+7+days
> > >>
> > >> Note that in addition to the KIP text itself, the linked JIRA already
> > >> existed and has a bunch of discussion on the subject.
> > >>
> > >> -Ewen
> > >>
> >
>
>
>
> --
> Sönke Liebau
> Partner
> Tel. +49 179 7940878
> OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
>
Re: [DISCUSS] KIP-186: Increase offsets retention default to 7 days
Posted by Sönke Liebau <so...@opencore.com.INVALID>.
Just had this create issues at a customer as well, +1
On Wed, Aug 9, 2017 at 11:46 AM, Mickael Maison <mi...@gmail.com>
wrote:
> Yes the current default is too short, +1
>
> On Wed, Aug 9, 2017 at 8:56 AM, Ismael Juma <is...@juma.me.uk> wrote:
> > Thanks for the KIP, +1 from me.
> >
> > Ismael
> >
> > On Wed, Aug 9, 2017 at 1:24 AM, Ewen Cheslack-Postava <ewen@confluent.io
> >
> > wrote:
> >
> >> Hi all,
> >>
> >> I posted a simple new KIP for a problem we see with a lot of users:
> >> KIP-186: Increase offsets retention default to 7 days
> >>
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> 186%3A+Increase+offsets+retention+default+to+7+days
> >>
> >> Note that in addition to the KIP text itself, the linked JIRA already
> >> existed and has a bunch of discussion on the subject.
> >>
> >> -Ewen
> >>
>
--
Sönke Liebau
Partner
Tel. +49 179 7940878
OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
Re: [DISCUSS] KIP-186: Increase offsets retention default to 7 days
Posted by Mickael Maison <mi...@gmail.com>.
Yes the current default is too short, +1
On Wed, Aug 9, 2017 at 8:56 AM, Ismael Juma <is...@juma.me.uk> wrote:
> Thanks for the KIP, +1 from me.
>
> Ismael
>
> On Wed, Aug 9, 2017 at 1:24 AM, Ewen Cheslack-Postava <ew...@confluent.io>
> wrote:
>
>> Hi all,
>>
>> I posted a simple new KIP for a problem we see with a lot of users:
>> KIP-186: Increase offsets retention default to 7 days
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> 186%3A+Increase+offsets+retention+default+to+7+days
>>
>> Note that in addition to the KIP text itself, the linked JIRA already
>> existed and has a bunch of discussion on the subject.
>>
>> -Ewen
>>
Re: [DISCUSS] KIP-186: Increase offsets retention default to 7 days
Posted by Ismael Juma <is...@juma.me.uk>.
Thanks for the KIP, +1 from me.
Ismael
On Wed, Aug 9, 2017 at 1:24 AM, Ewen Cheslack-Postava <ew...@confluent.io>
wrote:
> Hi all,
>
> I posted a simple new KIP for a problem we see with a lot of users:
> KIP-186: Increase offsets retention default to 7 days
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 186%3A+Increase+offsets+retention+default+to+7+days
>
> Note that in addition to the KIP text itself, the linked JIRA already
> existed and has a bunch of discussion on the subject.
>
> -Ewen
>
Re: [DISCUSS] KIP-186: Increase offsets retention default to 7 days
Posted by Stephane Maarek <st...@simplemachines.com.au>.
Any interest on having offsets.retention.minutes= log.retention.(ms|minutes|hours) as a dynamic setting if not set, and having the option the override to a constant value?
That would address different types of deployments as well, who modify the default log retention period
On 10/8/17, 5:11 am, "Apurva Mehta" <ap...@confluent.io> wrote:
Thanks for the KIP. +1 from me.
On Tue, Aug 8, 2017 at 5:24 PM, Ewen Cheslack-Postava <ew...@confluent.io>
wrote:
> Hi all,
>
> I posted a simple new KIP for a problem we see with a lot of users:
> KIP-186: Increase offsets retention default to 7 days
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 186%3A+Increase+offsets+retention+default+to+7+days
>
> Note that in addition to the KIP text itself, the linked JIRA already
> existed and has a bunch of discussion on the subject.
>
> -Ewen
>
Re: [DISCUSS] KIP-186: Increase offsets retention default to 7 days
Posted by Apurva Mehta <ap...@confluent.io>.
Thanks for the KIP. +1 from me.
On Tue, Aug 8, 2017 at 5:24 PM, Ewen Cheslack-Postava <ew...@confluent.io>
wrote:
> Hi all,
>
> I posted a simple new KIP for a problem we see with a lot of users:
> KIP-186: Increase offsets retention default to 7 days
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 186%3A+Increase+offsets+retention+default+to+7+days
>
> Note that in addition to the KIP text itself, the linked JIRA already
> existed and has a bunch of discussion on the subject.
>
> -Ewen
>
Re: [DISCUSS] KIP-186: Increase offsets retention default to 7 days
Posted by James Cheng <wu...@gmail.com>.
+1 from me!
-James
> On Aug 8, 2017, at 5:24 PM, Ewen Cheslack-Postava <ew...@confluent.io> wrote:
>
> Hi all,
>
> I posted a simple new KIP for a problem we see with a lot of users:
> KIP-186: Increase offsets retention default to 7 days
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-186%3A+Increase+offsets+retention+default+to+7+days
>
> Note that in addition to the KIP text itself, the linked JIRA already
> existed and has a bunch of discussion on the subject.
>
> -Ewen