You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Vahid S Hashemian <va...@us.ibm.com> on 2018/05/23 18:19:17 UTC
Re: [DISCUSS] KIP-289: Improve the default group id behavior in
KafkaConsumer
Hi Ted,
Thanks for reviewing the KIP. I updated the KIP and introduced an error
code for the scenario described.
--Vahid
From: Ted Yu <yu...@gmail.com>
To: dev@kafka.apache.org
Date: 04/27/2018 04:31 PM
Subject: Re: [DISCUSS] KIP-289: Improve the default group id
behavior in KafkaConsumer
bq. If they attempt an offset commit they will receive an error.
Can you outline what specific error would be encountered ?
Thanks
On Fri, Apr 27, 2018 at 2:17 PM, Vahid S Hashemian <
vahidhashemian@us.ibm.com> wrote:
> Hi all,
>
> I have drafted a proposal for improving the behavior of KafkaConsumer
when
> using the default group id:
>
https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 289%3A+Improve+the+default+group+id+behavior+in+KafkaConsumer
> The proposal based on the issue and suggestion reported in KAFKA-6774.
>
> Your feedback is welcome!
>
> Thanks.
> --Vahid
>
>
Re: [DISCUSS] KIP-289: Improve the default group id behavior in
KafkaConsumer
Posted by Vahid S Hashemian <va...@us.ibm.com>.
Hi Dhruvil,
Could you please share the reasoning behind your suggestion?
My understanding is that there is more restrictions around topic names
because they appear in folder names, and there are limitations by file
systems on what can be used in file/folder names or how long they can be.
Thanks.
--Vahid
From: Dhruvil Shah <dh...@confluent.io>
To: dev@kafka.apache.org
Date: 07/20/2018 11:02 AM
Subject: Re: [DISCUSS] KIP-289: Improve the default group id
behavior in KafkaConsumer
If we are looking into deprecating the empty group id, would it also make
sense to have the same character restriction for it as that for topic
names? We have stricter validation for topic names but none for group id
and transaction id. I think we should (eventually) make character
restriction the same across all entities. We may not necessarily want to
do
this as part of the proposed KIP but I wanted to get an opinion on it
anyway.
On Fri, Jul 20, 2018 at 10:22 AM Rajini Sivaram <ra...@gmail.com>
wrote:
> +1 to deprecate use of "" as group.id since it is odd to have a resource
> name that you cannot set ACLs for. Agree, we have to support older
clients
> though.
>
> Thanks,
>
> Rajini
>
> On Fri, Jul 20, 2018 at 5:25 PM, Jason Gustafson <ja...@confluent.io>
> wrote:
>
> > Hi Vahid,
> >
> > Sorry for getting to this so late. I think there are two things here:
> >
> > 1. The use of "" as a groupId has always been a dubious practice at
best.
> > We definitely ought to deprecate its use in the client. Perhaps in the
> next
> > major release, we can remove support completely. However, since older
> > clients depend on it, we may have to continue letting the broker
support
> it
> > to some extent. Perhaps we just need to bump the OffsetCommit request
API
> > and only accept the offset commit for older versions. You probably
have
> to
> > do this anyway if you want to introduce the new error code since old
> > clients will not expect it.
> >
> > 2. There should be a way for the consumer to indicate that it has no
> group
> > id and will not commit offsets. This is an explicit instruction that
the
> > consumer should not bother with coordinator lookup and such. We
currently
> > have some brittle logic in place to let users avoid the coordinator
> lookup,
> > but it is a bit error-prone. I was hoping that we could change the
> default
> > value of group.id to be null so that the user had to take an explicit
> > action to opt into coordinator management (groups or offsets).
However,
> it
> > is true that some users may be unknowingly depending on offset storage
if
> > they are using both the default group.id and the default
> > enable.auto.commit. Perhaps one option is to disable
enable.auto.commit
> > automatically if no group.id is specified? I am not sure if there are
> any
> > drawbacks, but my feeling is that implicit use of the empty group.id
to
> > commit offsets is more likely to be causing users unexpected problems
> than
> > actually providing a useful capability.
> >
> > Thoughts?
> >
> > Thanks,
> > Jason
> >
> >
> >
> >
> > On Mon, May 28, 2018 at 9:50 AM, Vahid S Hashemian <
> > vahidhashemian@us.ibm.com> wrote:
> >
> > > Hi Viktor,
> > >
> > > Thanks for sharing your opinion.
> > > So you're in favor of disallowing the empty ("") group id altogether
> > (even
> > > for fetching).
> > > Given that ideally no one should be using the empty group id (at
least
> in
> > > a production setting) I think the impact would be minimal in either
> case.
> > >
> > > But as you said, let's hear what others think and I'd be happy to
> modify
> > > the KIP if needed.
> > >
> > > Regards.
> > > --Vahid
> > >
> > >
> > >
> > >
> > > From: Viktor Somogyi <vi...@gmail.com>
> > > To: dev <de...@kafka.apache.org>
> > > Date: 05/28/2018 05:18 AM
> > > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > > behavior in KafkaConsumer
> > >
> > >
> > >
> > > Hi Vahid,
> > >
> > > (with the argument that using the default group id for offset commit
> > > should not be the user's intention in practice).
> > >
> > > Yea, so in my opinion too this use case doesn't seem too practical.
> Also
> > I
> > > think breaking the offset commit is not smaller from this
perspective
> > than
> > > breaking fetch and offset fetch. If we suppose that someone uses the
> > > default group id and we break the offset commit then that might be
> harder
> > > to detect than breaking the whole thing altogether. (If we think
about
> an
> > > upgrade situation.)
> > > So since we think it is not a practical use case, I think it would
be
> > > better to break altogether but ofc that's just my 2 cents :). Let's
> > gather
> > > other's input as well.
> > >
> > > Cheers,
> > > Viktor
> > >
> > > On Fri, May 25, 2018 at 5:43 PM, Vahid S Hashemian <
> > > vahidhashemian@us.ibm.com> wrote:
> > >
> > > > Hi Victor,
> > > >
> > > > Thanks for reviewing the KIP.
> > > >
> > > > Yes, to minimize the backward compatibility impact, there would be
no
> > > harm
> > > > in letting a stand-alone consumer consume messages under a ""
group
> id
> > > (as
> > > > long as there is no offset commit).
> > > > It would have to knowingly seek to an offset or rely on the
> > > > auto.offset.reset config for the starting offset.
> > > > This way the existing functionality would be preserved for the
most
> > part
> > > > (with the argument that using the default group id for offset
commit
> > > > should not be the user's intention in practice).
> > > >
> > > > Does it seem reasonable?
> > > >
> > > > Thanks.
> > > > --Vahid
> > > >
> > > >
> > > >
> > > >
> > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > To: dev <de...@kafka.apache.org>
> > > > Date: 05/25/2018 04:56 AM
> > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group
id
> > > > behavior in KafkaConsumer
> > > >
> > > >
> > > >
> > > > Hi Vahid,
> > > >
> > > > When reading your KIP I coldn't fully understand why did you
decide
> at
> > > > failing with "offset_commit" in case #2? Can't we fail with an
empty
> > > group
> > > > id even in "fetch" or "fetch_offset"? What was the reason for
> deciding
> > > to
> > > > fail at "offset_commit"? Was it because of upgrade compatibility
> > > reasons?
> > > >
> > > > Thanks,
> > > > Viktor
> > > >
> > > > On Thu, May 24, 2018 at 12:06 AM, Ted Yu <yu...@gmail.com>
> wrote:
> > > >
> > > > > Looks good to me.
> > > > > -------- Original message --------From: Vahid S Hashemian <
> > > > > vahidhashemian@us.ibm.com> Date: 5/23/18 11:19 AM (GMT-08:00)
> To:
> > > > > dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-289: Improve the
> > > default
> > > > > group id behavior in KafkaConsumer
> > > > > Hi Ted,
> > > > >
> > > > > Thanks for reviewing the KIP. I updated the KIP and introduced
an
> > > error
> > > > > code for the scenario described.
> > > > >
> > > > > --Vahid
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > From: Ted Yu <yu...@gmail.com>
> > > > > To: dev@kafka.apache.org
> > > > > Date: 04/27/2018 04:31 PM
> > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group
id
> > > > > behavior in KafkaConsumer
> > > > >
> > > > >
> > > > >
> > > > > bq. If they attempt an offset commit they will receive an error.
> > > > >
> > > > > Can you outline what specific error would be encountered ?
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Fri, Apr 27, 2018 at 2:17 PM, Vahid S Hashemian <
> > > > > vahidhashemian@us.ibm.com> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I have drafted a proposal for improving the behavior of
> > > KafkaConsumer
> > > > > when
> > > > > > using the default group id:
> > > > > >
> > > > >
> > > >
> > >
https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >
> > > >
> > > > >
> > > > > > 289%3A+Improve+the+default+group+id+behavior+in+KafkaConsumer
> > > > > > The proposal based on the issue and suggestion reported in
> > > KAFKA-6774.
> > > > > >
> > > > > > Your feedback is welcome!
> > > > > >
> > > > > > Thanks.
> > > > > > --Vahid
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
>
Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer
Posted by Dhruvil Shah <dh...@confluent.io>.
If we are looking into deprecating the empty group id, would it also make
sense to have the same character restriction for it as that for topic
names? We have stricter validation for topic names but none for group id
and transaction id. I think we should (eventually) make character
restriction the same across all entities. We may not necessarily want to do
this as part of the proposed KIP but I wanted to get an opinion on it
anyway.
On Fri, Jul 20, 2018 at 10:22 AM Rajini Sivaram <ra...@gmail.com>
wrote:
> +1 to deprecate use of "" as group.id since it is odd to have a resource
> name that you cannot set ACLs for. Agree, we have to support older clients
> though.
>
> Thanks,
>
> Rajini
>
> On Fri, Jul 20, 2018 at 5:25 PM, Jason Gustafson <ja...@confluent.io>
> wrote:
>
> > Hi Vahid,
> >
> > Sorry for getting to this so late. I think there are two things here:
> >
> > 1. The use of "" as a groupId has always been a dubious practice at best.
> > We definitely ought to deprecate its use in the client. Perhaps in the
> next
> > major release, we can remove support completely. However, since older
> > clients depend on it, we may have to continue letting the broker support
> it
> > to some extent. Perhaps we just need to bump the OffsetCommit request API
> > and only accept the offset commit for older versions. You probably have
> to
> > do this anyway if you want to introduce the new error code since old
> > clients will not expect it.
> >
> > 2. There should be a way for the consumer to indicate that it has no
> group
> > id and will not commit offsets. This is an explicit instruction that the
> > consumer should not bother with coordinator lookup and such. We currently
> > have some brittle logic in place to let users avoid the coordinator
> lookup,
> > but it is a bit error-prone. I was hoping that we could change the
> default
> > value of group.id to be null so that the user had to take an explicit
> > action to opt into coordinator management (groups or offsets). However,
> it
> > is true that some users may be unknowingly depending on offset storage if
> > they are using both the default group.id and the default
> > enable.auto.commit. Perhaps one option is to disable enable.auto.commit
> > automatically if no group.id is specified? I am not sure if there are
> any
> > drawbacks, but my feeling is that implicit use of the empty group.id to
> > commit offsets is more likely to be causing users unexpected problems
> than
> > actually providing a useful capability.
> >
> > Thoughts?
> >
> > Thanks,
> > Jason
> >
> >
> >
> >
> > On Mon, May 28, 2018 at 9:50 AM, Vahid S Hashemian <
> > vahidhashemian@us.ibm.com> wrote:
> >
> > > Hi Viktor,
> > >
> > > Thanks for sharing your opinion.
> > > So you're in favor of disallowing the empty ("") group id altogether
> > (even
> > > for fetching).
> > > Given that ideally no one should be using the empty group id (at least
> in
> > > a production setting) I think the impact would be minimal in either
> case.
> > >
> > > But as you said, let's hear what others think and I'd be happy to
> modify
> > > the KIP if needed.
> > >
> > > Regards.
> > > --Vahid
> > >
> > >
> > >
> > >
> > > From: Viktor Somogyi <vi...@gmail.com>
> > > To: dev <de...@kafka.apache.org>
> > > Date: 05/28/2018 05:18 AM
> > > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > > behavior in KafkaConsumer
> > >
> > >
> > >
> > > Hi Vahid,
> > >
> > > (with the argument that using the default group id for offset commit
> > > should not be the user's intention in practice).
> > >
> > > Yea, so in my opinion too this use case doesn't seem too practical.
> Also
> > I
> > > think breaking the offset commit is not smaller from this perspective
> > than
> > > breaking fetch and offset fetch. If we suppose that someone uses the
> > > default group id and we break the offset commit then that might be
> harder
> > > to detect than breaking the whole thing altogether. (If we think about
> an
> > > upgrade situation.)
> > > So since we think it is not a practical use case, I think it would be
> > > better to break altogether but ofc that's just my 2 cents :). Let's
> > gather
> > > other's input as well.
> > >
> > > Cheers,
> > > Viktor
> > >
> > > On Fri, May 25, 2018 at 5:43 PM, Vahid S Hashemian <
> > > vahidhashemian@us.ibm.com> wrote:
> > >
> > > > Hi Victor,
> > > >
> > > > Thanks for reviewing the KIP.
> > > >
> > > > Yes, to minimize the backward compatibility impact, there would be no
> > > harm
> > > > in letting a stand-alone consumer consume messages under a "" group
> id
> > > (as
> > > > long as there is no offset commit).
> > > > It would have to knowingly seek to an offset or rely on the
> > > > auto.offset.reset config for the starting offset.
> > > > This way the existing functionality would be preserved for the most
> > part
> > > > (with the argument that using the default group id for offset commit
> > > > should not be the user's intention in practice).
> > > >
> > > > Does it seem reasonable?
> > > >
> > > > Thanks.
> > > > --Vahid
> > > >
> > > >
> > > >
> > > >
> > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > To: dev <de...@kafka.apache.org>
> > > > Date: 05/25/2018 04:56 AM
> > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > > > behavior in KafkaConsumer
> > > >
> > > >
> > > >
> > > > Hi Vahid,
> > > >
> > > > When reading your KIP I coldn't fully understand why did you decide
> at
> > > > failing with "offset_commit" in case #2? Can't we fail with an empty
> > > group
> > > > id even in "fetch" or "fetch_offset"? What was the reason for
> deciding
> > > to
> > > > fail at "offset_commit"? Was it because of upgrade compatibility
> > > reasons?
> > > >
> > > > Thanks,
> > > > Viktor
> > > >
> > > > On Thu, May 24, 2018 at 12:06 AM, Ted Yu <yu...@gmail.com>
> wrote:
> > > >
> > > > > Looks good to me.
> > > > > -------- Original message --------From: Vahid S Hashemian <
> > > > > vahidhashemian@us.ibm.com> Date: 5/23/18 11:19 AM (GMT-08:00)
> To:
> > > > > dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-289: Improve the
> > > default
> > > > > group id behavior in KafkaConsumer
> > > > > Hi Ted,
> > > > >
> > > > > Thanks for reviewing the KIP. I updated the KIP and introduced an
> > > error
> > > > > code for the scenario described.
> > > > >
> > > > > --Vahid
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > From: Ted Yu <yu...@gmail.com>
> > > > > To: dev@kafka.apache.org
> > > > > Date: 04/27/2018 04:31 PM
> > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > > > > behavior in KafkaConsumer
> > > > >
> > > > >
> > > > >
> > > > > bq. If they attempt an offset commit they will receive an error.
> > > > >
> > > > > Can you outline what specific error would be encountered ?
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Fri, Apr 27, 2018 at 2:17 PM, Vahid S Hashemian <
> > > > > vahidhashemian@us.ibm.com> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I have drafted a proposal for improving the behavior of
> > > KafkaConsumer
> > > > > when
> > > > > > using the default group id:
> > > > > >
> > > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >
> > > >
> > > > >
> > > > > > 289%3A+Improve+the+default+group+id+behavior+in+KafkaConsumer
> > > > > > The proposal based on the issue and suggestion reported in
> > > KAFKA-6774.
> > > > > >
> > > > > > Your feedback is welcome!
> > > > > >
> > > > > > Thanks.
> > > > > > --Vahid
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
>
Re: [DISCUSS] KIP-289: Improve the default group id behavior in
KafkaConsumer
Posted by Colin McCabe <cm...@apache.org>.
Thanks, Vahid.
best,
Colin
On Fri, Aug 3, 2018, at 12:15, Vahid S Hashemian wrote:
> The KIP has been updated.
>
> If it looks good and there are no further comments I'll start a vote early
> next week.
>
> Thanks.
> --Vahid
>
>
>
>
> From: "Vahid S Hashemian" <va...@us.ibm.com>
> To: dev@kafka.apache.org
> Date: 08/03/2018 08:19 AM
> Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> behavior in KafkaConsumer
>
>
>
> Hi Jason, Stanislav,
>
> Thanks for your feedback. I'll update the KIP later today based on your
> suggestion.
> I'll send a note once the update is complete.
>
> Thanks.
> --Vahid
>
>
>
> From: Stanislav Kozlovski <st...@confluent.io>
> To: dev@kafka.apache.org
> Date: 08/03/2018 02:34 AM
> Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> behavior in KafkaConsumer
>
>
>
> Hi,
>
> I also believe the minimal change might be better. In my opinion, the less
> logic we can stuff in the broker - the better. The use ofempty group.id is
> not ideal and should be discouraged, but I don't inherently believe strict
> validation in the broker is useful, since it doesn't break anything
> internal (except maybe not being able to set ACLs).
> Having sufficient warnings or even restriction on the clients I feel might
> be a good enough compromise.
>
> That being said, I very much look forward to voting for this KIP.
>
> Thanks,
> Stanislav
>
> On Fri, Aug 3, 2018 at 7:51 AM Jason Gustafson <ja...@confluent.io> wrote:
>
> > Hey Vahid,
> >
> > I think we're in agreement. Perhaps since none of us feel too strongly,
> we
> > should go for the more minimal change? I'll vote for the KIP either way
> as
> > long as we can change the default group.id.
> >
> > Thanks,
> > Jason
> >
>
>
>
Re: [DISCUSS] KIP-289: Improve the default group id behavior in
KafkaConsumer
Posted by Vahid S Hashemian <va...@us.ibm.com>.
The KIP has been updated.
If it looks good and there are no further comments I'll start a vote early
next week.
Thanks.
--Vahid
From: "Vahid S Hashemian" <va...@us.ibm.com>
To: dev@kafka.apache.org
Date: 08/03/2018 08:19 AM
Subject: Re: [DISCUSS] KIP-289: Improve the default group id
behavior in KafkaConsumer
Hi Jason, Stanislav,
Thanks for your feedback. I'll update the KIP later today based on your
suggestion.
I'll send a note once the update is complete.
Thanks.
--Vahid
From: Stanislav Kozlovski <st...@confluent.io>
To: dev@kafka.apache.org
Date: 08/03/2018 02:34 AM
Subject: Re: [DISCUSS] KIP-289: Improve the default group id
behavior in KafkaConsumer
Hi,
I also believe the minimal change might be better. In my opinion, the less
logic we can stuff in the broker - the better. The use ofempty group.id is
not ideal and should be discouraged, but I don't inherently believe strict
validation in the broker is useful, since it doesn't break anything
internal (except maybe not being able to set ACLs).
Having sufficient warnings or even restriction on the clients I feel might
be a good enough compromise.
That being said, I very much look forward to voting for this KIP.
Thanks,
Stanislav
On Fri, Aug 3, 2018 at 7:51 AM Jason Gustafson <ja...@confluent.io> wrote:
> Hey Vahid,
>
> I think we're in agreement. Perhaps since none of us feel too strongly,
we
> should go for the more minimal change? I'll vote for the KIP either way
as
> long as we can change the default group.id.
>
> Thanks,
> Jason
>
Re: [DISCUSS] KIP-289: Improve the default group id behavior in
KafkaConsumer
Posted by Vahid S Hashemian <va...@us.ibm.com>.
Hi Jason, Stanislav,
Thanks for your feedback. I'll update the KIP later today based on your
suggestion.
I'll send a note once the update is complete.
Thanks.
--Vahid
From: Stanislav Kozlovski <st...@confluent.io>
To: dev@kafka.apache.org
Date: 08/03/2018 02:34 AM
Subject: Re: [DISCUSS] KIP-289: Improve the default group id
behavior in KafkaConsumer
Hi,
I also believe the minimal change might be better. In my opinion, the less
logic we can stuff in the broker - the better. The use ofempty group.id is
not ideal and should be discouraged, but I don't inherently believe strict
validation in the broker is useful, since it doesn't break anything
internal (except maybe not being able to set ACLs).
Having sufficient warnings or even restriction on the clients I feel might
be a good enough compromise.
That being said, I very much look forward to voting for this KIP.
Thanks,
Stanislav
On Fri, Aug 3, 2018 at 7:51 AM Jason Gustafson <ja...@confluent.io> wrote:
> Hey Vahid,
>
> I think we're in agreement. Perhaps since none of us feel too strongly,
we
> should go for the more minimal change? I'll vote for the KIP either way
as
> long as we can change the default group.id.
>
> Thanks,
> Jason
>
> On Thu, Aug 2, 2018 at 11:55 AM, Vahid S Hashemian <
> vahidhashemian@us.ibm.com> wrote:
>
> > Hi Jason, Colin,
> >
> > Thanks for your feedback.
> >
> > If we are going to keep supporting the empty group id like before then
it
> > makes sense to do what you suggested.
> >
> > Though, IMHO if using the empty group id for committing offsets is a
bad
> > practice we should stop supporting it at some point; and it would be
> > better to do it sooner before potentially more users adopt that.
> > I understand that changing the default would reduce the chance of
someone
> > using it, but it doesn't stop them. Note that we did not receive any
> > objections to dropping the support even right-away (a good sign).
> > We could keep supporting the old API version in case someone must use
> this
> > group id like before. But at least we won't expose new users to this
bad
> > practice.
> >
> > I, too, do now have a strong opinion about this. So if the current KIP
> > should not move forward as is, I'm okay with just changing the default
> and
> > deprecating the use on the client, as you suggested.
> >
> > Thanks.
> > --Vahid
> >
> >
> >
> > From: Colin McCabe <cm...@apache.org>
> > To: dev@kafka.apache.org
> > Date: 08/02/2018 11:23 AM
> > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > behavior in KafkaConsumer
> >
> >
> >
> > Thanks, Jason. I don't have a very strong opinion on this. But like
you
> > said, if we skip bumping the RPC versions, this would be a smaller
> change,
> > which might be good.
> >
> > best,
> > Colin
> >
> >
> > On Wed, Aug 1, 2018, at 17:43, Jason Gustafson wrote:
> > > Hey Vahid,
> > >
> > > I talked with Colin offline. I think specifically he felt the
version
> > bump
> > > on the broker was overkill since the broker still has to support the
> > empty
> > > group id for older versions. I had thought that eventually we would
be
> > able
> > > to remove those old versions, but it's true that this may not happen
> > until
> > > indefinitely far in the future. I think the main improvement here is
> > > changing the default group.id to null instead of "". I could go
either
> > way
> > > on whether bumping the protocol is useful. I do think it is helpful
> > though
> > > to signal clearly that it its use is deprecated and discouraged,
> > especially
> > > in light of the ACL problem. I guess we could just deprecate the use
on
> > the
> > > client. What do you think?
> > >
> > > Thanks,
> > > Jason
> > >
> > > On Wed, Aug 1, 2018 at 3:19 PM, Vahid S Hashemian
> > <vahidhashemian@us.ibm.com
> > > > wrote:
> > >
> > > > Thanks Jason for responding to Colin's concerns.
> > > >
> > > > If there are no other comment / feedback / objection I'll start a
> vote
> > > > soon.
> > > >
> > > > Thanks.
> > > > --Vahid
> > > >
> > > >
> > > >
> > > > From: Jason Gustafson <ja...@confluent.io>
> > > > To: dev <de...@kafka.apache.org>
> > > > Date: 07/27/2018 10:38 AM
> > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group
id
> > > > behavior in KafkaConsumer
> > > >
> > > >
> > > >
> > > > Hey Colin,
> > > >
> > > > The problem is both that the empty group id is the default value
and
> > that
> > > > it is actually accepted by the broker for offset commits. Combine
> that
> > > > with
> > > > the fact that auto commit is enabled by default and you users get
> > > > surprising behavior. If you look at a random Kafka cluster, you'll
> > > > probably
> > > > find a bunch of inadvertent offset commits for the empty group id.
I
> > was
> > > > hoping we could distinguish between users who are using the empty
> > group id
> > > > as an accident of the default configuration and those who use it
> > > > intentionally. By default, there will be no group id and the
consumer
> > will
> > > > not commit offsets. If a user has actually intentionally used the
> > empty
> > > > group id, however, it will continue to work. I actually think
there
> > are
> > > > probably very few people doing this (maybe even no one), but I
> thought
> > we
> > > > might err on the side of compatibility.
> > > >
> > > > The big incompatible change here is having brokers reject using
> > > > assign(...)
> > > > > with empty / null group.id.
> > > >
> > > >
> > > > This is not correct. In the proposal, the broker will only reject
the
> > > > empty
> > > > group id for the new version of OffsetCommit. Older clients, which
> > cannot
> > > > be changed, will continue to work because the old versions of the
> > > > OffsetCommit API still accept the empty group id. The null group
id
> is
> > > > different from the empty group id: it is not allowed in any
version
> of
> > the
> > > > API. It is basically a way to indicate that the consumer has no
> > dependence
> > > > on the coordinator at all, which we actually have a surprising
number
> > of
> > > > use cases for. Furthermore, if a user has an actual need for the
> empty
> > > > group id, it will still be allowed. We are just deprecating it.
> > > >
> > > > -Jason
> > > >
> > > > On Fri, Jul 27, 2018 at 9:56 AM, Colin McCabe <cm...@apache.org>
> > wrote:
> > > >
> > > > > Sorry if this is a silly question, but what's the rationale for
> > > > switching
> > > > > to using null for the default group id, rather than the empty
> > string?
> > > > > Continuing to use the empty string seems like less churn. And
> after
> > > > all,
> > > > > we're not using the empty string group name for anything else.
> > > > >
> > > > > The big incompatible change here is having brokers reject using
> > > > > assign(...) with empty / null group.id. If I understand
> correctly,
> > the
> > > > > KIP proposes that this change be made on the brokers on the next
> > > > > incompatible Kafka release. But that has nothing to do with
client
> > > > > versions. Why not just have a broker config which controls
this?
> > Maybe
> > > > "
> > > > > allow.assign.empty.group.id", or something like that. At first,
> the
> > > > > default will be true, and then eventually we can flip it over to
> > false.
> > > > >
> > > > > It seems like the main rationale for tying this behavior to the
> > Kafka
> > > > > client version is to force people to stop using the empty group
id
> > so
> > > > that
> > > > > they can upgrade their clients. But it's also possible that
people
> > will
> > > > > stop upgrading their Kafka clients instead. That would be
pretty
> > > > negative
> > > > > since they'd miss out on any efficiency and feature
improvements
> in
> > the
> > > > > new clients and eventually have to do more protocol downgrading,
> > etc.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > >
> > > > > On Thu, Jul 26, 2018, at 11:50, Vahid S Hashemian wrote:
> > > > > > Hi Jason,
> > > > > >
> > > > > > That makes sense.
> > > > > > I have updated the KIP based on the recent feedback.
> > > > > >
> > > > > > Thanks!
> > > > > > --Vahid
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > From: Jason Gustafson <ja...@confluent.io>
> > > > > > To: dev <de...@kafka.apache.org>
> > > > > > Date: 07/25/2018 02:23 PM
> > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default
group
> > id
> > > > > > behavior in KafkaConsumer
> > > > > >
> > > > > >
> > > > > >
> > > > > > Hi Vahid,
> > > > > >
> > > > > > I was thinking we'd only use the old API version if we had to.
> > That
> > > > is,
> > > > > > only if the user has explicitly configured "" as the group.id.
> > > > > Otherwise,
> > > > > > we'd just use the new one. Another option is to just drop
support
> > in
> > > > the
> > > > > > client for the empty group id, but usually we allow a
deprecation
> > > > period
> > > > > > for changes like this.
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > > On Wed, Jul 25, 2018 at 12:49 PM, Vahid S Hashemian <
> > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > >
> > > > > > > Hi Jason,
> > > > > > >
> > > > > > > Thanks for additional clarification.
> > > > > > >
> > > > > > > So the next version of the OffsetCommit API will return an
> > > > > > > INVALID_GROUP_ID error for empty group ids; but on the
client
> > side
> > > > we
> > > > > > call
> > > > > > > the older version of the client until the next major
release.
> > > > > > > The table below should summarize this.
> > > > > > >
> > > > > > > +-----------------------------------------------------+
> > > > > > > | Client (group.id="") |
> > > > > > > +-----------------------------------------------------+
> > > > > > > | pre-2.1 | 2.1 |
> 3.0
> > |
> > > > > > >
> > > > > > +-----+-----------+---------+------------------------+------
> > > > > ------------+
> > > > > > > | | V5 (cur.) | works | works | works
|
> > > > > > > + API
> > > > > >
> > +-----------+---------+------------------------+------------------+
> > > > > > > | | V6 | N/A | N/A (calls V5/warning) |
> > > > > INVALID_GROUP_ID
> > > > > > |
> > > > > > >
> > > > > > +-----+-----------+---------+------------------------+------
> > > > > ------------+
> > > > > > >
> > > > > > > Assumptions:
> > > > > > > * 2.1: The target release version for this KIP
> > > > > > > * 3.0: The next major release
> > > > > > >
> > > > > > > Please advise if you see an issue; otherwise, I'll update
the
> > KIP
> > > > > > > accordingly.
> > > > > > >
> > > > > > > Thanks!
> > > > > > > --Vahid
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > From: Jason Gustafson <ja...@confluent.io>
> > > > > > > To: dev <de...@kafka.apache.org>
> > > > > > > Date: 07/25/2018 12:08 AM
> > > > > > > Subject: ***UNCHECKED*** Re: [DISCUSS] KIP-289:
Improve
> > the
> > > > > > default
> > > > > > > group id behavior in KafkaConsumer
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Hey Vahid,
> > > > > > >
> > > > > > > Sorry for the confusion. I think we all agree that going
> > forward, we
> > > > > > > shouldn't support the empty group id, so the question is
just
> > around
> > > > > > > compatibility. I think we have to bump the OffsetCommit API
> > version
> > > > so
> > > > > > > that
> > > > > > > old clients which are unknowingly depending on the default
> empty
> > > > group
> > > > > > id
> > > > > > > will continue to work with new brokers. For new versions of
the
> > > > > client,
> > > > > > we
> > > > > > > can either drop support for the empty group id immediately
or
> we
> > can
> > > > > > give
> > > > > > > users a grace period. I was thinking we would do the latter.
We
> > can
> > > > > > change
> > > > > > > the default group.id, but in the case that a user has
> explicitly
> > > > > > > configured
> > > > > > > the empty group, then we can just use an old version of the
> > > > > OffsetCommit
> > > > > > > API which still supports it. In a future release, we can
drop
> > this
> > > > > > support
> > > > > > > and only use the latest OffsetCommit version. Does that make
> > sense?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Jason
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jul 24, 2018 at 12:36 PM, Vahid S Hashemian <
> > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > >
> > > > > > > > Hi Jason,
> > > > > > > >
> > > > > > > > Thanks for clarifying.
> > > > > > > >
> > > > > > > > So if we are going to continue supporting the empty group
id
> > as
> > > > > before
> > > > > > > > (with only an addition of a deprecation warning), and
disable
> > > > > > > > enable.auto.commit for the new default (null) group id on
the
> > > > client
> > > > > > > side,
> > > > > > > > do we really need to bump up the OffsetCommit version?
> > > > > > > >
> > > > > > > > You mentioned "If an explicit empty string is configured
for
> > the
> > > > > group
> > > > > > > id,
> > > > > > > > then maybe we keep the current behavior for compatibility"
> > which
> > > > > makes
> > > > > > > > sense to me, but I find it in conflict with your earlier
> > > > suggestion
> > > > > > "we
> > > > > > > > just need to bump the OffsetCommit request API and only
> accept
> > the
> > > > > > > offset
> > > > > > > > commit for older versions.". Maybe I'm missing something?
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > > --Vahid
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > From: Jason Gustafson <ja...@confluent.io>
> > > > > > > > To: dev <de...@kafka.apache.org>
> > > > > > > > Date: 07/23/2018 10:52 PM
> > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default
> > group
> > > > id
> > > > > > > > behavior in KafkaConsumer
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Hey Vahid,
> > > > > > > >
> > > > > > > > Thanks for the updates. Just to clarify, I was suggesting
> that
> > we
> > > > > > > disable
> > > > > > > > enable.auto.commit only if no explicit group.id is
> > configured.
> > If
> > > > an
> > > > > > > > explicit empty string is configured for the group id, then
> > maybe
> > > > we
> > > > > > keep
> > > > > > > > the current behavior for compatibility. We can log a
warning
> > > > > > mentioning
> > > > > > > > the
> > > > > > > > deprecation and we can use the old version of the
> OffsetCommit
> > API
> > > > > > that
> > > > > > > > allows the empty group id. In a later release, we can drop
> > this
> > > > > > support
> > > > > > > in
> > > > > > > > the client. Does that seem reasonable?
> > > > > > > >
> > > > > > > > By the way, instead of using the new ILLEGAL_OFFSET_COMMIT
> > error
> > > > > code,
> > > > > > > > couldn't we use INVALID_GROUP_ID?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Jason
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Jul 23, 2018 at 5:14 PM, Stanislav Kozlovski
> > > > > > > > <stanislav@confluent.io
> > > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hey Vahid,
> > > > > > > > >
> > > > > > > > > No I don't see an issue with it. I believe it to be the
> best
> > > > > > approach.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Stanisav
> > > > > > > > >
> > > > > > > > > On Mon, Jul 23, 2018 at 12:41 PM Vahid S Hashemian <
> > > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > > >
> > > > > > > > > > Hi Stanislav,
> > > > > > > > > >
> > > > > > > > > > Thanks for the feedback.
> > > > > > > > > > Do you see an issue with using `null` as the default
> group
> > id
> > > > (as
> > > > > > > > > > addressed by Jason in his response)?
> > > > > > > > > > This default group id would not support offset commits
> and
> > > > > > consumers
> > > > > > > > > would
> > > > > > > > > > use `auto.offset.reset` config when there is no
current
> > > > offset.
> > > > > > > > > >
> > > > > > > > > > Thanks.
> > > > > > > > > > --Vahid
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > From: Stanislav Kozlovski <st...@confluent.io>
> > > > > > > > > > To: dev@kafka.apache.org
> > > > > > > > > > Date: 07/20/2018 11:09 AM
> > > > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the
> default
> > > > group
> > > > > > id
> > > > > > > > > > behavior in KafkaConsumer
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I agree with Jason's notion that
> > > > > > > > > > > implicit use of the empty group.id to commit
offsets
> is
> > > > more
> > > > > > > likely
> > > > > > > > > to
> > > > > > > > > > be causing users unexpected problems than actually
> > providing a
> > > > > > > useful
> > > > > > > > > > capability.
> > > > > > > > > > I was initially confused that this is the behavior
when
> > > > > > > investigating
> > > > > > > > a
> > > > > > > > > > new-ish JIRA issue <
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> >
https://issues.apache.org/jira/browse/KAFKA-6758
> >
> > > >
> > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > > > > about
> > > > > > > > > > the same topic.
> > > > > > > > > > So, +1 to deprecating "" as a group.id
> > > > > > > > > >
> > > > > > > > > > The question after that becomes what the *default*
value
> > > > should
> > > > > be
> > > > > > -
> > > > > > > > > > should
> > > > > > > > > > we:
> > > > > > > > > > a) treat an unconfigured group.id consumer as a sort
of
> > > > > > intermittent
> > > > > > > > > > consumer where you don't store offsets at all (thereby
> > making
> > > > the
> > > > > > > user
> > > > > > > > > > explicitly sign up for them)
> > > > > > > > > > b) have a default value which makes use of them? I
sort
> of
> > > > like
> > > > > > the
> > > > > > > > > > former.
> > > > > > > > > >
> > > > > > > > > > @Dhruvil, thinking about it at a high-level - yes. I
> can't
> > > > think
> > > > > > of
> > > > > > > a
> > > > > > > > > > situation where it makes sense to name something an
empty
> > > > string
> > > > > > as
> > > > > > > > far
> > > > > > > > > as
> > > > > > > > > > I'm aware - to me it seems like potential for
confusion
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Jul 20, 2018 at 10:22 AM Rajini Sivaram
> > > > > > > > <rajinisivaram@gmail.com
> > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > +1 to deprecate use of "" as group.id since it is
odd
> to
> > > > have
> > > > > a
> > > > > > > > > resource
> > > > > > > > > > > name that you cannot set ACLs for. Agree, we have to
> > support
> > > > > > older
> > > > > > > > > > clients
> > > > > > > > > > > though.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > >
> > > > > > > > > > > Rajini
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Jul 20, 2018 at 5:25 PM, Jason Gustafson
> > > > > > > > <ja...@confluent.io>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Vahid,
> > > > > > > > > > > >
> > > > > > > > > > > > Sorry for getting to this so late. I think there
are
> > two
> > > > > > things
> > > > > > > > here:
> > > > > > > > > > > >
> > > > > > > > > > > > 1. The use of "" as a groupId has always been a
> > dubious
> > > > > > practice
> > > > > > > > at
> > > > > > > > > > best.
> > > > > > > > > > > > We definitely ought to deprecate its use in the
> > client.
> > > > > > Perhaps
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > next
> > > > > > > > > > > > major release, we can remove support completely.
> > However,
> > > > > > since
> > > > > > > > older
> > > > > > > > > > > > clients depend on it, we may have to continue
letting
> > the
> > > > > > broker
> > > > > > > > > > support
> > > > > > > > > > > it
> > > > > > > > > > > > to some extent. Perhaps we just need to bump the
> > > > OffsetCommit
> > > > > > > > request
> > > > > > > > > > API
> > > > > > > > > > > > and only accept the offset commit for older
versions.
> > You
> > > > > > > probably
> > > > > > > > > > have
> > > > > > > > > > > to
> > > > > > > > > > > > do this anyway if you want to introduce the new
error
> > code
> > > > > > since
> > > > > > > > old
> > > > > > > > > > > > clients will not expect it.
> > > > > > > > > > > >
> > > > > > > > > > > > 2. There should be a way for the consumer to
indicate
> > that
> > > > it
> > > > > > > has
> > > > > > > > no
> > > > > > > > > > > group
> > > > > > > > > > > > id and will not commit offsets. This is an
explicit
> > > > > > instruction
> > > > > > > > that
> > > > > > > > > > the
> > > > > > > > > > > > consumer should not bother with coordinator lookup
> and
> > > > such.
> > > > > > We
> > > > > > > > > > currently
> > > > > > > > > > > > have some brittle logic in place to let users
avoid
> > the
> > > > > > > > coordinator
> > > > > > > > > > > lookup,
> > > > > > > > > > > > but it is a bit error-prone. I was hoping that we
> > could
> > > > > change
> > > > > > > the
> > > > > > > > > > > default
> > > > > > > > > > > > value of group.id to be null so that the user had
to
> > take
> > > > an
> > > > > > > > > explicit
> > > > > > > > > > > > action to opt into coordinator management (groups
or
> > > > > offsets).
> > > > > > > > > > However,
> > > > > > > > > > > it
> > > > > > > > > > > > is true that some users may be unknowingly
depending
> > on
> > > > > offset
> > > > > > > > > storage
> > > > > > > > > > if
> > > > > > > > > > > > they are using both the default group.id and the
> > default
> > > > > > > > > > > > enable.auto.commit. Perhaps one option is to
disable
> > > > > > > > > > enable.auto.commit
> > > > > > > > > > > > automatically if no group.id is specified? I am
not
> > sure
> > > > if
> > > > > > > there
> > > > > > > > > are
> > > > > > > > > > > any
> > > > > > > > > > > > drawbacks, but my feeling is that implicit use of
the
> > > > empty
> > > > > > > > group.id
> > > > > > > > > > to
> > > > > > > > > > > > commit offsets is more likely to be causing users
> > > > unexpected
> > > > > > > > problems
> > > > > > > > > > > than
> > > > > > > > > > > > actually providing a useful capability.
> > > > > > > > > > > >
> > > > > > > > > > > > Thoughts?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Jason
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, May 28, 2018 at 9:50 AM, Vahid S Hashemian
<
> > > > > > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Viktor,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for sharing your opinion.
> > > > > > > > > > > > > So you're in favor of disallowing the empty ("")
> > group
> > > > id
> > > > > > > > > altogether
> > > > > > > > > > > > (even
> > > > > > > > > > > > > for fetching).
> > > > > > > > > > > > > Given that ideally no one should be using the
empty
> > > > group
> > > > > id
> > > > > > > (at
> > > > > > > > > > least
> > > > > > > > > > > in
> > > > > > > > > > > > > a production setting) I think the impact would
be
> > > > minimal
> > > > > in
> > > > > > > > either
> > > > > > > > > > > case.
> > > > > > > > > > > > >
> > > > > > > > > > > > > But as you said, let's hear what others think
and
> > I'd be
> > > > > > happy
> > > > > > > > to
> > > > > > > > > > > modify
> > > > > > > > > > > > > the KIP if needed.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regards.
> > > > > > > > > > > > > --Vahid
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > > > > > > > > > > To: dev <de...@kafka.apache.org>
> > > > > > > > > > > > > Date: 05/28/2018 05:18 AM
> > > > > > > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve
the
> > > > default
> > > > > > > group
> > > > > > > > id
> > > > > > > > > > > > > behavior in KafkaConsumer
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Vahid,
> > > > > > > > > > > > >
> > > > > > > > > > > > > (with the argument that using the default group
id
> > for
> > > > > > offset
> > > > > > > > > commit
> > > > > > > > > > > > > should not be the user's intention in practice).
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yea, so in my opinion too this use case doesn't
> seem
> > too
> > > > > > > > practical.
> > > > > > > > > > > Also
> > > > > > > > > > > > I
> > > > > > > > > > > > > think breaking the offset commit is not smaller
> from
> > > > this
> > > > > > > > > > perspective
> > > > > > > > > > > > than
> > > > > > > > > > > > > breaking fetch and offset fetch. If we suppose
that
> > > > someone
> > > > > > > uses
> > > > > > > > > the
> > > > > > > > > > > > > default group id and we break the offset commit
> then
> > > > that
> > > > > > > might
> > > > > > > > be
> > > > > > > > > > > harder
> > > > > > > > > > > > > to detect than breaking the whole thing
altogether.
> > (If
> > > > we
> > > > > > > think
> > > > > > > > > > about
> > > > > > > > > > > an
> > > > > > > > > > > > > upgrade situation.)
> > > > > > > > > > > > > So since we think it is not a practical use
case, I
> > > > think
> > > > > it
> > > > > > > > would
> > > > > > > > > > be
> > > > > > > > > > > > > better to break altogether but ofc that's just
my 2
> > > > cents
> > > > > > :).
> > > > > > > > Let's
> > > > > > > > > > > > gather
> > > > > > > > > > > > > other's input as well.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > Viktor
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, May 25, 2018 at 5:43 PM, Vahid S
Hashemian
> <
> > > > > > > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Victor,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for reviewing the KIP.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Yes, to minimize the backward compatibility
> > impact,
> > > > there
> > > > > > > > would
> > > > > > > > > be
> > > > > > > > > > no
> > > > > > > > > > > > > harm
> > > > > > > > > > > > > > in letting a stand-alone consumer consume
> messages
> > > > under
> > > > > a
> > > > > > > ""
> > > > > > > > > > group
> > > > > > > > > > > id
> > > > > > > > > > > > > (as
> > > > > > > > > > > > > > long as there is no offset commit).
> > > > > > > > > > > > > > It would have to knowingly seek to an offset
or
> > rely
> > > > on
> > > > > > the
> > > > > > > > > > > > > > auto.offset.reset config for the starting
offset.
> > > > > > > > > > > > > > This way the existing functionality would be
> > preserved
> > > > > for
> > > > > > > the
> > > > > > > > > > most
> > > > > > > > > > > > part
> > > > > > > > > > > > > > (with the argument that using the default
group
> id
> > for
> > > > > > > offset
> > > > > > > > > > commit
> > > > > > > > > > > > > > should not be the user's intention in
practice).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Does it seem reasonable?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks.
> > > > > > > > > > > > > > --Vahid
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > From: Viktor Somogyi
<vi...@gmail.com>
> > > > > > > > > > > > > > To: dev <de...@kafka.apache.org>
> > > > > > > > > > > > > > Date: 05/25/2018 04:56 AM
> > > > > > > > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve
> the
> > > > > default
> > > > > > > > group
> > > > > > > > > > id
> > > > > > > > > > > > > > behavior in KafkaConsumer
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Vahid,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > When reading your KIP I coldn't fully
understand
> > why
> > > > did
> > > > > > you
> > > > > > > > > > decide
> > > > > > > > > > > at
> > > > > > > > > > > > > > failing with "offset_commit" in case #2? Can't
we
> > fail
> > > > > > with
> > > > > > > an
> > > > > > > > > > empty
> > > > > > > > > > > > > group
> > > > > > > > > > > > > > id even in "fetch" or "fetch_offset"? What was
> the
> > > > reason
> > > > > > > for
> > > > > > > > > > > deciding
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > fail at "offset_commit"? Was it because of
> upgrade
> > > > > > > > compatibility
> > > > > > > > > > > > > reasons?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > Viktor
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, May 24, 2018 at 12:06 AM, Ted Yu
> > > > > > > <yu...@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Looks good to me.
> > > > > > > > > > > > > > > -------- Original message --------From:
Vahid S
> > > > > > Hashemian
> > > > > > > <
> > > > > > > > > > > > > > > vahidhashemian@us.ibm.com> Date: 5/23/18
> 11:19
> > AM
> > > > > > > > > (GMT-08:00)
> > > > > > > > > > > To:
> > > > > > > > > > > > > > > dev@kafka.apache.org Subject: Re: [DISCUSS]
> > KIP-289:
> > > > > > > Improve
> > > > > > > > > the
> > > > > > > > > > > > > default
> > > > > > > > > > > > > > > group id behavior in KafkaConsumer
> > > > > > > > > > > > > > > Hi Ted,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for reviewing the KIP. I updated the
KIP
> > and
> > > > > > > > introduced
> > > > > > > > > > an
> > > > > > > > > > > > > error
> > > > > > > > > > > > > > > code for the scenario described.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > --Vahid
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > From: Ted Yu <yu...@gmail.com>
> > > > > > > > > > > > > > > To: dev@kafka.apache.org
> > > > > > > > > > > > > > > Date: 04/27/2018 04:31 PM
> > > > > > > > > > > > > > > Subject: Re: [DISCUSS] KIP-289:
Improve
> > the
> > > > > > default
> > > > > > > > > group
> > > > > > > > > > id
> > > > > > > > > > > > > > > behavior in KafkaConsumer
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > bq. If they attempt an offset commit they
will
> > > > receive
> > > > > > an
> > > > > > > > > error.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Can you outline what specific error would be
> > > > > encountered
> > > > > > ?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Apr 27, 2018 at 2:17 PM, Vahid S
> > Hashemian <
> > > > > > > > > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I have drafted a proposal for improving
the
> > > > behavior
> > > > > > of
> > > > > > > > > > > > > KafkaConsumer
> > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > using the default group id:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> >
https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >
> > > >
> > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > 289%3A+Improve+the+default+group+id+behavior+in+
> > > > > > > > > KafkaConsumer
> > > > > > > > > > > > > > > > The proposal based on the issue and
> suggestion
> > > > > > reported
> > > > > > > in
> > > > > > > > > > > > > KAFKA-6774.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Your feedback is welcome!
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks.
> > > > > > > > > > > > > > > > --Vahid
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Best,
> > > > > > > > > > Stanislav
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Best,
> > > > > > > > > Stanislav
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> >
> >
> >
> >
> >
> >
>
--
Best,
Stanislav
Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer
Posted by Stanislav Kozlovski <st...@confluent.io>.
Hi,
I also believe the minimal change might be better. In my opinion, the less
logic we can stuff in the broker - the better. The use ofempty group.id is
not ideal and should be discouraged, but I don't inherently believe strict
validation in the broker is useful, since it doesn't break anything
internal (except maybe not being able to set ACLs).
Having sufficient warnings or even restriction on the clients I feel might
be a good enough compromise.
That being said, I very much look forward to voting for this KIP.
Thanks,
Stanislav
On Fri, Aug 3, 2018 at 7:51 AM Jason Gustafson <ja...@confluent.io> wrote:
> Hey Vahid,
>
> I think we're in agreement. Perhaps since none of us feel too strongly, we
> should go for the more minimal change? I'll vote for the KIP either way as
> long as we can change the default group.id.
>
> Thanks,
> Jason
>
> On Thu, Aug 2, 2018 at 11:55 AM, Vahid S Hashemian <
> vahidhashemian@us.ibm.com> wrote:
>
> > Hi Jason, Colin,
> >
> > Thanks for your feedback.
> >
> > If we are going to keep supporting the empty group id like before then it
> > makes sense to do what you suggested.
> >
> > Though, IMHO if using the empty group id for committing offsets is a bad
> > practice we should stop supporting it at some point; and it would be
> > better to do it sooner before potentially more users adopt that.
> > I understand that changing the default would reduce the chance of someone
> > using it, but it doesn't stop them. Note that we did not receive any
> > objections to dropping the support even right-away (a good sign).
> > We could keep supporting the old API version in case someone must use
> this
> > group id like before. But at least we won't expose new users to this bad
> > practice.
> >
> > I, too, do now have a strong opinion about this. So if the current KIP
> > should not move forward as is, I'm okay with just changing the default
> and
> > deprecating the use on the client, as you suggested.
> >
> > Thanks.
> > --Vahid
> >
> >
> >
> > From: Colin McCabe <cm...@apache.org>
> > To: dev@kafka.apache.org
> > Date: 08/02/2018 11:23 AM
> > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > behavior in KafkaConsumer
> >
> >
> >
> > Thanks, Jason. I don't have a very strong opinion on this. But like you
> > said, if we skip bumping the RPC versions, this would be a smaller
> change,
> > which might be good.
> >
> > best,
> > Colin
> >
> >
> > On Wed, Aug 1, 2018, at 17:43, Jason Gustafson wrote:
> > > Hey Vahid,
> > >
> > > I talked with Colin offline. I think specifically he felt the version
> > bump
> > > on the broker was overkill since the broker still has to support the
> > empty
> > > group id for older versions. I had thought that eventually we would be
> > able
> > > to remove those old versions, but it's true that this may not happen
> > until
> > > indefinitely far in the future. I think the main improvement here is
> > > changing the default group.id to null instead of "". I could go either
> > way
> > > on whether bumping the protocol is useful. I do think it is helpful
> > though
> > > to signal clearly that it its use is deprecated and discouraged,
> > especially
> > > in light of the ACL problem. I guess we could just deprecate the use on
> > the
> > > client. What do you think?
> > >
> > > Thanks,
> > > Jason
> > >
> > > On Wed, Aug 1, 2018 at 3:19 PM, Vahid S Hashemian
> > <vahidhashemian@us.ibm.com
> > > > wrote:
> > >
> > > > Thanks Jason for responding to Colin's concerns.
> > > >
> > > > If there are no other comment / feedback / objection I'll start a
> vote
> > > > soon.
> > > >
> > > > Thanks.
> > > > --Vahid
> > > >
> > > >
> > > >
> > > > From: Jason Gustafson <ja...@confluent.io>
> > > > To: dev <de...@kafka.apache.org>
> > > > Date: 07/27/2018 10:38 AM
> > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > > > behavior in KafkaConsumer
> > > >
> > > >
> > > >
> > > > Hey Colin,
> > > >
> > > > The problem is both that the empty group id is the default value and
> > that
> > > > it is actually accepted by the broker for offset commits. Combine
> that
> > > > with
> > > > the fact that auto commit is enabled by default and you users get
> > > > surprising behavior. If you look at a random Kafka cluster, you'll
> > > > probably
> > > > find a bunch of inadvertent offset commits for the empty group id. I
> > was
> > > > hoping we could distinguish between users who are using the empty
> > group id
> > > > as an accident of the default configuration and those who use it
> > > > intentionally. By default, there will be no group id and the consumer
> > will
> > > > not commit offsets. If a user has actually intentionally used the
> > empty
> > > > group id, however, it will continue to work. I actually think there
> > are
> > > > probably very few people doing this (maybe even no one), but I
> thought
> > we
> > > > might err on the side of compatibility.
> > > >
> > > > The big incompatible change here is having brokers reject using
> > > > assign(...)
> > > > > with empty / null group.id.
> > > >
> > > >
> > > > This is not correct. In the proposal, the broker will only reject the
> > > > empty
> > > > group id for the new version of OffsetCommit. Older clients, which
> > cannot
> > > > be changed, will continue to work because the old versions of the
> > > > OffsetCommit API still accept the empty group id. The null group id
> is
> > > > different from the empty group id: it is not allowed in any version
> of
> > the
> > > > API. It is basically a way to indicate that the consumer has no
> > dependence
> > > > on the coordinator at all, which we actually have a surprising number
> > of
> > > > use cases for. Furthermore, if a user has an actual need for the
> empty
> > > > group id, it will still be allowed. We are just deprecating it.
> > > >
> > > > -Jason
> > > >
> > > > On Fri, Jul 27, 2018 at 9:56 AM, Colin McCabe <cm...@apache.org>
> > wrote:
> > > >
> > > > > Sorry if this is a silly question, but what's the rationale for
> > > > switching
> > > > > to using null for the default group id, rather than the empty
> > string?
> > > > > Continuing to use the empty string seems like less churn. And
> after
> > > > all,
> > > > > we're not using the empty string group name for anything else.
> > > > >
> > > > > The big incompatible change here is having brokers reject using
> > > > > assign(...) with empty / null group.id. If I understand
> correctly,
> > the
> > > > > KIP proposes that this change be made on the brokers on the next
> > > > > incompatible Kafka release. But that has nothing to do with client
> > > > > versions. Why not just have a broker config which controls this?
> > Maybe
> > > > "
> > > > > allow.assign.empty.group.id", or something like that. At first,
> the
> > > > > default will be true, and then eventually we can flip it over to
> > false.
> > > > >
> > > > > It seems like the main rationale for tying this behavior to the
> > Kafka
> > > > > client version is to force people to stop using the empty group id
> > so
> > > > that
> > > > > they can upgrade their clients. But it's also possible that people
> > will
> > > > > stop upgrading their Kafka clients instead. That would be pretty
> > > > negative
> > > > > since they'd miss out on any efficiency and feature improvements
> in
> > the
> > > > > new clients and eventually have to do more protocol downgrading,
> > etc.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > >
> > > > > On Thu, Jul 26, 2018, at 11:50, Vahid S Hashemian wrote:
> > > > > > Hi Jason,
> > > > > >
> > > > > > That makes sense.
> > > > > > I have updated the KIP based on the recent feedback.
> > > > > >
> > > > > > Thanks!
> > > > > > --Vahid
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > From: Jason Gustafson <ja...@confluent.io>
> > > > > > To: dev <de...@kafka.apache.org>
> > > > > > Date: 07/25/2018 02:23 PM
> > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group
> > id
> > > > > > behavior in KafkaConsumer
> > > > > >
> > > > > >
> > > > > >
> > > > > > Hi Vahid,
> > > > > >
> > > > > > I was thinking we'd only use the old API version if we had to.
> > That
> > > > is,
> > > > > > only if the user has explicitly configured "" as the group.id.
> > > > > Otherwise,
> > > > > > we'd just use the new one. Another option is to just drop support
> > in
> > > > the
> > > > > > client for the empty group id, but usually we allow a deprecation
> > > > period
> > > > > > for changes like this.
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > > On Wed, Jul 25, 2018 at 12:49 PM, Vahid S Hashemian <
> > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > >
> > > > > > > Hi Jason,
> > > > > > >
> > > > > > > Thanks for additional clarification.
> > > > > > >
> > > > > > > So the next version of the OffsetCommit API will return an
> > > > > > > INVALID_GROUP_ID error for empty group ids; but on the client
> > side
> > > > we
> > > > > > call
> > > > > > > the older version of the client until the next major release.
> > > > > > > The table below should summarize this.
> > > > > > >
> > > > > > > +-----------------------------------------------------+
> > > > > > > | Client (group.id="") |
> > > > > > > +-----------------------------------------------------+
> > > > > > > | pre-2.1 | 2.1 |
> 3.0
> > |
> > > > > > >
> > > > > > +-----+-----------+---------+------------------------+------
> > > > > ------------+
> > > > > > > | | V5 (cur.) | works | works | works |
> > > > > > > + API
> > > > > >
> > +-----------+---------+------------------------+------------------+
> > > > > > > | | V6 | N/A | N/A (calls V5/warning) |
> > > > > INVALID_GROUP_ID
> > > > > > |
> > > > > > >
> > > > > > +-----+-----------+---------+------------------------+------
> > > > > ------------+
> > > > > > >
> > > > > > > Assumptions:
> > > > > > > * 2.1: The target release version for this KIP
> > > > > > > * 3.0: The next major release
> > > > > > >
> > > > > > > Please advise if you see an issue; otherwise, I'll update the
> > KIP
> > > > > > > accordingly.
> > > > > > >
> > > > > > > Thanks!
> > > > > > > --Vahid
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > From: Jason Gustafson <ja...@confluent.io>
> > > > > > > To: dev <de...@kafka.apache.org>
> > > > > > > Date: 07/25/2018 12:08 AM
> > > > > > > Subject: ***UNCHECKED*** Re: [DISCUSS] KIP-289: Improve
> > the
> > > > > > default
> > > > > > > group id behavior in KafkaConsumer
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Hey Vahid,
> > > > > > >
> > > > > > > Sorry for the confusion. I think we all agree that going
> > forward, we
> > > > > > > shouldn't support the empty group id, so the question is just
> > around
> > > > > > > compatibility. I think we have to bump the OffsetCommit API
> > version
> > > > so
> > > > > > > that
> > > > > > > old clients which are unknowingly depending on the default
> empty
> > > > group
> > > > > > id
> > > > > > > will continue to work with new brokers. For new versions of the
> > > > > client,
> > > > > > we
> > > > > > > can either drop support for the empty group id immediately or
> we
> > can
> > > > > > give
> > > > > > > users a grace period. I was thinking we would do the latter. We
> > can
> > > > > > change
> > > > > > > the default group.id, but in the case that a user has
> explicitly
> > > > > > > configured
> > > > > > > the empty group, then we can just use an old version of the
> > > > > OffsetCommit
> > > > > > > API which still supports it. In a future release, we can drop
> > this
> > > > > > support
> > > > > > > and only use the latest OffsetCommit version. Does that make
> > sense?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Jason
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jul 24, 2018 at 12:36 PM, Vahid S Hashemian <
> > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > >
> > > > > > > > Hi Jason,
> > > > > > > >
> > > > > > > > Thanks for clarifying.
> > > > > > > >
> > > > > > > > So if we are going to continue supporting the empty group id
> > as
> > > > > before
> > > > > > > > (with only an addition of a deprecation warning), and disable
> > > > > > > > enable.auto.commit for the new default (null) group id on the
> > > > client
> > > > > > > side,
> > > > > > > > do we really need to bump up the OffsetCommit version?
> > > > > > > >
> > > > > > > > You mentioned "If an explicit empty string is configured for
> > the
> > > > > group
> > > > > > > id,
> > > > > > > > then maybe we keep the current behavior for compatibility"
> > which
> > > > > makes
> > > > > > > > sense to me, but I find it in conflict with your earlier
> > > > suggestion
> > > > > > "we
> > > > > > > > just need to bump the OffsetCommit request API and only
> accept
> > the
> > > > > > > offset
> > > > > > > > commit for older versions.". Maybe I'm missing something?
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > > --Vahid
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > From: Jason Gustafson <ja...@confluent.io>
> > > > > > > > To: dev <de...@kafka.apache.org>
> > > > > > > > Date: 07/23/2018 10:52 PM
> > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default
> > group
> > > > id
> > > > > > > > behavior in KafkaConsumer
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Hey Vahid,
> > > > > > > >
> > > > > > > > Thanks for the updates. Just to clarify, I was suggesting
> that
> > we
> > > > > > > disable
> > > > > > > > enable.auto.commit only if no explicit group.id is
> > configured.
> > If
> > > > an
> > > > > > > > explicit empty string is configured for the group id, then
> > maybe
> > > > we
> > > > > > keep
> > > > > > > > the current behavior for compatibility. We can log a warning
> > > > > > mentioning
> > > > > > > > the
> > > > > > > > deprecation and we can use the old version of the
> OffsetCommit
> > API
> > > > > > that
> > > > > > > > allows the empty group id. In a later release, we can drop
> > this
> > > > > > support
> > > > > > > in
> > > > > > > > the client. Does that seem reasonable?
> > > > > > > >
> > > > > > > > By the way, instead of using the new ILLEGAL_OFFSET_COMMIT
> > error
> > > > > code,
> > > > > > > > couldn't we use INVALID_GROUP_ID?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Jason
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Jul 23, 2018 at 5:14 PM, Stanislav Kozlovski
> > > > > > > > <stanislav@confluent.io
> > > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hey Vahid,
> > > > > > > > >
> > > > > > > > > No I don't see an issue with it. I believe it to be the
> best
> > > > > > approach.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Stanisav
> > > > > > > > >
> > > > > > > > > On Mon, Jul 23, 2018 at 12:41 PM Vahid S Hashemian <
> > > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > > >
> > > > > > > > > > Hi Stanislav,
> > > > > > > > > >
> > > > > > > > > > Thanks for the feedback.
> > > > > > > > > > Do you see an issue with using `null` as the default
> group
> > id
> > > > (as
> > > > > > > > > > addressed by Jason in his response)?
> > > > > > > > > > This default group id would not support offset commits
> and
> > > > > > consumers
> > > > > > > > > would
> > > > > > > > > > use `auto.offset.reset` config when there is no current
> > > > offset.
> > > > > > > > > >
> > > > > > > > > > Thanks.
> > > > > > > > > > --Vahid
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > From: Stanislav Kozlovski <st...@confluent.io>
> > > > > > > > > > To: dev@kafka.apache.org
> > > > > > > > > > Date: 07/20/2018 11:09 AM
> > > > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the
> default
> > > > group
> > > > > > id
> > > > > > > > > > behavior in KafkaConsumer
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I agree with Jason's notion that
> > > > > > > > > > > implicit use of the empty group.id to commit offsets
> is
> > > > more
> > > > > > > likely
> > > > > > > > > to
> > > > > > > > > > be causing users unexpected problems than actually
> > providing a
> > > > > > > useful
> > > > > > > > > > capability.
> > > > > > > > > > I was initially confused that this is the behavior when
> > > > > > > investigating
> > > > > > > > a
> > > > > > > > > > new-ish JIRA issue <
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > https://issues.apache.org/jira/browse/KAFKA-6758
> >
> > > >
> > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > > > > about
> > > > > > > > > > the same topic.
> > > > > > > > > > So, +1 to deprecating "" as a group.id
> > > > > > > > > >
> > > > > > > > > > The question after that becomes what the *default* value
> > > > should
> > > > > be
> > > > > > -
> > > > > > > > > > should
> > > > > > > > > > we:
> > > > > > > > > > a) treat an unconfigured group.id consumer as a sort of
> > > > > > intermittent
> > > > > > > > > > consumer where you don't store offsets at all (thereby
> > making
> > > > the
> > > > > > > user
> > > > > > > > > > explicitly sign up for them)
> > > > > > > > > > b) have a default value which makes use of them? I sort
> of
> > > > like
> > > > > > the
> > > > > > > > > > former.
> > > > > > > > > >
> > > > > > > > > > @Dhruvil, thinking about it at a high-level - yes. I
> can't
> > > > think
> > > > > > of
> > > > > > > a
> > > > > > > > > > situation where it makes sense to name something an empty
> > > > string
> > > > > > as
> > > > > > > > far
> > > > > > > > > as
> > > > > > > > > > I'm aware - to me it seems like potential for confusion
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Jul 20, 2018 at 10:22 AM Rajini Sivaram
> > > > > > > > <rajinisivaram@gmail.com
> > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > +1 to deprecate use of "" as group.id since it is odd
> to
> > > > have
> > > > > a
> > > > > > > > > resource
> > > > > > > > > > > name that you cannot set ACLs for. Agree, we have to
> > support
> > > > > > older
> > > > > > > > > > clients
> > > > > > > > > > > though.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > >
> > > > > > > > > > > Rajini
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Jul 20, 2018 at 5:25 PM, Jason Gustafson
> > > > > > > > <ja...@confluent.io>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Vahid,
> > > > > > > > > > > >
> > > > > > > > > > > > Sorry for getting to this so late. I think there are
> > two
> > > > > > things
> > > > > > > > here:
> > > > > > > > > > > >
> > > > > > > > > > > > 1. The use of "" as a groupId has always been a
> > dubious
> > > > > > practice
> > > > > > > > at
> > > > > > > > > > best.
> > > > > > > > > > > > We definitely ought to deprecate its use in the
> > client.
> > > > > > Perhaps
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > next
> > > > > > > > > > > > major release, we can remove support completely.
> > However,
> > > > > > since
> > > > > > > > older
> > > > > > > > > > > > clients depend on it, we may have to continue letting
> > the
> > > > > > broker
> > > > > > > > > > support
> > > > > > > > > > > it
> > > > > > > > > > > > to some extent. Perhaps we just need to bump the
> > > > OffsetCommit
> > > > > > > > request
> > > > > > > > > > API
> > > > > > > > > > > > and only accept the offset commit for older versions.
> > You
> > > > > > > probably
> > > > > > > > > > have
> > > > > > > > > > > to
> > > > > > > > > > > > do this anyway if you want to introduce the new error
> > code
> > > > > > since
> > > > > > > > old
> > > > > > > > > > > > clients will not expect it.
> > > > > > > > > > > >
> > > > > > > > > > > > 2. There should be a way for the consumer to indicate
> > that
> > > > it
> > > > > > > has
> > > > > > > > no
> > > > > > > > > > > group
> > > > > > > > > > > > id and will not commit offsets. This is an explicit
> > > > > > instruction
> > > > > > > > that
> > > > > > > > > > the
> > > > > > > > > > > > consumer should not bother with coordinator lookup
> and
> > > > such.
> > > > > > We
> > > > > > > > > > currently
> > > > > > > > > > > > have some brittle logic in place to let users avoid
> > the
> > > > > > > > coordinator
> > > > > > > > > > > lookup,
> > > > > > > > > > > > but it is a bit error-prone. I was hoping that we
> > could
> > > > > change
> > > > > > > the
> > > > > > > > > > > default
> > > > > > > > > > > > value of group.id to be null so that the user had to
> > take
> > > > an
> > > > > > > > > explicit
> > > > > > > > > > > > action to opt into coordinator management (groups or
> > > > > offsets).
> > > > > > > > > > However,
> > > > > > > > > > > it
> > > > > > > > > > > > is true that some users may be unknowingly depending
> > on
> > > > > offset
> > > > > > > > > storage
> > > > > > > > > > if
> > > > > > > > > > > > they are using both the default group.id and the
> > default
> > > > > > > > > > > > enable.auto.commit. Perhaps one option is to disable
> > > > > > > > > > enable.auto.commit
> > > > > > > > > > > > automatically if no group.id is specified? I am not
> > sure
> > > > if
> > > > > > > there
> > > > > > > > > are
> > > > > > > > > > > any
> > > > > > > > > > > > drawbacks, but my feeling is that implicit use of the
> > > > empty
> > > > > > > > group.id
> > > > > > > > > > to
> > > > > > > > > > > > commit offsets is more likely to be causing users
> > > > unexpected
> > > > > > > > problems
> > > > > > > > > > > than
> > > > > > > > > > > > actually providing a useful capability.
> > > > > > > > > > > >
> > > > > > > > > > > > Thoughts?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Jason
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, May 28, 2018 at 9:50 AM, Vahid S Hashemian <
> > > > > > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Viktor,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for sharing your opinion.
> > > > > > > > > > > > > So you're in favor of disallowing the empty ("")
> > group
> > > > id
> > > > > > > > > altogether
> > > > > > > > > > > > (even
> > > > > > > > > > > > > for fetching).
> > > > > > > > > > > > > Given that ideally no one should be using the empty
> > > > group
> > > > > id
> > > > > > > (at
> > > > > > > > > > least
> > > > > > > > > > > in
> > > > > > > > > > > > > a production setting) I think the impact would be
> > > > minimal
> > > > > in
> > > > > > > > either
> > > > > > > > > > > case.
> > > > > > > > > > > > >
> > > > > > > > > > > > > But as you said, let's hear what others think and
> > I'd be
> > > > > > happy
> > > > > > > > to
> > > > > > > > > > > modify
> > > > > > > > > > > > > the KIP if needed.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regards.
> > > > > > > > > > > > > --Vahid
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > > > > > > > > > > To: dev <de...@kafka.apache.org>
> > > > > > > > > > > > > Date: 05/28/2018 05:18 AM
> > > > > > > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the
> > > > default
> > > > > > > group
> > > > > > > > id
> > > > > > > > > > > > > behavior in KafkaConsumer
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Vahid,
> > > > > > > > > > > > >
> > > > > > > > > > > > > (with the argument that using the default group id
> > for
> > > > > > offset
> > > > > > > > > commit
> > > > > > > > > > > > > should not be the user's intention in practice).
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yea, so in my opinion too this use case doesn't
> seem
> > too
> > > > > > > > practical.
> > > > > > > > > > > Also
> > > > > > > > > > > > I
> > > > > > > > > > > > > think breaking the offset commit is not smaller
> from
> > > > this
> > > > > > > > > > perspective
> > > > > > > > > > > > than
> > > > > > > > > > > > > breaking fetch and offset fetch. If we suppose that
> > > > someone
> > > > > > > uses
> > > > > > > > > the
> > > > > > > > > > > > > default group id and we break the offset commit
> then
> > > > that
> > > > > > > might
> > > > > > > > be
> > > > > > > > > > > harder
> > > > > > > > > > > > > to detect than breaking the whole thing altogether.
> > (If
> > > > we
> > > > > > > think
> > > > > > > > > > about
> > > > > > > > > > > an
> > > > > > > > > > > > > upgrade situation.)
> > > > > > > > > > > > > So since we think it is not a practical use case, I
> > > > think
> > > > > it
> > > > > > > > would
> > > > > > > > > > be
> > > > > > > > > > > > > better to break altogether but ofc that's just my 2
> > > > cents
> > > > > > :).
> > > > > > > > Let's
> > > > > > > > > > > > gather
> > > > > > > > > > > > > other's input as well.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > Viktor
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, May 25, 2018 at 5:43 PM, Vahid S Hashemian
> <
> > > > > > > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Victor,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for reviewing the KIP.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Yes, to minimize the backward compatibility
> > impact,
> > > > there
> > > > > > > > would
> > > > > > > > > be
> > > > > > > > > > no
> > > > > > > > > > > > > harm
> > > > > > > > > > > > > > in letting a stand-alone consumer consume
> messages
> > > > under
> > > > > a
> > > > > > > ""
> > > > > > > > > > group
> > > > > > > > > > > id
> > > > > > > > > > > > > (as
> > > > > > > > > > > > > > long as there is no offset commit).
> > > > > > > > > > > > > > It would have to knowingly seek to an offset or
> > rely
> > > > on
> > > > > > the
> > > > > > > > > > > > > > auto.offset.reset config for the starting offset.
> > > > > > > > > > > > > > This way the existing functionality would be
> > preserved
> > > > > for
> > > > > > > the
> > > > > > > > > > most
> > > > > > > > > > > > part
> > > > > > > > > > > > > > (with the argument that using the default group
> id
> > for
> > > > > > > offset
> > > > > > > > > > commit
> > > > > > > > > > > > > > should not be the user's intention in practice).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Does it seem reasonable?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks.
> > > > > > > > > > > > > > --Vahid
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > > > > > > > > > > > To: dev <de...@kafka.apache.org>
> > > > > > > > > > > > > > Date: 05/25/2018 04:56 AM
> > > > > > > > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve
> the
> > > > > default
> > > > > > > > group
> > > > > > > > > > id
> > > > > > > > > > > > > > behavior in KafkaConsumer
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Vahid,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > When reading your KIP I coldn't fully understand
> > why
> > > > did
> > > > > > you
> > > > > > > > > > decide
> > > > > > > > > > > at
> > > > > > > > > > > > > > failing with "offset_commit" in case #2? Can't we
> > fail
> > > > > > with
> > > > > > > an
> > > > > > > > > > empty
> > > > > > > > > > > > > group
> > > > > > > > > > > > > > id even in "fetch" or "fetch_offset"? What was
> the
> > > > reason
> > > > > > > for
> > > > > > > > > > > deciding
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > fail at "offset_commit"? Was it because of
> upgrade
> > > > > > > > compatibility
> > > > > > > > > > > > > reasons?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > Viktor
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, May 24, 2018 at 12:06 AM, Ted Yu
> > > > > > > <yu...@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Looks good to me.
> > > > > > > > > > > > > > > -------- Original message --------From: Vahid S
> > > > > > Hashemian
> > > > > > > <
> > > > > > > > > > > > > > > vahidhashemian@us.ibm.com> Date: 5/23/18
> 11:19
> > AM
> > > > > > > > > (GMT-08:00)
> > > > > > > > > > > To:
> > > > > > > > > > > > > > > dev@kafka.apache.org Subject: Re: [DISCUSS]
> > KIP-289:
> > > > > > > Improve
> > > > > > > > > the
> > > > > > > > > > > > > default
> > > > > > > > > > > > > > > group id behavior in KafkaConsumer
> > > > > > > > > > > > > > > Hi Ted,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for reviewing the KIP. I updated the KIP
> > and
> > > > > > > > introduced
> > > > > > > > > > an
> > > > > > > > > > > > > error
> > > > > > > > > > > > > > > code for the scenario described.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > --Vahid
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > From: Ted Yu <yu...@gmail.com>
> > > > > > > > > > > > > > > To: dev@kafka.apache.org
> > > > > > > > > > > > > > > Date: 04/27/2018 04:31 PM
> > > > > > > > > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve
> > the
> > > > > > default
> > > > > > > > > group
> > > > > > > > > > id
> > > > > > > > > > > > > > > behavior in KafkaConsumer
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > bq. If they attempt an offset commit they will
> > > > receive
> > > > > > an
> > > > > > > > > error.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Can you outline what specific error would be
> > > > > encountered
> > > > > > ?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Apr 27, 2018 at 2:17 PM, Vahid S
> > Hashemian <
> > > > > > > > > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I have drafted a proposal for improving the
> > > > behavior
> > > > > > of
> > > > > > > > > > > > > KafkaConsumer
> > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > using the default group id:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >
> > > >
> > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > 289%3A+Improve+the+default+group+id+behavior+in+
> > > > > > > > > KafkaConsumer
> > > > > > > > > > > > > > > > The proposal based on the issue and
> suggestion
> > > > > > reported
> > > > > > > in
> > > > > > > > > > > > > KAFKA-6774.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Your feedback is welcome!
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks.
> > > > > > > > > > > > > > > > --Vahid
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Best,
> > > > > > > > > > Stanislav
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Best,
> > > > > > > > > Stanislav
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> >
> >
> >
> >
> >
> >
>
--
Best,
Stanislav
Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer
Posted by Jason Gustafson <ja...@confluent.io>.
Hey Vahid,
I think we're in agreement. Perhaps since none of us feel too strongly, we
should go for the more minimal change? I'll vote for the KIP either way as
long as we can change the default group.id.
Thanks,
Jason
On Thu, Aug 2, 2018 at 11:55 AM, Vahid S Hashemian <
vahidhashemian@us.ibm.com> wrote:
> Hi Jason, Colin,
>
> Thanks for your feedback.
>
> If we are going to keep supporting the empty group id like before then it
> makes sense to do what you suggested.
>
> Though, IMHO if using the empty group id for committing offsets is a bad
> practice we should stop supporting it at some point; and it would be
> better to do it sooner before potentially more users adopt that.
> I understand that changing the default would reduce the chance of someone
> using it, but it doesn't stop them. Note that we did not receive any
> objections to dropping the support even right-away (a good sign).
> We could keep supporting the old API version in case someone must use this
> group id like before. But at least we won't expose new users to this bad
> practice.
>
> I, too, do now have a strong opinion about this. So if the current KIP
> should not move forward as is, I'm okay with just changing the default and
> deprecating the use on the client, as you suggested.
>
> Thanks.
> --Vahid
>
>
>
> From: Colin McCabe <cm...@apache.org>
> To: dev@kafka.apache.org
> Date: 08/02/2018 11:23 AM
> Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> behavior in KafkaConsumer
>
>
>
> Thanks, Jason. I don't have a very strong opinion on this. But like you
> said, if we skip bumping the RPC versions, this would be a smaller change,
> which might be good.
>
> best,
> Colin
>
>
> On Wed, Aug 1, 2018, at 17:43, Jason Gustafson wrote:
> > Hey Vahid,
> >
> > I talked with Colin offline. I think specifically he felt the version
> bump
> > on the broker was overkill since the broker still has to support the
> empty
> > group id for older versions. I had thought that eventually we would be
> able
> > to remove those old versions, but it's true that this may not happen
> until
> > indefinitely far in the future. I think the main improvement here is
> > changing the default group.id to null instead of "". I could go either
> way
> > on whether bumping the protocol is useful. I do think it is helpful
> though
> > to signal clearly that it its use is deprecated and discouraged,
> especially
> > in light of the ACL problem. I guess we could just deprecate the use on
> the
> > client. What do you think?
> >
> > Thanks,
> > Jason
> >
> > On Wed, Aug 1, 2018 at 3:19 PM, Vahid S Hashemian
> <vahidhashemian@us.ibm.com
> > > wrote:
> >
> > > Thanks Jason for responding to Colin's concerns.
> > >
> > > If there are no other comment / feedback / objection I'll start a vote
> > > soon.
> > >
> > > Thanks.
> > > --Vahid
> > >
> > >
> > >
> > > From: Jason Gustafson <ja...@confluent.io>
> > > To: dev <de...@kafka.apache.org>
> > > Date: 07/27/2018 10:38 AM
> > > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > > behavior in KafkaConsumer
> > >
> > >
> > >
> > > Hey Colin,
> > >
> > > The problem is both that the empty group id is the default value and
> that
> > > it is actually accepted by the broker for offset commits. Combine that
> > > with
> > > the fact that auto commit is enabled by default and you users get
> > > surprising behavior. If you look at a random Kafka cluster, you'll
> > > probably
> > > find a bunch of inadvertent offset commits for the empty group id. I
> was
> > > hoping we could distinguish between users who are using the empty
> group id
> > > as an accident of the default configuration and those who use it
> > > intentionally. By default, there will be no group id and the consumer
> will
> > > not commit offsets. If a user has actually intentionally used the
> empty
> > > group id, however, it will continue to work. I actually think there
> are
> > > probably very few people doing this (maybe even no one), but I thought
> we
> > > might err on the side of compatibility.
> > >
> > > The big incompatible change here is having brokers reject using
> > > assign(...)
> > > > with empty / null group.id.
> > >
> > >
> > > This is not correct. In the proposal, the broker will only reject the
> > > empty
> > > group id for the new version of OffsetCommit. Older clients, which
> cannot
> > > be changed, will continue to work because the old versions of the
> > > OffsetCommit API still accept the empty group id. The null group id is
> > > different from the empty group id: it is not allowed in any version of
> the
> > > API. It is basically a way to indicate that the consumer has no
> dependence
> > > on the coordinator at all, which we actually have a surprising number
> of
> > > use cases for. Furthermore, if a user has an actual need for the empty
> > > group id, it will still be allowed. We are just deprecating it.
> > >
> > > -Jason
> > >
> > > On Fri, Jul 27, 2018 at 9:56 AM, Colin McCabe <cm...@apache.org>
> wrote:
> > >
> > > > Sorry if this is a silly question, but what's the rationale for
> > > switching
> > > > to using null for the default group id, rather than the empty
> string?
> > > > Continuing to use the empty string seems like less churn. And after
> > > all,
> > > > we're not using the empty string group name for anything else.
> > > >
> > > > The big incompatible change here is having brokers reject using
> > > > assign(...) with empty / null group.id. If I understand correctly,
> the
> > > > KIP proposes that this change be made on the brokers on the next
> > > > incompatible Kafka release. But that has nothing to do with client
> > > > versions. Why not just have a broker config which controls this?
> Maybe
> > > "
> > > > allow.assign.empty.group.id", or something like that. At first, the
> > > > default will be true, and then eventually we can flip it over to
> false.
> > > >
> > > > It seems like the main rationale for tying this behavior to the
> Kafka
> > > > client version is to force people to stop using the empty group id
> so
> > > that
> > > > they can upgrade their clients. But it's also possible that people
> will
> > > > stop upgrading their Kafka clients instead. That would be pretty
> > > negative
> > > > since they'd miss out on any efficiency and feature improvements in
> the
> > > > new clients and eventually have to do more protocol downgrading,
> etc.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > On Thu, Jul 26, 2018, at 11:50, Vahid S Hashemian wrote:
> > > > > Hi Jason,
> > > > >
> > > > > That makes sense.
> > > > > I have updated the KIP based on the recent feedback.
> > > > >
> > > > > Thanks!
> > > > > --Vahid
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > From: Jason Gustafson <ja...@confluent.io>
> > > > > To: dev <de...@kafka.apache.org>
> > > > > Date: 07/25/2018 02:23 PM
> > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group
> id
> > > > > behavior in KafkaConsumer
> > > > >
> > > > >
> > > > >
> > > > > Hi Vahid,
> > > > >
> > > > > I was thinking we'd only use the old API version if we had to.
> That
> > > is,
> > > > > only if the user has explicitly configured "" as the group.id.
> > > > Otherwise,
> > > > > we'd just use the new one. Another option is to just drop support
> in
> > > the
> > > > > client for the empty group id, but usually we allow a deprecation
> > > period
> > > > > for changes like this.
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Wed, Jul 25, 2018 at 12:49 PM, Vahid S Hashemian <
> > > > > vahidhashemian@us.ibm.com> wrote:
> > > > >
> > > > > > Hi Jason,
> > > > > >
> > > > > > Thanks for additional clarification.
> > > > > >
> > > > > > So the next version of the OffsetCommit API will return an
> > > > > > INVALID_GROUP_ID error for empty group ids; but on the client
> side
> > > we
> > > > > call
> > > > > > the older version of the client until the next major release.
> > > > > > The table below should summarize this.
> > > > > >
> > > > > > +-----------------------------------------------------+
> > > > > > | Client (group.id="") |
> > > > > > +-----------------------------------------------------+
> > > > > > | pre-2.1 | 2.1 | 3.0
> |
> > > > > >
> > > > > +-----+-----------+---------+------------------------+------
> > > > ------------+
> > > > > > | | V5 (cur.) | works | works | works |
> > > > > > + API
> > > > >
> +-----------+---------+------------------------+------------------+
> > > > > > | | V6 | N/A | N/A (calls V5/warning) |
> > > > INVALID_GROUP_ID
> > > > > |
> > > > > >
> > > > > +-----+-----------+---------+------------------------+------
> > > > ------------+
> > > > > >
> > > > > > Assumptions:
> > > > > > * 2.1: The target release version for this KIP
> > > > > > * 3.0: The next major release
> > > > > >
> > > > > > Please advise if you see an issue; otherwise, I'll update the
> KIP
> > > > > > accordingly.
> > > > > >
> > > > > > Thanks!
> > > > > > --Vahid
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > From: Jason Gustafson <ja...@confluent.io>
> > > > > > To: dev <de...@kafka.apache.org>
> > > > > > Date: 07/25/2018 12:08 AM
> > > > > > Subject: ***UNCHECKED*** Re: [DISCUSS] KIP-289: Improve
> the
> > > > > default
> > > > > > group id behavior in KafkaConsumer
> > > > > >
> > > > > >
> > > > > >
> > > > > > Hey Vahid,
> > > > > >
> > > > > > Sorry for the confusion. I think we all agree that going
> forward, we
> > > > > > shouldn't support the empty group id, so the question is just
> around
> > > > > > compatibility. I think we have to bump the OffsetCommit API
> version
> > > so
> > > > > > that
> > > > > > old clients which are unknowingly depending on the default empty
> > > group
> > > > > id
> > > > > > will continue to work with new brokers. For new versions of the
> > > > client,
> > > > > we
> > > > > > can either drop support for the empty group id immediately or we
> can
> > > > > give
> > > > > > users a grace period. I was thinking we would do the latter. We
> can
> > > > > change
> > > > > > the default group.id, but in the case that a user has explicitly
> > > > > > configured
> > > > > > the empty group, then we can just use an old version of the
> > > > OffsetCommit
> > > > > > API which still supports it. In a future release, we can drop
> this
> > > > > support
> > > > > > and only use the latest OffsetCommit version. Does that make
> sense?
> > > > > >
> > > > > > Thanks,
> > > > > > Jason
> > > > > >
> > > > > >
> > > > > > On Tue, Jul 24, 2018 at 12:36 PM, Vahid S Hashemian <
> > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > >
> > > > > > > Hi Jason,
> > > > > > >
> > > > > > > Thanks for clarifying.
> > > > > > >
> > > > > > > So if we are going to continue supporting the empty group id
> as
> > > > before
> > > > > > > (with only an addition of a deprecation warning), and disable
> > > > > > > enable.auto.commit for the new default (null) group id on the
> > > client
> > > > > > side,
> > > > > > > do we really need to bump up the OffsetCommit version?
> > > > > > >
> > > > > > > You mentioned "If an explicit empty string is configured for
> the
> > > > group
> > > > > > id,
> > > > > > > then maybe we keep the current behavior for compatibility"
> which
> > > > makes
> > > > > > > sense to me, but I find it in conflict with your earlier
> > > suggestion
> > > > > "we
> > > > > > > just need to bump the OffsetCommit request API and only accept
> the
> > > > > > offset
> > > > > > > commit for older versions.". Maybe I'm missing something?
> > > > > > >
> > > > > > > Thanks!
> > > > > > > --Vahid
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > From: Jason Gustafson <ja...@confluent.io>
> > > > > > > To: dev <de...@kafka.apache.org>
> > > > > > > Date: 07/23/2018 10:52 PM
> > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default
> group
> > > id
> > > > > > > behavior in KafkaConsumer
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Hey Vahid,
> > > > > > >
> > > > > > > Thanks for the updates. Just to clarify, I was suggesting that
> we
> > > > > > disable
> > > > > > > enable.auto.commit only if no explicit group.id is
> configured.
> If
> > > an
> > > > > > > explicit empty string is configured for the group id, then
> maybe
> > > we
> > > > > keep
> > > > > > > the current behavior for compatibility. We can log a warning
> > > > > mentioning
> > > > > > > the
> > > > > > > deprecation and we can use the old version of the OffsetCommit
> API
> > > > > that
> > > > > > > allows the empty group id. In a later release, we can drop
> this
> > > > > support
> > > > > > in
> > > > > > > the client. Does that seem reasonable?
> > > > > > >
> > > > > > > By the way, instead of using the new ILLEGAL_OFFSET_COMMIT
> error
> > > > code,
> > > > > > > couldn't we use INVALID_GROUP_ID?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Jason
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jul 23, 2018 at 5:14 PM, Stanislav Kozlovski
> > > > > > > <stanislav@confluent.io
> > > > > > > > wrote:
> > > > > > >
> > > > > > > > Hey Vahid,
> > > > > > > >
> > > > > > > > No I don't see an issue with it. I believe it to be the best
> > > > > approach.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Stanisav
> > > > > > > >
> > > > > > > > On Mon, Jul 23, 2018 at 12:41 PM Vahid S Hashemian <
> > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > >
> > > > > > > > > Hi Stanislav,
> > > > > > > > >
> > > > > > > > > Thanks for the feedback.
> > > > > > > > > Do you see an issue with using `null` as the default group
> id
> > > (as
> > > > > > > > > addressed by Jason in his response)?
> > > > > > > > > This default group id would not support offset commits and
> > > > > consumers
> > > > > > > > would
> > > > > > > > > use `auto.offset.reset` config when there is no current
> > > offset.
> > > > > > > > >
> > > > > > > > > Thanks.
> > > > > > > > > --Vahid
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > From: Stanislav Kozlovski <st...@confluent.io>
> > > > > > > > > To: dev@kafka.apache.org
> > > > > > > > > Date: 07/20/2018 11:09 AM
> > > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default
> > > group
> > > > > id
> > > > > > > > > behavior in KafkaConsumer
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I agree with Jason's notion that
> > > > > > > > > > implicit use of the empty group.id to commit offsets is
> > > more
> > > > > > likely
> > > > > > > > to
> > > > > > > > > be causing users unexpected problems than actually
> providing a
> > > > > > useful
> > > > > > > > > capability.
> > > > > > > > > I was initially confused that this is the behavior when
> > > > > > investigating
> > > > > > > a
> > > > > > > > > new-ish JIRA issue <
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> https://issues.apache.org/jira/browse/KAFKA-6758
>
> > >
> > > > >
> > > > > >
> > > > > > >
> > > > > > > > > > about
> > > > > > > > > the same topic.
> > > > > > > > > So, +1 to deprecating "" as a group.id
> > > > > > > > >
> > > > > > > > > The question after that becomes what the *default* value
> > > should
> > > > be
> > > > > -
> > > > > > > > > should
> > > > > > > > > we:
> > > > > > > > > a) treat an unconfigured group.id consumer as a sort of
> > > > > intermittent
> > > > > > > > > consumer where you don't store offsets at all (thereby
> making
> > > the
> > > > > > user
> > > > > > > > > explicitly sign up for them)
> > > > > > > > > b) have a default value which makes use of them? I sort of
> > > like
> > > > > the
> > > > > > > > > former.
> > > > > > > > >
> > > > > > > > > @Dhruvil, thinking about it at a high-level - yes. I can't
> > > think
> > > > > of
> > > > > > a
> > > > > > > > > situation where it makes sense to name something an empty
> > > string
> > > > > as
> > > > > > > far
> > > > > > > > as
> > > > > > > > > I'm aware - to me it seems like potential for confusion
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Jul 20, 2018 at 10:22 AM Rajini Sivaram
> > > > > > > <rajinisivaram@gmail.com
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > +1 to deprecate use of "" as group.id since it is odd to
> > > have
> > > > a
> > > > > > > > resource
> > > > > > > > > > name that you cannot set ACLs for. Agree, we have to
> support
> > > > > older
> > > > > > > > > clients
> > > > > > > > > > though.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Rajini
> > > > > > > > > >
> > > > > > > > > > On Fri, Jul 20, 2018 at 5:25 PM, Jason Gustafson
> > > > > > > <ja...@confluent.io>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Vahid,
> > > > > > > > > > >
> > > > > > > > > > > Sorry for getting to this so late. I think there are
> two
> > > > > things
> > > > > > > here:
> > > > > > > > > > >
> > > > > > > > > > > 1. The use of "" as a groupId has always been a
> dubious
> > > > > practice
> > > > > > > at
> > > > > > > > > best.
> > > > > > > > > > > We definitely ought to deprecate its use in the
> client.
> > > > > Perhaps
> > > > > > in
> > > > > > > > the
> > > > > > > > > > next
> > > > > > > > > > > major release, we can remove support completely.
> However,
> > > > > since
> > > > > > > older
> > > > > > > > > > > clients depend on it, we may have to continue letting
> the
> > > > > broker
> > > > > > > > > support
> > > > > > > > > > it
> > > > > > > > > > > to some extent. Perhaps we just need to bump the
> > > OffsetCommit
> > > > > > > request
> > > > > > > > > API
> > > > > > > > > > > and only accept the offset commit for older versions.
> You
> > > > > > probably
> > > > > > > > > have
> > > > > > > > > > to
> > > > > > > > > > > do this anyway if you want to introduce the new error
> code
> > > > > since
> > > > > > > old
> > > > > > > > > > > clients will not expect it.
> > > > > > > > > > >
> > > > > > > > > > > 2. There should be a way for the consumer to indicate
> that
> > > it
> > > > > > has
> > > > > > > no
> > > > > > > > > > group
> > > > > > > > > > > id and will not commit offsets. This is an explicit
> > > > > instruction
> > > > > > > that
> > > > > > > > > the
> > > > > > > > > > > consumer should not bother with coordinator lookup and
> > > such.
> > > > > We
> > > > > > > > > currently
> > > > > > > > > > > have some brittle logic in place to let users avoid
> the
> > > > > > > coordinator
> > > > > > > > > > lookup,
> > > > > > > > > > > but it is a bit error-prone. I was hoping that we
> could
> > > > change
> > > > > > the
> > > > > > > > > > default
> > > > > > > > > > > value of group.id to be null so that the user had to
> take
> > > an
> > > > > > > > explicit
> > > > > > > > > > > action to opt into coordinator management (groups or
> > > > offsets).
> > > > > > > > > However,
> > > > > > > > > > it
> > > > > > > > > > > is true that some users may be unknowingly depending
> on
> > > > offset
> > > > > > > > storage
> > > > > > > > > if
> > > > > > > > > > > they are using both the default group.id and the
> default
> > > > > > > > > > > enable.auto.commit. Perhaps one option is to disable
> > > > > > > > > enable.auto.commit
> > > > > > > > > > > automatically if no group.id is specified? I am not
> sure
> > > if
> > > > > > there
> > > > > > > > are
> > > > > > > > > > any
> > > > > > > > > > > drawbacks, but my feeling is that implicit use of the
> > > empty
> > > > > > > group.id
> > > > > > > > > to
> > > > > > > > > > > commit offsets is more likely to be causing users
> > > unexpected
> > > > > > > problems
> > > > > > > > > > than
> > > > > > > > > > > actually providing a useful capability.
> > > > > > > > > > >
> > > > > > > > > > > Thoughts?
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Jason
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Mon, May 28, 2018 at 9:50 AM, Vahid S Hashemian <
> > > > > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Viktor,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for sharing your opinion.
> > > > > > > > > > > > So you're in favor of disallowing the empty ("")
> group
> > > id
> > > > > > > > altogether
> > > > > > > > > > > (even
> > > > > > > > > > > > for fetching).
> > > > > > > > > > > > Given that ideally no one should be using the empty
> > > group
> > > > id
> > > > > > (at
> > > > > > > > > least
> > > > > > > > > > in
> > > > > > > > > > > > a production setting) I think the impact would be
> > > minimal
> > > > in
> > > > > > > either
> > > > > > > > > > case.
> > > > > > > > > > > >
> > > > > > > > > > > > But as you said, let's hear what others think and
> I'd be
> > > > > happy
> > > > > > > to
> > > > > > > > > > modify
> > > > > > > > > > > > the KIP if needed.
> > > > > > > > > > > >
> > > > > > > > > > > > Regards.
> > > > > > > > > > > > --Vahid
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > > > > > > > > > To: dev <de...@kafka.apache.org>
> > > > > > > > > > > > Date: 05/28/2018 05:18 AM
> > > > > > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the
> > > default
> > > > > > group
> > > > > > > id
> > > > > > > > > > > > behavior in KafkaConsumer
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Vahid,
> > > > > > > > > > > >
> > > > > > > > > > > > (with the argument that using the default group id
> for
> > > > > offset
> > > > > > > > commit
> > > > > > > > > > > > should not be the user's intention in practice).
> > > > > > > > > > > >
> > > > > > > > > > > > Yea, so in my opinion too this use case doesn't seem
> too
> > > > > > > practical.
> > > > > > > > > > Also
> > > > > > > > > > > I
> > > > > > > > > > > > think breaking the offset commit is not smaller from
> > > this
> > > > > > > > > perspective
> > > > > > > > > > > than
> > > > > > > > > > > > breaking fetch and offset fetch. If we suppose that
> > > someone
> > > > > > uses
> > > > > > > > the
> > > > > > > > > > > > default group id and we break the offset commit then
> > > that
> > > > > > might
> > > > > > > be
> > > > > > > > > > harder
> > > > > > > > > > > > to detect than breaking the whole thing altogether.
> (If
> > > we
> > > > > > think
> > > > > > > > > about
> > > > > > > > > > an
> > > > > > > > > > > > upgrade situation.)
> > > > > > > > > > > > So since we think it is not a practical use case, I
> > > think
> > > > it
> > > > > > > would
> > > > > > > > > be
> > > > > > > > > > > > better to break altogether but ofc that's just my 2
> > > cents
> > > > > :).
> > > > > > > Let's
> > > > > > > > > > > gather
> > > > > > > > > > > > other's input as well.
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers,
> > > > > > > > > > > > Viktor
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, May 25, 2018 at 5:43 PM, Vahid S Hashemian <
> > > > > > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Victor,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for reviewing the KIP.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yes, to minimize the backward compatibility
> impact,
> > > there
> > > > > > > would
> > > > > > > > be
> > > > > > > > > no
> > > > > > > > > > > > harm
> > > > > > > > > > > > > in letting a stand-alone consumer consume messages
> > > under
> > > > a
> > > > > > ""
> > > > > > > > > group
> > > > > > > > > > id
> > > > > > > > > > > > (as
> > > > > > > > > > > > > long as there is no offset commit).
> > > > > > > > > > > > > It would have to knowingly seek to an offset or
> rely
> > > on
> > > > > the
> > > > > > > > > > > > > auto.offset.reset config for the starting offset.
> > > > > > > > > > > > > This way the existing functionality would be
> preserved
> > > > for
> > > > > > the
> > > > > > > > > most
> > > > > > > > > > > part
> > > > > > > > > > > > > (with the argument that using the default group id
> for
> > > > > > offset
> > > > > > > > > commit
> > > > > > > > > > > > > should not be the user's intention in practice).
> > > > > > > > > > > > >
> > > > > > > > > > > > > Does it seem reasonable?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks.
> > > > > > > > > > > > > --Vahid
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > > > > > > > > > > To: dev <de...@kafka.apache.org>
> > > > > > > > > > > > > Date: 05/25/2018 04:56 AM
> > > > > > > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the
> > > > default
> > > > > > > group
> > > > > > > > > id
> > > > > > > > > > > > > behavior in KafkaConsumer
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Vahid,
> > > > > > > > > > > > >
> > > > > > > > > > > > > When reading your KIP I coldn't fully understand
> why
> > > did
> > > > > you
> > > > > > > > > decide
> > > > > > > > > > at
> > > > > > > > > > > > > failing with "offset_commit" in case #2? Can't we
> fail
> > > > > with
> > > > > > an
> > > > > > > > > empty
> > > > > > > > > > > > group
> > > > > > > > > > > > > id even in "fetch" or "fetch_offset"? What was the
> > > reason
> > > > > > for
> > > > > > > > > > deciding
> > > > > > > > > > > > to
> > > > > > > > > > > > > fail at "offset_commit"? Was it because of upgrade
> > > > > > > compatibility
> > > > > > > > > > > > reasons?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Viktor
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, May 24, 2018 at 12:06 AM, Ted Yu
> > > > > > <yu...@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Looks good to me.
> > > > > > > > > > > > > > -------- Original message --------From: Vahid S
> > > > > Hashemian
> > > > > > <
> > > > > > > > > > > > > > vahidhashemian@us.ibm.com> Date: 5/23/18 11:19
> AM
> > > > > > > > (GMT-08:00)
> > > > > > > > > > To:
> > > > > > > > > > > > > > dev@kafka.apache.org Subject: Re: [DISCUSS]
> KIP-289:
> > > > > > Improve
> > > > > > > > the
> > > > > > > > > > > > default
> > > > > > > > > > > > > > group id behavior in KafkaConsumer
> > > > > > > > > > > > > > Hi Ted,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for reviewing the KIP. I updated the KIP
> and
> > > > > > > introduced
> > > > > > > > > an
> > > > > > > > > > > > error
> > > > > > > > > > > > > > code for the scenario described.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --Vahid
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > From: Ted Yu <yu...@gmail.com>
> > > > > > > > > > > > > > To: dev@kafka.apache.org
> > > > > > > > > > > > > > Date: 04/27/2018 04:31 PM
> > > > > > > > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve
> the
> > > > > default
> > > > > > > > group
> > > > > > > > > id
> > > > > > > > > > > > > > behavior in KafkaConsumer
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > bq. If they attempt an offset commit they will
> > > receive
> > > > > an
> > > > > > > > error.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Can you outline what specific error would be
> > > > encountered
> > > > > ?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Apr 27, 2018 at 2:17 PM, Vahid S
> Hashemian <
> > > > > > > > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I have drafted a proposal for improving the
> > > behavior
> > > > > of
> > > > > > > > > > > > KafkaConsumer
> > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > using the default group id:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>
> > >
> > > > >
> > > > > >
> > > > > > >
> > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> 289%3A+Improve+the+default+group+id+behavior+in+
> > > > > > > > KafkaConsumer
> > > > > > > > > > > > > > > The proposal based on the issue and suggestion
> > > > > reported
> > > > > > in
> > > > > > > > > > > > KAFKA-6774.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Your feedback is welcome!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks.
> > > > > > > > > > > > > > > --Vahid
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Best,
> > > > > > > > > Stanislav
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Best,
> > > > > > > > Stanislav
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > >
> > >
>
>
>
>
>
>
Re: [DISCUSS] KIP-289: Improve the default group id behavior in
KafkaConsumer
Posted by Vahid S Hashemian <va...@us.ibm.com>.
Hi Jason, Colin,
Thanks for your feedback.
If we are going to keep supporting the empty group id like before then it
makes sense to do what you suggested.
Though, IMHO if using the empty group id for committing offsets is a bad
practice we should stop supporting it at some point; and it would be
better to do it sooner before potentially more users adopt that.
I understand that changing the default would reduce the chance of someone
using it, but it doesn't stop them. Note that we did not receive any
objections to dropping the support even right-away (a good sign).
We could keep supporting the old API version in case someone must use this
group id like before. But at least we won't expose new users to this bad
practice.
I, too, do now have a strong opinion about this. So if the current KIP
should not move forward as is, I'm okay with just changing the default and
deprecating the use on the client, as you suggested.
Thanks.
--Vahid
From: Colin McCabe <cm...@apache.org>
To: dev@kafka.apache.org
Date: 08/02/2018 11:23 AM
Subject: Re: [DISCUSS] KIP-289: Improve the default group id
behavior in KafkaConsumer
Thanks, Jason. I don't have a very strong opinion on this. But like you
said, if we skip bumping the RPC versions, this would be a smaller change,
which might be good.
best,
Colin
On Wed, Aug 1, 2018, at 17:43, Jason Gustafson wrote:
> Hey Vahid,
>
> I talked with Colin offline. I think specifically he felt the version
bump
> on the broker was overkill since the broker still has to support the
empty
> group id for older versions. I had thought that eventually we would be
able
> to remove those old versions, but it's true that this may not happen
until
> indefinitely far in the future. I think the main improvement here is
> changing the default group.id to null instead of "". I could go either
way
> on whether bumping the protocol is useful. I do think it is helpful
though
> to signal clearly that it its use is deprecated and discouraged,
especially
> in light of the ACL problem. I guess we could just deprecate the use on
the
> client. What do you think?
>
> Thanks,
> Jason
>
> On Wed, Aug 1, 2018 at 3:19 PM, Vahid S Hashemian
<vahidhashemian@us.ibm.com
> > wrote:
>
> > Thanks Jason for responding to Colin's concerns.
> >
> > If there are no other comment / feedback / objection I'll start a vote
> > soon.
> >
> > Thanks.
> > --Vahid
> >
> >
> >
> > From: Jason Gustafson <ja...@confluent.io>
> > To: dev <de...@kafka.apache.org>
> > Date: 07/27/2018 10:38 AM
> > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > behavior in KafkaConsumer
> >
> >
> >
> > Hey Colin,
> >
> > The problem is both that the empty group id is the default value and
that
> > it is actually accepted by the broker for offset commits. Combine that
> > with
> > the fact that auto commit is enabled by default and you users get
> > surprising behavior. If you look at a random Kafka cluster, you'll
> > probably
> > find a bunch of inadvertent offset commits for the empty group id. I
was
> > hoping we could distinguish between users who are using the empty
group id
> > as an accident of the default configuration and those who use it
> > intentionally. By default, there will be no group id and the consumer
will
> > not commit offsets. If a user has actually intentionally used the
empty
> > group id, however, it will continue to work. I actually think there
are
> > probably very few people doing this (maybe even no one), but I thought
we
> > might err on the side of compatibility.
> >
> > The big incompatible change here is having brokers reject using
> > assign(...)
> > > with empty / null group.id.
> >
> >
> > This is not correct. In the proposal, the broker will only reject the
> > empty
> > group id for the new version of OffsetCommit. Older clients, which
cannot
> > be changed, will continue to work because the old versions of the
> > OffsetCommit API still accept the empty group id. The null group id is
> > different from the empty group id: it is not allowed in any version of
the
> > API. It is basically a way to indicate that the consumer has no
dependence
> > on the coordinator at all, which we actually have a surprising number
of
> > use cases for. Furthermore, if a user has an actual need for the empty
> > group id, it will still be allowed. We are just deprecating it.
> >
> > -Jason
> >
> > On Fri, Jul 27, 2018 at 9:56 AM, Colin McCabe <cm...@apache.org>
wrote:
> >
> > > Sorry if this is a silly question, but what's the rationale for
> > switching
> > > to using null for the default group id, rather than the empty
string?
> > > Continuing to use the empty string seems like less churn. And after
> > all,
> > > we're not using the empty string group name for anything else.
> > >
> > > The big incompatible change here is having brokers reject using
> > > assign(...) with empty / null group.id. If I understand correctly,
the
> > > KIP proposes that this change be made on the brokers on the next
> > > incompatible Kafka release. But that has nothing to do with client
> > > versions. Why not just have a broker config which controls this?
Maybe
> > "
> > > allow.assign.empty.group.id", or something like that. At first, the
> > > default will be true, and then eventually we can flip it over to
false.
> > >
> > > It seems like the main rationale for tying this behavior to the
Kafka
> > > client version is to force people to stop using the empty group id
so
> > that
> > > they can upgrade their clients. But it's also possible that people
will
> > > stop upgrading their Kafka clients instead. That would be pretty
> > negative
> > > since they'd miss out on any efficiency and feature improvements in
the
> > > new clients and eventually have to do more protocol downgrading,
etc.
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Thu, Jul 26, 2018, at 11:50, Vahid S Hashemian wrote:
> > > > Hi Jason,
> > > >
> > > > That makes sense.
> > > > I have updated the KIP based on the recent feedback.
> > > >
> > > > Thanks!
> > > > --Vahid
> > > >
> > > >
> > > >
> > > >
> > > > From: Jason Gustafson <ja...@confluent.io>
> > > > To: dev <de...@kafka.apache.org>
> > > > Date: 07/25/2018 02:23 PM
> > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group
id
> > > > behavior in KafkaConsumer
> > > >
> > > >
> > > >
> > > > Hi Vahid,
> > > >
> > > > I was thinking we'd only use the old API version if we had to.
That
> > is,
> > > > only if the user has explicitly configured "" as the group.id.
> > > Otherwise,
> > > > we'd just use the new one. Another option is to just drop support
in
> > the
> > > > client for the empty group id, but usually we allow a deprecation
> > period
> > > > for changes like this.
> > > >
> > > > -Jason
> > > >
> > > > On Wed, Jul 25, 2018 at 12:49 PM, Vahid S Hashemian <
> > > > vahidhashemian@us.ibm.com> wrote:
> > > >
> > > > > Hi Jason,
> > > > >
> > > > > Thanks for additional clarification.
> > > > >
> > > > > So the next version of the OffsetCommit API will return an
> > > > > INVALID_GROUP_ID error for empty group ids; but on the client
side
> > we
> > > > call
> > > > > the older version of the client until the next major release.
> > > > > The table below should summarize this.
> > > > >
> > > > > +-----------------------------------------------------+
> > > > > | Client (group.id="") |
> > > > > +-----------------------------------------------------+
> > > > > | pre-2.1 | 2.1 | 3.0
|
> > > > >
> > > > +-----+-----------+---------+------------------------+------
> > > ------------+
> > > > > | | V5 (cur.) | works | works | works |
> > > > > + API
> > > >
+-----------+---------+------------------------+------------------+
> > > > > | | V6 | N/A | N/A (calls V5/warning) |
> > > INVALID_GROUP_ID
> > > > |
> > > > >
> > > > +-----+-----------+---------+------------------------+------
> > > ------------+
> > > > >
> > > > > Assumptions:
> > > > > * 2.1: The target release version for this KIP
> > > > > * 3.0: The next major release
> > > > >
> > > > > Please advise if you see an issue; otherwise, I'll update the
KIP
> > > > > accordingly.
> > > > >
> > > > > Thanks!
> > > > > --Vahid
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > From: Jason Gustafson <ja...@confluent.io>
> > > > > To: dev <de...@kafka.apache.org>
> > > > > Date: 07/25/2018 12:08 AM
> > > > > Subject: ***UNCHECKED*** Re: [DISCUSS] KIP-289: Improve
the
> > > > default
> > > > > group id behavior in KafkaConsumer
> > > > >
> > > > >
> > > > >
> > > > > Hey Vahid,
> > > > >
> > > > > Sorry for the confusion. I think we all agree that going
forward, we
> > > > > shouldn't support the empty group id, so the question is just
around
> > > > > compatibility. I think we have to bump the OffsetCommit API
version
> > so
> > > > > that
> > > > > old clients which are unknowingly depending on the default empty
> > group
> > > > id
> > > > > will continue to work with new brokers. For new versions of the
> > > client,
> > > > we
> > > > > can either drop support for the empty group id immediately or we
can
> > > > give
> > > > > users a grace period. I was thinking we would do the latter. We
can
> > > > change
> > > > > the default group.id, but in the case that a user has explicitly
> > > > > configured
> > > > > the empty group, then we can just use an old version of the
> > > OffsetCommit
> > > > > API which still supports it. In a future release, we can drop
this
> > > > support
> > > > > and only use the latest OffsetCommit version. Does that make
sense?
> > > > >
> > > > > Thanks,
> > > > > Jason
> > > > >
> > > > >
> > > > > On Tue, Jul 24, 2018 at 12:36 PM, Vahid S Hashemian <
> > > > > vahidhashemian@us.ibm.com> wrote:
> > > > >
> > > > > > Hi Jason,
> > > > > >
> > > > > > Thanks for clarifying.
> > > > > >
> > > > > > So if we are going to continue supporting the empty group id
as
> > > before
> > > > > > (with only an addition of a deprecation warning), and disable
> > > > > > enable.auto.commit for the new default (null) group id on the
> > client
> > > > > side,
> > > > > > do we really need to bump up the OffsetCommit version?
> > > > > >
> > > > > > You mentioned "If an explicit empty string is configured for
the
> > > group
> > > > > id,
> > > > > > then maybe we keep the current behavior for compatibility"
which
> > > makes
> > > > > > sense to me, but I find it in conflict with your earlier
> > suggestion
> > > > "we
> > > > > > just need to bump the OffsetCommit request API and only accept
the
> > > > > offset
> > > > > > commit for older versions.". Maybe I'm missing something?
> > > > > >
> > > > > > Thanks!
> > > > > > --Vahid
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > From: Jason Gustafson <ja...@confluent.io>
> > > > > > To: dev <de...@kafka.apache.org>
> > > > > > Date: 07/23/2018 10:52 PM
> > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default
group
> > id
> > > > > > behavior in KafkaConsumer
> > > > > >
> > > > > >
> > > > > >
> > > > > > Hey Vahid,
> > > > > >
> > > > > > Thanks for the updates. Just to clarify, I was suggesting that
we
> > > > > disable
> > > > > > enable.auto.commit only if no explicit group.id is configured.
If
> > an
> > > > > > explicit empty string is configured for the group id, then
maybe
> > we
> > > > keep
> > > > > > the current behavior for compatibility. We can log a warning
> > > > mentioning
> > > > > > the
> > > > > > deprecation and we can use the old version of the OffsetCommit
API
> > > > that
> > > > > > allows the empty group id. In a later release, we can drop
this
> > > > support
> > > > > in
> > > > > > the client. Does that seem reasonable?
> > > > > >
> > > > > > By the way, instead of using the new ILLEGAL_OFFSET_COMMIT
error
> > > code,
> > > > > > couldn't we use INVALID_GROUP_ID?
> > > > > >
> > > > > > Thanks,
> > > > > > Jason
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Jul 23, 2018 at 5:14 PM, Stanislav Kozlovski
> > > > > > <stanislav@confluent.io
> > > > > > > wrote:
> > > > > >
> > > > > > > Hey Vahid,
> > > > > > >
> > > > > > > No I don't see an issue with it. I believe it to be the best
> > > > approach.
> > > > > > >
> > > > > > > Best,
> > > > > > > Stanisav
> > > > > > >
> > > > > > > On Mon, Jul 23, 2018 at 12:41 PM Vahid S Hashemian <
> > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > >
> > > > > > > > Hi Stanislav,
> > > > > > > >
> > > > > > > > Thanks for the feedback.
> > > > > > > > Do you see an issue with using `null` as the default group
id
> > (as
> > > > > > > > addressed by Jason in his response)?
> > > > > > > > This default group id would not support offset commits and
> > > > consumers
> > > > > > > would
> > > > > > > > use `auto.offset.reset` config when there is no current
> > offset.
> > > > > > > >
> > > > > > > > Thanks.
> > > > > > > > --Vahid
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > From: Stanislav Kozlovski <st...@confluent.io>
> > > > > > > > To: dev@kafka.apache.org
> > > > > > > > Date: 07/20/2018 11:09 AM
> > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default
> > group
> > > > id
> > > > > > > > behavior in KafkaConsumer
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > I agree with Jason's notion that
> > > > > > > > > implicit use of the empty group.id to commit offsets is
> > more
> > > > > likely
> > > > > > > to
> > > > > > > > be causing users unexpected problems than actually
providing a
> > > > > useful
> > > > > > > > capability.
> > > > > > > > I was initially confused that this is the behavior when
> > > > > investigating
> > > > > > a
> > > > > > > > new-ish JIRA issue <
> > > > > > > >
> > > > > >
> > > > >
> > > >
> >
https://issues.apache.org/jira/browse/KAFKA-6758
> >
> > > >
> > > > >
> > > > > >
> > > > > > > > > about
> > > > > > > > the same topic.
> > > > > > > > So, +1 to deprecating "" as a group.id
> > > > > > > >
> > > > > > > > The question after that becomes what the *default* value
> > should
> > > be
> > > > -
> > > > > > > > should
> > > > > > > > we:
> > > > > > > > a) treat an unconfigured group.id consumer as a sort of
> > > > intermittent
> > > > > > > > consumer where you don't store offsets at all (thereby
making
> > the
> > > > > user
> > > > > > > > explicitly sign up for them)
> > > > > > > > b) have a default value which makes use of them? I sort of
> > like
> > > > the
> > > > > > > > former.
> > > > > > > >
> > > > > > > > @Dhruvil, thinking about it at a high-level - yes. I can't
> > think
> > > > of
> > > > > a
> > > > > > > > situation where it makes sense to name something an empty
> > string
> > > > as
> > > > > > far
> > > > > > > as
> > > > > > > > I'm aware - to me it seems like potential for confusion
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Jul 20, 2018 at 10:22 AM Rajini Sivaram
> > > > > > <rajinisivaram@gmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > +1 to deprecate use of "" as group.id since it is odd to
> > have
> > > a
> > > > > > > resource
> > > > > > > > > name that you cannot set ACLs for. Agree, we have to
support
> > > > older
> > > > > > > > clients
> > > > > > > > > though.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Rajini
> > > > > > > > >
> > > > > > > > > On Fri, Jul 20, 2018 at 5:25 PM, Jason Gustafson
> > > > > > <ja...@confluent.io>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Vahid,
> > > > > > > > > >
> > > > > > > > > > Sorry for getting to this so late. I think there are
two
> > > > things
> > > > > > here:
> > > > > > > > > >
> > > > > > > > > > 1. The use of "" as a groupId has always been a
dubious
> > > > practice
> > > > > > at
> > > > > > > > best.
> > > > > > > > > > We definitely ought to deprecate its use in the
client.
> > > > Perhaps
> > > > > in
> > > > > > > the
> > > > > > > > > next
> > > > > > > > > > major release, we can remove support completely.
However,
> > > > since
> > > > > > older
> > > > > > > > > > clients depend on it, we may have to continue letting
the
> > > > broker
> > > > > > > > support
> > > > > > > > > it
> > > > > > > > > > to some extent. Perhaps we just need to bump the
> > OffsetCommit
> > > > > > request
> > > > > > > > API
> > > > > > > > > > and only accept the offset commit for older versions.
You
> > > > > probably
> > > > > > > > have
> > > > > > > > > to
> > > > > > > > > > do this anyway if you want to introduce the new error
code
> > > > since
> > > > > > old
> > > > > > > > > > clients will not expect it.
> > > > > > > > > >
> > > > > > > > > > 2. There should be a way for the consumer to indicate
that
> > it
> > > > > has
> > > > > > no
> > > > > > > > > group
> > > > > > > > > > id and will not commit offsets. This is an explicit
> > > > instruction
> > > > > > that
> > > > > > > > the
> > > > > > > > > > consumer should not bother with coordinator lookup and
> > such.
> > > > We
> > > > > > > > currently
> > > > > > > > > > have some brittle logic in place to let users avoid
the
> > > > > > coordinator
> > > > > > > > > lookup,
> > > > > > > > > > but it is a bit error-prone. I was hoping that we
could
> > > change
> > > > > the
> > > > > > > > > default
> > > > > > > > > > value of group.id to be null so that the user had to
take
> > an
> > > > > > > explicit
> > > > > > > > > > action to opt into coordinator management (groups or
> > > offsets).
> > > > > > > > However,
> > > > > > > > > it
> > > > > > > > > > is true that some users may be unknowingly depending
on
> > > offset
> > > > > > > storage
> > > > > > > > if
> > > > > > > > > > they are using both the default group.id and the
default
> > > > > > > > > > enable.auto.commit. Perhaps one option is to disable
> > > > > > > > enable.auto.commit
> > > > > > > > > > automatically if no group.id is specified? I am not
sure
> > if
> > > > > there
> > > > > > > are
> > > > > > > > > any
> > > > > > > > > > drawbacks, but my feeling is that implicit use of the
> > empty
> > > > > > group.id
> > > > > > > > to
> > > > > > > > > > commit offsets is more likely to be causing users
> > unexpected
> > > > > > problems
> > > > > > > > > than
> > > > > > > > > > actually providing a useful capability.
> > > > > > > > > >
> > > > > > > > > > Thoughts?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Jason
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, May 28, 2018 at 9:50 AM, Vahid S Hashemian <
> > > > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Viktor,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for sharing your opinion.
> > > > > > > > > > > So you're in favor of disallowing the empty ("")
group
> > id
> > > > > > > altogether
> > > > > > > > > > (even
> > > > > > > > > > > for fetching).
> > > > > > > > > > > Given that ideally no one should be using the empty
> > group
> > > id
> > > > > (at
> > > > > > > > least
> > > > > > > > > in
> > > > > > > > > > > a production setting) I think the impact would be
> > minimal
> > > in
> > > > > > either
> > > > > > > > > case.
> > > > > > > > > > >
> > > > > > > > > > > But as you said, let's hear what others think and
I'd be
> > > > happy
> > > > > > to
> > > > > > > > > modify
> > > > > > > > > > > the KIP if needed.
> > > > > > > > > > >
> > > > > > > > > > > Regards.
> > > > > > > > > > > --Vahid
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > > > > > > > > To: dev <de...@kafka.apache.org>
> > > > > > > > > > > Date: 05/28/2018 05:18 AM
> > > > > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the
> > default
> > > > > group
> > > > > > id
> > > > > > > > > > > behavior in KafkaConsumer
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hi Vahid,
> > > > > > > > > > >
> > > > > > > > > > > (with the argument that using the default group id
for
> > > > offset
> > > > > > > commit
> > > > > > > > > > > should not be the user's intention in practice).
> > > > > > > > > > >
> > > > > > > > > > > Yea, so in my opinion too this use case doesn't seem
too
> > > > > > practical.
> > > > > > > > > Also
> > > > > > > > > > I
> > > > > > > > > > > think breaking the offset commit is not smaller from
> > this
> > > > > > > > perspective
> > > > > > > > > > than
> > > > > > > > > > > breaking fetch and offset fetch. If we suppose that
> > someone
> > > > > uses
> > > > > > > the
> > > > > > > > > > > default group id and we break the offset commit then
> > that
> > > > > might
> > > > > > be
> > > > > > > > > harder
> > > > > > > > > > > to detect than breaking the whole thing altogether.
(If
> > we
> > > > > think
> > > > > > > > about
> > > > > > > > > an
> > > > > > > > > > > upgrade situation.)
> > > > > > > > > > > So since we think it is not a practical use case, I
> > think
> > > it
> > > > > > would
> > > > > > > > be
> > > > > > > > > > > better to break altogether but ofc that's just my 2
> > cents
> > > > :).
> > > > > > Let's
> > > > > > > > > > gather
> > > > > > > > > > > other's input as well.
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > > Viktor
> > > > > > > > > > >
> > > > > > > > > > > On Fri, May 25, 2018 at 5:43 PM, Vahid S Hashemian <
> > > > > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Victor,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for reviewing the KIP.
> > > > > > > > > > > >
> > > > > > > > > > > > Yes, to minimize the backward compatibility
impact,
> > there
> > > > > > would
> > > > > > > be
> > > > > > > > no
> > > > > > > > > > > harm
> > > > > > > > > > > > in letting a stand-alone consumer consume messages
> > under
> > > a
> > > > > ""
> > > > > > > > group
> > > > > > > > > id
> > > > > > > > > > > (as
> > > > > > > > > > > > long as there is no offset commit).
> > > > > > > > > > > > It would have to knowingly seek to an offset or
rely
> > on
> > > > the
> > > > > > > > > > > > auto.offset.reset config for the starting offset.
> > > > > > > > > > > > This way the existing functionality would be
preserved
> > > for
> > > > > the
> > > > > > > > most
> > > > > > > > > > part
> > > > > > > > > > > > (with the argument that using the default group id
for
> > > > > offset
> > > > > > > > commit
> > > > > > > > > > > > should not be the user's intention in practice).
> > > > > > > > > > > >
> > > > > > > > > > > > Does it seem reasonable?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks.
> > > > > > > > > > > > --Vahid
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > > > > > > > > > To: dev <de...@kafka.apache.org>
> > > > > > > > > > > > Date: 05/25/2018 04:56 AM
> > > > > > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the
> > > default
> > > > > > group
> > > > > > > > id
> > > > > > > > > > > > behavior in KafkaConsumer
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Vahid,
> > > > > > > > > > > >
> > > > > > > > > > > > When reading your KIP I coldn't fully understand
why
> > did
> > > > you
> > > > > > > > decide
> > > > > > > > > at
> > > > > > > > > > > > failing with "offset_commit" in case #2? Can't we
fail
> > > > with
> > > > > an
> > > > > > > > empty
> > > > > > > > > > > group
> > > > > > > > > > > > id even in "fetch" or "fetch_offset"? What was the
> > reason
> > > > > for
> > > > > > > > > deciding
> > > > > > > > > > > to
> > > > > > > > > > > > fail at "offset_commit"? Was it because of upgrade
> > > > > > compatibility
> > > > > > > > > > > reasons?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Viktor
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, May 24, 2018 at 12:06 AM, Ted Yu
> > > > > <yu...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Looks good to me.
> > > > > > > > > > > > > -------- Original message --------From: Vahid S
> > > > Hashemian
> > > > > <
> > > > > > > > > > > > > vahidhashemian@us.ibm.com> Date: 5/23/18 11:19
AM
> > > > > > > (GMT-08:00)
> > > > > > > > > To:
> > > > > > > > > > > > > dev@kafka.apache.org Subject: Re: [DISCUSS]
KIP-289:
> > > > > Improve
> > > > > > > the
> > > > > > > > > > > default
> > > > > > > > > > > > > group id behavior in KafkaConsumer
> > > > > > > > > > > > > Hi Ted,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for reviewing the KIP. I updated the KIP
and
> > > > > > introduced
> > > > > > > > an
> > > > > > > > > > > error
> > > > > > > > > > > > > code for the scenario described.
> > > > > > > > > > > > >
> > > > > > > > > > > > > --Vahid
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > From: Ted Yu <yu...@gmail.com>
> > > > > > > > > > > > > To: dev@kafka.apache.org
> > > > > > > > > > > > > Date: 04/27/2018 04:31 PM
> > > > > > > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve
the
> > > > default
> > > > > > > group
> > > > > > > > id
> > > > > > > > > > > > > behavior in KafkaConsumer
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > bq. If they attempt an offset commit they will
> > receive
> > > > an
> > > > > > > error.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Can you outline what specific error would be
> > > encountered
> > > > ?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Apr 27, 2018 at 2:17 PM, Vahid S
Hashemian <
> > > > > > > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I have drafted a proposal for improving the
> > behavior
> > > > of
> > > > > > > > > > > KafkaConsumer
> > > > > > > > > > > > > when
> > > > > > > > > > > > > > using the default group id:
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> >
https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >
> > > >
> > > > >
> > > > > >
> > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
289%3A+Improve+the+default+group+id+behavior+in+
> > > > > > > KafkaConsumer
> > > > > > > > > > > > > > The proposal based on the issue and suggestion
> > > > reported
> > > > > in
> > > > > > > > > > > KAFKA-6774.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Your feedback is welcome!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks.
> > > > > > > > > > > > > > --Vahid
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Best,
> > > > > > > > Stanislav
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best,
> > > > > > > Stanislav
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
> >
> >
> >
> >
Re: [DISCUSS] KIP-289: Improve the default group id behavior in
KafkaConsumer
Posted by Colin McCabe <cm...@apache.org>.
Thanks, Jason. I don't have a very strong opinion on this. But like you said, if we skip bumping the RPC versions, this would be a smaller change, which might be good.
best,
Colin
On Wed, Aug 1, 2018, at 17:43, Jason Gustafson wrote:
> Hey Vahid,
>
> I talked with Colin offline. I think specifically he felt the version bump
> on the broker was overkill since the broker still has to support the empty
> group id for older versions. I had thought that eventually we would be able
> to remove those old versions, but it's true that this may not happen until
> indefinitely far in the future. I think the main improvement here is
> changing the default group.id to null instead of "". I could go either way
> on whether bumping the protocol is useful. I do think it is helpful though
> to signal clearly that it its use is deprecated and discouraged, especially
> in light of the ACL problem. I guess we could just deprecate the use on the
> client. What do you think?
>
> Thanks,
> Jason
>
> On Wed, Aug 1, 2018 at 3:19 PM, Vahid S Hashemian <vahidhashemian@us.ibm.com
> > wrote:
>
> > Thanks Jason for responding to Colin's concerns.
> >
> > If there are no other comment / feedback / objection I'll start a vote
> > soon.
> >
> > Thanks.
> > --Vahid
> >
> >
> >
> > From: Jason Gustafson <ja...@confluent.io>
> > To: dev <de...@kafka.apache.org>
> > Date: 07/27/2018 10:38 AM
> > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > behavior in KafkaConsumer
> >
> >
> >
> > Hey Colin,
> >
> > The problem is both that the empty group id is the default value and that
> > it is actually accepted by the broker for offset commits. Combine that
> > with
> > the fact that auto commit is enabled by default and you users get
> > surprising behavior. If you look at a random Kafka cluster, you'll
> > probably
> > find a bunch of inadvertent offset commits for the empty group id. I was
> > hoping we could distinguish between users who are using the empty group id
> > as an accident of the default configuration and those who use it
> > intentionally. By default, there will be no group id and the consumer will
> > not commit offsets. If a user has actually intentionally used the empty
> > group id, however, it will continue to work. I actually think there are
> > probably very few people doing this (maybe even no one), but I thought we
> > might err on the side of compatibility.
> >
> > The big incompatible change here is having brokers reject using
> > assign(...)
> > > with empty / null group.id.
> >
> >
> > This is not correct. In the proposal, the broker will only reject the
> > empty
> > group id for the new version of OffsetCommit. Older clients, which cannot
> > be changed, will continue to work because the old versions of the
> > OffsetCommit API still accept the empty group id. The null group id is
> > different from the empty group id: it is not allowed in any version of the
> > API. It is basically a way to indicate that the consumer has no dependence
> > on the coordinator at all, which we actually have a surprising number of
> > use cases for. Furthermore, if a user has an actual need for the empty
> > group id, it will still be allowed. We are just deprecating it.
> >
> > -Jason
> >
> > On Fri, Jul 27, 2018 at 9:56 AM, Colin McCabe <cm...@apache.org> wrote:
> >
> > > Sorry if this is a silly question, but what's the rationale for
> > switching
> > > to using null for the default group id, rather than the empty string?
> > > Continuing to use the empty string seems like less churn. And after
> > all,
> > > we're not using the empty string group name for anything else.
> > >
> > > The big incompatible change here is having brokers reject using
> > > assign(...) with empty / null group.id. If I understand correctly, the
> > > KIP proposes that this change be made on the brokers on the next
> > > incompatible Kafka release. But that has nothing to do with client
> > > versions. Why not just have a broker config which controls this? Maybe
> > "
> > > allow.assign.empty.group.id", or something like that. At first, the
> > > default will be true, and then eventually we can flip it over to false.
> > >
> > > It seems like the main rationale for tying this behavior to the Kafka
> > > client version is to force people to stop using the empty group id so
> > that
> > > they can upgrade their clients. But it's also possible that people will
> > > stop upgrading their Kafka clients instead. That would be pretty
> > negative
> > > since they'd miss out on any efficiency and feature improvements in the
> > > new clients and eventually have to do more protocol downgrading, etc.
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Thu, Jul 26, 2018, at 11:50, Vahid S Hashemian wrote:
> > > > Hi Jason,
> > > >
> > > > That makes sense.
> > > > I have updated the KIP based on the recent feedback.
> > > >
> > > > Thanks!
> > > > --Vahid
> > > >
> > > >
> > > >
> > > >
> > > > From: Jason Gustafson <ja...@confluent.io>
> > > > To: dev <de...@kafka.apache.org>
> > > > Date: 07/25/2018 02:23 PM
> > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > > > behavior in KafkaConsumer
> > > >
> > > >
> > > >
> > > > Hi Vahid,
> > > >
> > > > I was thinking we'd only use the old API version if we had to. That
> > is,
> > > > only if the user has explicitly configured "" as the group.id.
> > > Otherwise,
> > > > we'd just use the new one. Another option is to just drop support in
> > the
> > > > client for the empty group id, but usually we allow a deprecation
> > period
> > > > for changes like this.
> > > >
> > > > -Jason
> > > >
> > > > On Wed, Jul 25, 2018 at 12:49 PM, Vahid S Hashemian <
> > > > vahidhashemian@us.ibm.com> wrote:
> > > >
> > > > > Hi Jason,
> > > > >
> > > > > Thanks for additional clarification.
> > > > >
> > > > > So the next version of the OffsetCommit API will return an
> > > > > INVALID_GROUP_ID error for empty group ids; but on the client side
> > we
> > > > call
> > > > > the older version of the client until the next major release.
> > > > > The table below should summarize this.
> > > > >
> > > > > +-----------------------------------------------------+
> > > > > | Client (group.id="") |
> > > > > +-----------------------------------------------------+
> > > > > | pre-2.1 | 2.1 | 3.0 |
> > > > >
> > > > +-----+-----------+---------+------------------------+------
> > > ------------+
> > > > > | | V5 (cur.) | works | works | works |
> > > > > + API
> > > > +-----------+---------+------------------------+------------------+
> > > > > | | V6 | N/A | N/A (calls V5/warning) |
> > > INVALID_GROUP_ID
> > > > |
> > > > >
> > > > +-----+-----------+---------+------------------------+------
> > > ------------+
> > > > >
> > > > > Assumptions:
> > > > > * 2.1: The target release version for this KIP
> > > > > * 3.0: The next major release
> > > > >
> > > > > Please advise if you see an issue; otherwise, I'll update the KIP
> > > > > accordingly.
> > > > >
> > > > > Thanks!
> > > > > --Vahid
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > From: Jason Gustafson <ja...@confluent.io>
> > > > > To: dev <de...@kafka.apache.org>
> > > > > Date: 07/25/2018 12:08 AM
> > > > > Subject: ***UNCHECKED*** Re: [DISCUSS] KIP-289: Improve the
> > > > default
> > > > > group id behavior in KafkaConsumer
> > > > >
> > > > >
> > > > >
> > > > > Hey Vahid,
> > > > >
> > > > > Sorry for the confusion. I think we all agree that going forward, we
> > > > > shouldn't support the empty group id, so the question is just around
> > > > > compatibility. I think we have to bump the OffsetCommit API version
> > so
> > > > > that
> > > > > old clients which are unknowingly depending on the default empty
> > group
> > > > id
> > > > > will continue to work with new brokers. For new versions of the
> > > client,
> > > > we
> > > > > can either drop support for the empty group id immediately or we can
> > > > give
> > > > > users a grace period. I was thinking we would do the latter. We can
> > > > change
> > > > > the default group.id, but in the case that a user has explicitly
> > > > > configured
> > > > > the empty group, then we can just use an old version of the
> > > OffsetCommit
> > > > > API which still supports it. In a future release, we can drop this
> > > > support
> > > > > and only use the latest OffsetCommit version. Does that make sense?
> > > > >
> > > > > Thanks,
> > > > > Jason
> > > > >
> > > > >
> > > > > On Tue, Jul 24, 2018 at 12:36 PM, Vahid S Hashemian <
> > > > > vahidhashemian@us.ibm.com> wrote:
> > > > >
> > > > > > Hi Jason,
> > > > > >
> > > > > > Thanks for clarifying.
> > > > > >
> > > > > > So if we are going to continue supporting the empty group id as
> > > before
> > > > > > (with only an addition of a deprecation warning), and disable
> > > > > > enable.auto.commit for the new default (null) group id on the
> > client
> > > > > side,
> > > > > > do we really need to bump up the OffsetCommit version?
> > > > > >
> > > > > > You mentioned "If an explicit empty string is configured for the
> > > group
> > > > > id,
> > > > > > then maybe we keep the current behavior for compatibility" which
> > > makes
> > > > > > sense to me, but I find it in conflict with your earlier
> > suggestion
> > > > "we
> > > > > > just need to bump the OffsetCommit request API and only accept the
> > > > > offset
> > > > > > commit for older versions.". Maybe I'm missing something?
> > > > > >
> > > > > > Thanks!
> > > > > > --Vahid
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > From: Jason Gustafson <ja...@confluent.io>
> > > > > > To: dev <de...@kafka.apache.org>
> > > > > > Date: 07/23/2018 10:52 PM
> > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group
> > id
> > > > > > behavior in KafkaConsumer
> > > > > >
> > > > > >
> > > > > >
> > > > > > Hey Vahid,
> > > > > >
> > > > > > Thanks for the updates. Just to clarify, I was suggesting that we
> > > > > disable
> > > > > > enable.auto.commit only if no explicit group.id is configured. If
> > an
> > > > > > explicit empty string is configured for the group id, then maybe
> > we
> > > > keep
> > > > > > the current behavior for compatibility. We can log a warning
> > > > mentioning
> > > > > > the
> > > > > > deprecation and we can use the old version of the OffsetCommit API
> > > > that
> > > > > > allows the empty group id. In a later release, we can drop this
> > > > support
> > > > > in
> > > > > > the client. Does that seem reasonable?
> > > > > >
> > > > > > By the way, instead of using the new ILLEGAL_OFFSET_COMMIT error
> > > code,
> > > > > > couldn't we use INVALID_GROUP_ID?
> > > > > >
> > > > > > Thanks,
> > > > > > Jason
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Jul 23, 2018 at 5:14 PM, Stanislav Kozlovski
> > > > > > <stanislav@confluent.io
> > > > > > > wrote:
> > > > > >
> > > > > > > Hey Vahid,
> > > > > > >
> > > > > > > No I don't see an issue with it. I believe it to be the best
> > > > approach.
> > > > > > >
> > > > > > > Best,
> > > > > > > Stanisav
> > > > > > >
> > > > > > > On Mon, Jul 23, 2018 at 12:41 PM Vahid S Hashemian <
> > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > >
> > > > > > > > Hi Stanislav,
> > > > > > > >
> > > > > > > > Thanks for the feedback.
> > > > > > > > Do you see an issue with using `null` as the default group id
> > (as
> > > > > > > > addressed by Jason in his response)?
> > > > > > > > This default group id would not support offset commits and
> > > > consumers
> > > > > > > would
> > > > > > > > use `auto.offset.reset` config when there is no current
> > offset.
> > > > > > > >
> > > > > > > > Thanks.
> > > > > > > > --Vahid
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > From: Stanislav Kozlovski <st...@confluent.io>
> > > > > > > > To: dev@kafka.apache.org
> > > > > > > > Date: 07/20/2018 11:09 AM
> > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default
> > group
> > > > id
> > > > > > > > behavior in KafkaConsumer
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > I agree with Jason's notion that
> > > > > > > > > implicit use of the empty group.id to commit offsets is
> > more
> > > > > likely
> > > > > > > to
> > > > > > > > be causing users unexpected problems than actually providing a
> > > > > useful
> > > > > > > > capability.
> > > > > > > > I was initially confused that this is the behavior when
> > > > > investigating
> > > > > > a
> > > > > > > > new-ish JIRA issue <
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > https://issues.apache.org/jira/browse/KAFKA-6758
> >
> > > >
> > > > >
> > > > > >
> > > > > > > > > about
> > > > > > > > the same topic.
> > > > > > > > So, +1 to deprecating "" as a group.id
> > > > > > > >
> > > > > > > > The question after that becomes what the *default* value
> > should
> > > be
> > > > -
> > > > > > > > should
> > > > > > > > we:
> > > > > > > > a) treat an unconfigured group.id consumer as a sort of
> > > > intermittent
> > > > > > > > consumer where you don't store offsets at all (thereby making
> > the
> > > > > user
> > > > > > > > explicitly sign up for them)
> > > > > > > > b) have a default value which makes use of them? I sort of
> > like
> > > > the
> > > > > > > > former.
> > > > > > > >
> > > > > > > > @Dhruvil, thinking about it at a high-level - yes. I can't
> > think
> > > > of
> > > > > a
> > > > > > > > situation where it makes sense to name something an empty
> > string
> > > > as
> > > > > > far
> > > > > > > as
> > > > > > > > I'm aware - to me it seems like potential for confusion
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Jul 20, 2018 at 10:22 AM Rajini Sivaram
> > > > > > <rajinisivaram@gmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > +1 to deprecate use of "" as group.id since it is odd to
> > have
> > > a
> > > > > > > resource
> > > > > > > > > name that you cannot set ACLs for. Agree, we have to support
> > > > older
> > > > > > > > clients
> > > > > > > > > though.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Rajini
> > > > > > > > >
> > > > > > > > > On Fri, Jul 20, 2018 at 5:25 PM, Jason Gustafson
> > > > > > <ja...@confluent.io>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Vahid,
> > > > > > > > > >
> > > > > > > > > > Sorry for getting to this so late. I think there are two
> > > > things
> > > > > > here:
> > > > > > > > > >
> > > > > > > > > > 1. The use of "" as a groupId has always been a dubious
> > > > practice
> > > > > > at
> > > > > > > > best.
> > > > > > > > > > We definitely ought to deprecate its use in the client.
> > > > Perhaps
> > > > > in
> > > > > > > the
> > > > > > > > > next
> > > > > > > > > > major release, we can remove support completely. However,
> > > > since
> > > > > > older
> > > > > > > > > > clients depend on it, we may have to continue letting the
> > > > broker
> > > > > > > > support
> > > > > > > > > it
> > > > > > > > > > to some extent. Perhaps we just need to bump the
> > OffsetCommit
> > > > > > request
> > > > > > > > API
> > > > > > > > > > and only accept the offset commit for older versions. You
> > > > > probably
> > > > > > > > have
> > > > > > > > > to
> > > > > > > > > > do this anyway if you want to introduce the new error code
> > > > since
> > > > > > old
> > > > > > > > > > clients will not expect it.
> > > > > > > > > >
> > > > > > > > > > 2. There should be a way for the consumer to indicate that
> > it
> > > > > has
> > > > > > no
> > > > > > > > > group
> > > > > > > > > > id and will not commit offsets. This is an explicit
> > > > instruction
> > > > > > that
> > > > > > > > the
> > > > > > > > > > consumer should not bother with coordinator lookup and
> > such.
> > > > We
> > > > > > > > currently
> > > > > > > > > > have some brittle logic in place to let users avoid the
> > > > > > coordinator
> > > > > > > > > lookup,
> > > > > > > > > > but it is a bit error-prone. I was hoping that we could
> > > change
> > > > > the
> > > > > > > > > default
> > > > > > > > > > value of group.id to be null so that the user had to take
> > an
> > > > > > > explicit
> > > > > > > > > > action to opt into coordinator management (groups or
> > > offsets).
> > > > > > > > However,
> > > > > > > > > it
> > > > > > > > > > is true that some users may be unknowingly depending on
> > > offset
> > > > > > > storage
> > > > > > > > if
> > > > > > > > > > they are using both the default group.id and the default
> > > > > > > > > > enable.auto.commit. Perhaps one option is to disable
> > > > > > > > enable.auto.commit
> > > > > > > > > > automatically if no group.id is specified? I am not sure
> > if
> > > > > there
> > > > > > > are
> > > > > > > > > any
> > > > > > > > > > drawbacks, but my feeling is that implicit use of the
> > empty
> > > > > > group.id
> > > > > > > > to
> > > > > > > > > > commit offsets is more likely to be causing users
> > unexpected
> > > > > > problems
> > > > > > > > > than
> > > > > > > > > > actually providing a useful capability.
> > > > > > > > > >
> > > > > > > > > > Thoughts?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Jason
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, May 28, 2018 at 9:50 AM, Vahid S Hashemian <
> > > > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Viktor,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for sharing your opinion.
> > > > > > > > > > > So you're in favor of disallowing the empty ("") group
> > id
> > > > > > > altogether
> > > > > > > > > > (even
> > > > > > > > > > > for fetching).
> > > > > > > > > > > Given that ideally no one should be using the empty
> > group
> > > id
> > > > > (at
> > > > > > > > least
> > > > > > > > > in
> > > > > > > > > > > a production setting) I think the impact would be
> > minimal
> > > in
> > > > > > either
> > > > > > > > > case.
> > > > > > > > > > >
> > > > > > > > > > > But as you said, let's hear what others think and I'd be
> > > > happy
> > > > > > to
> > > > > > > > > modify
> > > > > > > > > > > the KIP if needed.
> > > > > > > > > > >
> > > > > > > > > > > Regards.
> > > > > > > > > > > --Vahid
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > > > > > > > > To: dev <de...@kafka.apache.org>
> > > > > > > > > > > Date: 05/28/2018 05:18 AM
> > > > > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the
> > default
> > > > > group
> > > > > > id
> > > > > > > > > > > behavior in KafkaConsumer
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hi Vahid,
> > > > > > > > > > >
> > > > > > > > > > > (with the argument that using the default group id for
> > > > offset
> > > > > > > commit
> > > > > > > > > > > should not be the user's intention in practice).
> > > > > > > > > > >
> > > > > > > > > > > Yea, so in my opinion too this use case doesn't seem too
> > > > > > practical.
> > > > > > > > > Also
> > > > > > > > > > I
> > > > > > > > > > > think breaking the offset commit is not smaller from
> > this
> > > > > > > > perspective
> > > > > > > > > > than
> > > > > > > > > > > breaking fetch and offset fetch. If we suppose that
> > someone
> > > > > uses
> > > > > > > the
> > > > > > > > > > > default group id and we break the offset commit then
> > that
> > > > > might
> > > > > > be
> > > > > > > > > harder
> > > > > > > > > > > to detect than breaking the whole thing altogether. (If
> > we
> > > > > think
> > > > > > > > about
> > > > > > > > > an
> > > > > > > > > > > upgrade situation.)
> > > > > > > > > > > So since we think it is not a practical use case, I
> > think
> > > it
> > > > > > would
> > > > > > > > be
> > > > > > > > > > > better to break altogether but ofc that's just my 2
> > cents
> > > > :).
> > > > > > Let's
> > > > > > > > > > gather
> > > > > > > > > > > other's input as well.
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > > Viktor
> > > > > > > > > > >
> > > > > > > > > > > On Fri, May 25, 2018 at 5:43 PM, Vahid S Hashemian <
> > > > > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Victor,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for reviewing the KIP.
> > > > > > > > > > > >
> > > > > > > > > > > > Yes, to minimize the backward compatibility impact,
> > there
> > > > > > would
> > > > > > > be
> > > > > > > > no
> > > > > > > > > > > harm
> > > > > > > > > > > > in letting a stand-alone consumer consume messages
> > under
> > > a
> > > > > ""
> > > > > > > > group
> > > > > > > > > id
> > > > > > > > > > > (as
> > > > > > > > > > > > long as there is no offset commit).
> > > > > > > > > > > > It would have to knowingly seek to an offset or rely
> > on
> > > > the
> > > > > > > > > > > > auto.offset.reset config for the starting offset.
> > > > > > > > > > > > This way the existing functionality would be preserved
> > > for
> > > > > the
> > > > > > > > most
> > > > > > > > > > part
> > > > > > > > > > > > (with the argument that using the default group id for
> > > > > offset
> > > > > > > > commit
> > > > > > > > > > > > should not be the user's intention in practice).
> > > > > > > > > > > >
> > > > > > > > > > > > Does it seem reasonable?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks.
> > > > > > > > > > > > --Vahid
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > > > > > > > > > To: dev <de...@kafka.apache.org>
> > > > > > > > > > > > Date: 05/25/2018 04:56 AM
> > > > > > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the
> > > default
> > > > > > group
> > > > > > > > id
> > > > > > > > > > > > behavior in KafkaConsumer
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Vahid,
> > > > > > > > > > > >
> > > > > > > > > > > > When reading your KIP I coldn't fully understand why
> > did
> > > > you
> > > > > > > > decide
> > > > > > > > > at
> > > > > > > > > > > > failing with "offset_commit" in case #2? Can't we fail
> > > > with
> > > > > an
> > > > > > > > empty
> > > > > > > > > > > group
> > > > > > > > > > > > id even in "fetch" or "fetch_offset"? What was the
> > reason
> > > > > for
> > > > > > > > > deciding
> > > > > > > > > > > to
> > > > > > > > > > > > fail at "offset_commit"? Was it because of upgrade
> > > > > > compatibility
> > > > > > > > > > > reasons?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Viktor
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, May 24, 2018 at 12:06 AM, Ted Yu
> > > > > <yu...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Looks good to me.
> > > > > > > > > > > > > -------- Original message --------From: Vahid S
> > > > Hashemian
> > > > > <
> > > > > > > > > > > > > vahidhashemian@us.ibm.com> Date: 5/23/18 11:19 AM
> > > > > > > (GMT-08:00)
> > > > > > > > > To:
> > > > > > > > > > > > > dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-289:
> > > > > Improve
> > > > > > > the
> > > > > > > > > > > default
> > > > > > > > > > > > > group id behavior in KafkaConsumer
> > > > > > > > > > > > > Hi Ted,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for reviewing the KIP. I updated the KIP and
> > > > > > introduced
> > > > > > > > an
> > > > > > > > > > > error
> > > > > > > > > > > > > code for the scenario described.
> > > > > > > > > > > > >
> > > > > > > > > > > > > --Vahid
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > From: Ted Yu <yu...@gmail.com>
> > > > > > > > > > > > > To: dev@kafka.apache.org
> > > > > > > > > > > > > Date: 04/27/2018 04:31 PM
> > > > > > > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the
> > > > default
> > > > > > > group
> > > > > > > > id
> > > > > > > > > > > > > behavior in KafkaConsumer
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > bq. If they attempt an offset commit they will
> > receive
> > > > an
> > > > > > > error.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Can you outline what specific error would be
> > > encountered
> > > > ?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Apr 27, 2018 at 2:17 PM, Vahid S Hashemian <
> > > > > > > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I have drafted a proposal for improving the
> > behavior
> > > > of
> > > > > > > > > > > KafkaConsumer
> > > > > > > > > > > > > when
> > > > > > > > > > > > > > using the default group id:
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >
> > > >
> > > > >
> > > > > >
> > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 289%3A+Improve+the+default+group+id+behavior+in+
> > > > > > > KafkaConsumer
> > > > > > > > > > > > > > The proposal based on the issue and suggestion
> > > > reported
> > > > > in
> > > > > > > > > > > KAFKA-6774.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Your feedback is welcome!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks.
> > > > > > > > > > > > > > --Vahid
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Best,
> > > > > > > > Stanislav
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best,
> > > > > > > Stanislav
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
> >
> >
> >
> >
Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer
Posted by Jason Gustafson <ja...@confluent.io>.
Hey Vahid,
I talked with Colin offline. I think specifically he felt the version bump
on the broker was overkill since the broker still has to support the empty
group id for older versions. I had thought that eventually we would be able
to remove those old versions, but it's true that this may not happen until
indefinitely far in the future. I think the main improvement here is
changing the default group.id to null instead of "". I could go either way
on whether bumping the protocol is useful. I do think it is helpful though
to signal clearly that it its use is deprecated and discouraged, especially
in light of the ACL problem. I guess we could just deprecate the use on the
client. What do you think?
Thanks,
Jason
On Wed, Aug 1, 2018 at 3:19 PM, Vahid S Hashemian <vahidhashemian@us.ibm.com
> wrote:
> Thanks Jason for responding to Colin's concerns.
>
> If there are no other comment / feedback / objection I'll start a vote
> soon.
>
> Thanks.
> --Vahid
>
>
>
> From: Jason Gustafson <ja...@confluent.io>
> To: dev <de...@kafka.apache.org>
> Date: 07/27/2018 10:38 AM
> Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> behavior in KafkaConsumer
>
>
>
> Hey Colin,
>
> The problem is both that the empty group id is the default value and that
> it is actually accepted by the broker for offset commits. Combine that
> with
> the fact that auto commit is enabled by default and you users get
> surprising behavior. If you look at a random Kafka cluster, you'll
> probably
> find a bunch of inadvertent offset commits for the empty group id. I was
> hoping we could distinguish between users who are using the empty group id
> as an accident of the default configuration and those who use it
> intentionally. By default, there will be no group id and the consumer will
> not commit offsets. If a user has actually intentionally used the empty
> group id, however, it will continue to work. I actually think there are
> probably very few people doing this (maybe even no one), but I thought we
> might err on the side of compatibility.
>
> The big incompatible change here is having brokers reject using
> assign(...)
> > with empty / null group.id.
>
>
> This is not correct. In the proposal, the broker will only reject the
> empty
> group id for the new version of OffsetCommit. Older clients, which cannot
> be changed, will continue to work because the old versions of the
> OffsetCommit API still accept the empty group id. The null group id is
> different from the empty group id: it is not allowed in any version of the
> API. It is basically a way to indicate that the consumer has no dependence
> on the coordinator at all, which we actually have a surprising number of
> use cases for. Furthermore, if a user has an actual need for the empty
> group id, it will still be allowed. We are just deprecating it.
>
> -Jason
>
> On Fri, Jul 27, 2018 at 9:56 AM, Colin McCabe <cm...@apache.org> wrote:
>
> > Sorry if this is a silly question, but what's the rationale for
> switching
> > to using null for the default group id, rather than the empty string?
> > Continuing to use the empty string seems like less churn. And after
> all,
> > we're not using the empty string group name for anything else.
> >
> > The big incompatible change here is having brokers reject using
> > assign(...) with empty / null group.id. If I understand correctly, the
> > KIP proposes that this change be made on the brokers on the next
> > incompatible Kafka release. But that has nothing to do with client
> > versions. Why not just have a broker config which controls this? Maybe
> "
> > allow.assign.empty.group.id", or something like that. At first, the
> > default will be true, and then eventually we can flip it over to false.
> >
> > It seems like the main rationale for tying this behavior to the Kafka
> > client version is to force people to stop using the empty group id so
> that
> > they can upgrade their clients. But it's also possible that people will
> > stop upgrading their Kafka clients instead. That would be pretty
> negative
> > since they'd miss out on any efficiency and feature improvements in the
> > new clients and eventually have to do more protocol downgrading, etc.
> >
> > best,
> > Colin
> >
> >
> > On Thu, Jul 26, 2018, at 11:50, Vahid S Hashemian wrote:
> > > Hi Jason,
> > >
> > > That makes sense.
> > > I have updated the KIP based on the recent feedback.
> > >
> > > Thanks!
> > > --Vahid
> > >
> > >
> > >
> > >
> > > From: Jason Gustafson <ja...@confluent.io>
> > > To: dev <de...@kafka.apache.org>
> > > Date: 07/25/2018 02:23 PM
> > > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > > behavior in KafkaConsumer
> > >
> > >
> > >
> > > Hi Vahid,
> > >
> > > I was thinking we'd only use the old API version if we had to. That
> is,
> > > only if the user has explicitly configured "" as the group.id.
> > Otherwise,
> > > we'd just use the new one. Another option is to just drop support in
> the
> > > client for the empty group id, but usually we allow a deprecation
> period
> > > for changes like this.
> > >
> > > -Jason
> > >
> > > On Wed, Jul 25, 2018 at 12:49 PM, Vahid S Hashemian <
> > > vahidhashemian@us.ibm.com> wrote:
> > >
> > > > Hi Jason,
> > > >
> > > > Thanks for additional clarification.
> > > >
> > > > So the next version of the OffsetCommit API will return an
> > > > INVALID_GROUP_ID error for empty group ids; but on the client side
> we
> > > call
> > > > the older version of the client until the next major release.
> > > > The table below should summarize this.
> > > >
> > > > +-----------------------------------------------------+
> > > > | Client (group.id="") |
> > > > +-----------------------------------------------------+
> > > > | pre-2.1 | 2.1 | 3.0 |
> > > >
> > > +-----+-----------+---------+------------------------+------
> > ------------+
> > > > | | V5 (cur.) | works | works | works |
> > > > + API
> > > +-----------+---------+------------------------+------------------+
> > > > | | V6 | N/A | N/A (calls V5/warning) |
> > INVALID_GROUP_ID
> > > |
> > > >
> > > +-----+-----------+---------+------------------------+------
> > ------------+
> > > >
> > > > Assumptions:
> > > > * 2.1: The target release version for this KIP
> > > > * 3.0: The next major release
> > > >
> > > > Please advise if you see an issue; otherwise, I'll update the KIP
> > > > accordingly.
> > > >
> > > > Thanks!
> > > > --Vahid
> > > >
> > > >
> > > >
> > > >
> > > > From: Jason Gustafson <ja...@confluent.io>
> > > > To: dev <de...@kafka.apache.org>
> > > > Date: 07/25/2018 12:08 AM
> > > > Subject: ***UNCHECKED*** Re: [DISCUSS] KIP-289: Improve the
> > > default
> > > > group id behavior in KafkaConsumer
> > > >
> > > >
> > > >
> > > > Hey Vahid,
> > > >
> > > > Sorry for the confusion. I think we all agree that going forward, we
> > > > shouldn't support the empty group id, so the question is just around
> > > > compatibility. I think we have to bump the OffsetCommit API version
> so
> > > > that
> > > > old clients which are unknowingly depending on the default empty
> group
> > > id
> > > > will continue to work with new brokers. For new versions of the
> > client,
> > > we
> > > > can either drop support for the empty group id immediately or we can
> > > give
> > > > users a grace period. I was thinking we would do the latter. We can
> > > change
> > > > the default group.id, but in the case that a user has explicitly
> > > > configured
> > > > the empty group, then we can just use an old version of the
> > OffsetCommit
> > > > API which still supports it. In a future release, we can drop this
> > > support
> > > > and only use the latest OffsetCommit version. Does that make sense?
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > >
> > > > On Tue, Jul 24, 2018 at 12:36 PM, Vahid S Hashemian <
> > > > vahidhashemian@us.ibm.com> wrote:
> > > >
> > > > > Hi Jason,
> > > > >
> > > > > Thanks for clarifying.
> > > > >
> > > > > So if we are going to continue supporting the empty group id as
> > before
> > > > > (with only an addition of a deprecation warning), and disable
> > > > > enable.auto.commit for the new default (null) group id on the
> client
> > > > side,
> > > > > do we really need to bump up the OffsetCommit version?
> > > > >
> > > > > You mentioned "If an explicit empty string is configured for the
> > group
> > > > id,
> > > > > then maybe we keep the current behavior for compatibility" which
> > makes
> > > > > sense to me, but I find it in conflict with your earlier
> suggestion
> > > "we
> > > > > just need to bump the OffsetCommit request API and only accept the
> > > > offset
> > > > > commit for older versions.". Maybe I'm missing something?
> > > > >
> > > > > Thanks!
> > > > > --Vahid
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > From: Jason Gustafson <ja...@confluent.io>
> > > > > To: dev <de...@kafka.apache.org>
> > > > > Date: 07/23/2018 10:52 PM
> > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group
> id
> > > > > behavior in KafkaConsumer
> > > > >
> > > > >
> > > > >
> > > > > Hey Vahid,
> > > > >
> > > > > Thanks for the updates. Just to clarify, I was suggesting that we
> > > > disable
> > > > > enable.auto.commit only if no explicit group.id is configured. If
> an
> > > > > explicit empty string is configured for the group id, then maybe
> we
> > > keep
> > > > > the current behavior for compatibility. We can log a warning
> > > mentioning
> > > > > the
> > > > > deprecation and we can use the old version of the OffsetCommit API
> > > that
> > > > > allows the empty group id. In a later release, we can drop this
> > > support
> > > > in
> > > > > the client. Does that seem reasonable?
> > > > >
> > > > > By the way, instead of using the new ILLEGAL_OFFSET_COMMIT error
> > code,
> > > > > couldn't we use INVALID_GROUP_ID?
> > > > >
> > > > > Thanks,
> > > > > Jason
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Jul 23, 2018 at 5:14 PM, Stanislav Kozlovski
> > > > > <stanislav@confluent.io
> > > > > > wrote:
> > > > >
> > > > > > Hey Vahid,
> > > > > >
> > > > > > No I don't see an issue with it. I believe it to be the best
> > > approach.
> > > > > >
> > > > > > Best,
> > > > > > Stanisav
> > > > > >
> > > > > > On Mon, Jul 23, 2018 at 12:41 PM Vahid S Hashemian <
> > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > >
> > > > > > > Hi Stanislav,
> > > > > > >
> > > > > > > Thanks for the feedback.
> > > > > > > Do you see an issue with using `null` as the default group id
> (as
> > > > > > > addressed by Jason in his response)?
> > > > > > > This default group id would not support offset commits and
> > > consumers
> > > > > > would
> > > > > > > use `auto.offset.reset` config when there is no current
> offset.
> > > > > > >
> > > > > > > Thanks.
> > > > > > > --Vahid
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > From: Stanislav Kozlovski <st...@confluent.io>
> > > > > > > To: dev@kafka.apache.org
> > > > > > > Date: 07/20/2018 11:09 AM
> > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default
> group
> > > id
> > > > > > > behavior in KafkaConsumer
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > I agree with Jason's notion that
> > > > > > > > implicit use of the empty group.id to commit offsets is
> more
> > > > likely
> > > > > > to
> > > > > > > be causing users unexpected problems than actually providing a
> > > > useful
> > > > > > > capability.
> > > > > > > I was initially confused that this is the behavior when
> > > > investigating
> > > > > a
> > > > > > > new-ish JIRA issue <
> > > > > > >
> > > > >
> > > >
> > >
> https://issues.apache.org/jira/browse/KAFKA-6758
>
> > >
> > > >
> > > > >
> > > > > > > > about
> > > > > > > the same topic.
> > > > > > > So, +1 to deprecating "" as a group.id
> > > > > > >
> > > > > > > The question after that becomes what the *default* value
> should
> > be
> > > -
> > > > > > > should
> > > > > > > we:
> > > > > > > a) treat an unconfigured group.id consumer as a sort of
> > > intermittent
> > > > > > > consumer where you don't store offsets at all (thereby making
> the
> > > > user
> > > > > > > explicitly sign up for them)
> > > > > > > b) have a default value which makes use of them? I sort of
> like
> > > the
> > > > > > > former.
> > > > > > >
> > > > > > > @Dhruvil, thinking about it at a high-level - yes. I can't
> think
> > > of
> > > > a
> > > > > > > situation where it makes sense to name something an empty
> string
> > > as
> > > > > far
> > > > > > as
> > > > > > > I'm aware - to me it seems like potential for confusion
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Jul 20, 2018 at 10:22 AM Rajini Sivaram
> > > > > <rajinisivaram@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1 to deprecate use of "" as group.id since it is odd to
> have
> > a
> > > > > > resource
> > > > > > > > name that you cannot set ACLs for. Agree, we have to support
> > > older
> > > > > > > clients
> > > > > > > > though.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Rajini
> > > > > > > >
> > > > > > > > On Fri, Jul 20, 2018 at 5:25 PM, Jason Gustafson
> > > > > <ja...@confluent.io>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Vahid,
> > > > > > > > >
> > > > > > > > > Sorry for getting to this so late. I think there are two
> > > things
> > > > > here:
> > > > > > > > >
> > > > > > > > > 1. The use of "" as a groupId has always been a dubious
> > > practice
> > > > > at
> > > > > > > best.
> > > > > > > > > We definitely ought to deprecate its use in the client.
> > > Perhaps
> > > > in
> > > > > > the
> > > > > > > > next
> > > > > > > > > major release, we can remove support completely. However,
> > > since
> > > > > older
> > > > > > > > > clients depend on it, we may have to continue letting the
> > > broker
> > > > > > > support
> > > > > > > > it
> > > > > > > > > to some extent. Perhaps we just need to bump the
> OffsetCommit
> > > > > request
> > > > > > > API
> > > > > > > > > and only accept the offset commit for older versions. You
> > > > probably
> > > > > > > have
> > > > > > > > to
> > > > > > > > > do this anyway if you want to introduce the new error code
> > > since
> > > > > old
> > > > > > > > > clients will not expect it.
> > > > > > > > >
> > > > > > > > > 2. There should be a way for the consumer to indicate that
> it
> > > > has
> > > > > no
> > > > > > > > group
> > > > > > > > > id and will not commit offsets. This is an explicit
> > > instruction
> > > > > that
> > > > > > > the
> > > > > > > > > consumer should not bother with coordinator lookup and
> such.
> > > We
> > > > > > > currently
> > > > > > > > > have some brittle logic in place to let users avoid the
> > > > > coordinator
> > > > > > > > lookup,
> > > > > > > > > but it is a bit error-prone. I was hoping that we could
> > change
> > > > the
> > > > > > > > default
> > > > > > > > > value of group.id to be null so that the user had to take
> an
> > > > > > explicit
> > > > > > > > > action to opt into coordinator management (groups or
> > offsets).
> > > > > > > However,
> > > > > > > > it
> > > > > > > > > is true that some users may be unknowingly depending on
> > offset
> > > > > > storage
> > > > > > > if
> > > > > > > > > they are using both the default group.id and the default
> > > > > > > > > enable.auto.commit. Perhaps one option is to disable
> > > > > > > enable.auto.commit
> > > > > > > > > automatically if no group.id is specified? I am not sure
> if
> > > > there
> > > > > > are
> > > > > > > > any
> > > > > > > > > drawbacks, but my feeling is that implicit use of the
> empty
> > > > > group.id
> > > > > > > to
> > > > > > > > > commit offsets is more likely to be causing users
> unexpected
> > > > > problems
> > > > > > > > than
> > > > > > > > > actually providing a useful capability.
> > > > > > > > >
> > > > > > > > > Thoughts?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Jason
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, May 28, 2018 at 9:50 AM, Vahid S Hashemian <
> > > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > > >
> > > > > > > > > > Hi Viktor,
> > > > > > > > > >
> > > > > > > > > > Thanks for sharing your opinion.
> > > > > > > > > > So you're in favor of disallowing the empty ("") group
> id
> > > > > > altogether
> > > > > > > > > (even
> > > > > > > > > > for fetching).
> > > > > > > > > > Given that ideally no one should be using the empty
> group
> > id
> > > > (at
> > > > > > > least
> > > > > > > > in
> > > > > > > > > > a production setting) I think the impact would be
> minimal
> > in
> > > > > either
> > > > > > > > case.
> > > > > > > > > >
> > > > > > > > > > But as you said, let's hear what others think and I'd be
> > > happy
> > > > > to
> > > > > > > > modify
> > > > > > > > > > the KIP if needed.
> > > > > > > > > >
> > > > > > > > > > Regards.
> > > > > > > > > > --Vahid
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > > > > > > > To: dev <de...@kafka.apache.org>
> > > > > > > > > > Date: 05/28/2018 05:18 AM
> > > > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the
> default
> > > > group
> > > > > id
> > > > > > > > > > behavior in KafkaConsumer
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hi Vahid,
> > > > > > > > > >
> > > > > > > > > > (with the argument that using the default group id for
> > > offset
> > > > > > commit
> > > > > > > > > > should not be the user's intention in practice).
> > > > > > > > > >
> > > > > > > > > > Yea, so in my opinion too this use case doesn't seem too
> > > > > practical.
> > > > > > > > Also
> > > > > > > > > I
> > > > > > > > > > think breaking the offset commit is not smaller from
> this
> > > > > > > perspective
> > > > > > > > > than
> > > > > > > > > > breaking fetch and offset fetch. If we suppose that
> someone
> > > > uses
> > > > > > the
> > > > > > > > > > default group id and we break the offset commit then
> that
> > > > might
> > > > > be
> > > > > > > > harder
> > > > > > > > > > to detect than breaking the whole thing altogether. (If
> we
> > > > think
> > > > > > > about
> > > > > > > > an
> > > > > > > > > > upgrade situation.)
> > > > > > > > > > So since we think it is not a practical use case, I
> think
> > it
> > > > > would
> > > > > > > be
> > > > > > > > > > better to break altogether but ofc that's just my 2
> cents
> > > :).
> > > > > Let's
> > > > > > > > > gather
> > > > > > > > > > other's input as well.
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > > Viktor
> > > > > > > > > >
> > > > > > > > > > On Fri, May 25, 2018 at 5:43 PM, Vahid S Hashemian <
> > > > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Victor,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for reviewing the KIP.
> > > > > > > > > > >
> > > > > > > > > > > Yes, to minimize the backward compatibility impact,
> there
> > > > > would
> > > > > > be
> > > > > > > no
> > > > > > > > > > harm
> > > > > > > > > > > in letting a stand-alone consumer consume messages
> under
> > a
> > > > ""
> > > > > > > group
> > > > > > > > id
> > > > > > > > > > (as
> > > > > > > > > > > long as there is no offset commit).
> > > > > > > > > > > It would have to knowingly seek to an offset or rely
> on
> > > the
> > > > > > > > > > > auto.offset.reset config for the starting offset.
> > > > > > > > > > > This way the existing functionality would be preserved
> > for
> > > > the
> > > > > > > most
> > > > > > > > > part
> > > > > > > > > > > (with the argument that using the default group id for
> > > > offset
> > > > > > > commit
> > > > > > > > > > > should not be the user's intention in practice).
> > > > > > > > > > >
> > > > > > > > > > > Does it seem reasonable?
> > > > > > > > > > >
> > > > > > > > > > > Thanks.
> > > > > > > > > > > --Vahid
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > > > > > > > > To: dev <de...@kafka.apache.org>
> > > > > > > > > > > Date: 05/25/2018 04:56 AM
> > > > > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the
> > default
> > > > > group
> > > > > > > id
> > > > > > > > > > > behavior in KafkaConsumer
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hi Vahid,
> > > > > > > > > > >
> > > > > > > > > > > When reading your KIP I coldn't fully understand why
> did
> > > you
> > > > > > > decide
> > > > > > > > at
> > > > > > > > > > > failing with "offset_commit" in case #2? Can't we fail
> > > with
> > > > an
> > > > > > > empty
> > > > > > > > > > group
> > > > > > > > > > > id even in "fetch" or "fetch_offset"? What was the
> reason
> > > > for
> > > > > > > > deciding
> > > > > > > > > > to
> > > > > > > > > > > fail at "offset_commit"? Was it because of upgrade
> > > > > compatibility
> > > > > > > > > > reasons?
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Viktor
> > > > > > > > > > >
> > > > > > > > > > > On Thu, May 24, 2018 at 12:06 AM, Ted Yu
> > > > <yu...@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Looks good to me.
> > > > > > > > > > > > -------- Original message --------From: Vahid S
> > > Hashemian
> > > > <
> > > > > > > > > > > > vahidhashemian@us.ibm.com> Date: 5/23/18 11:19 AM
> > > > > > (GMT-08:00)
> > > > > > > > To:
> > > > > > > > > > > > dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-289:
> > > > Improve
> > > > > > the
> > > > > > > > > > default
> > > > > > > > > > > > group id behavior in KafkaConsumer
> > > > > > > > > > > > Hi Ted,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for reviewing the KIP. I updated the KIP and
> > > > > introduced
> > > > > > > an
> > > > > > > > > > error
> > > > > > > > > > > > code for the scenario described.
> > > > > > > > > > > >
> > > > > > > > > > > > --Vahid
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > From: Ted Yu <yu...@gmail.com>
> > > > > > > > > > > > To: dev@kafka.apache.org
> > > > > > > > > > > > Date: 04/27/2018 04:31 PM
> > > > > > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the
> > > default
> > > > > > group
> > > > > > > id
> > > > > > > > > > > > behavior in KafkaConsumer
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > bq. If they attempt an offset commit they will
> receive
> > > an
> > > > > > error.
> > > > > > > > > > > >
> > > > > > > > > > > > Can you outline what specific error would be
> > encountered
> > > ?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Apr 27, 2018 at 2:17 PM, Vahid S Hashemian <
> > > > > > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I have drafted a proposal for improving the
> behavior
> > > of
> > > > > > > > > > KafkaConsumer
> > > > > > > > > > > > when
> > > > > > > > > > > > > using the default group id:
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>
> > >
> > > >
> > > > >
> > > > > > >
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > > 289%3A+Improve+the+default+group+id+behavior+in+
> > > > > > KafkaConsumer
> > > > > > > > > > > > > The proposal based on the issue and suggestion
> > > reported
> > > > in
> > > > > > > > > > KAFKA-6774.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Your feedback is welcome!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks.
> > > > > > > > > > > > > --Vahid
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best,
> > > > > > > Stanislav
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > Best,
> > > > > > Stanislav
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
>
>
>
>
>
Re: [DISCUSS] KIP-289: Improve the default group id behavior in
KafkaConsumer
Posted by Vahid S Hashemian <va...@us.ibm.com>.
Thanks Jason for responding to Colin's concerns.
If there are no other comment / feedback / objection I'll start a vote
soon.
Thanks.
--Vahid
From: Jason Gustafson <ja...@confluent.io>
To: dev <de...@kafka.apache.org>
Date: 07/27/2018 10:38 AM
Subject: Re: [DISCUSS] KIP-289: Improve the default group id
behavior in KafkaConsumer
Hey Colin,
The problem is both that the empty group id is the default value and that
it is actually accepted by the broker for offset commits. Combine that
with
the fact that auto commit is enabled by default and you users get
surprising behavior. If you look at a random Kafka cluster, you'll
probably
find a bunch of inadvertent offset commits for the empty group id. I was
hoping we could distinguish between users who are using the empty group id
as an accident of the default configuration and those who use it
intentionally. By default, there will be no group id and the consumer will
not commit offsets. If a user has actually intentionally used the empty
group id, however, it will continue to work. I actually think there are
probably very few people doing this (maybe even no one), but I thought we
might err on the side of compatibility.
The big incompatible change here is having brokers reject using
assign(...)
> with empty / null group.id.
This is not correct. In the proposal, the broker will only reject the
empty
group id for the new version of OffsetCommit. Older clients, which cannot
be changed, will continue to work because the old versions of the
OffsetCommit API still accept the empty group id. The null group id is
different from the empty group id: it is not allowed in any version of the
API. It is basically a way to indicate that the consumer has no dependence
on the coordinator at all, which we actually have a surprising number of
use cases for. Furthermore, if a user has an actual need for the empty
group id, it will still be allowed. We are just deprecating it.
-Jason
On Fri, Jul 27, 2018 at 9:56 AM, Colin McCabe <cm...@apache.org> wrote:
> Sorry if this is a silly question, but what's the rationale for
switching
> to using null for the default group id, rather than the empty string?
> Continuing to use the empty string seems like less churn. And after
all,
> we're not using the empty string group name for anything else.
>
> The big incompatible change here is having brokers reject using
> assign(...) with empty / null group.id. If I understand correctly, the
> KIP proposes that this change be made on the brokers on the next
> incompatible Kafka release. But that has nothing to do with client
> versions. Why not just have a broker config which controls this? Maybe
"
> allow.assign.empty.group.id", or something like that. At first, the
> default will be true, and then eventually we can flip it over to false.
>
> It seems like the main rationale for tying this behavior to the Kafka
> client version is to force people to stop using the empty group id so
that
> they can upgrade their clients. But it's also possible that people will
> stop upgrading their Kafka clients instead. That would be pretty
negative
> since they'd miss out on any efficiency and feature improvements in the
> new clients and eventually have to do more protocol downgrading, etc.
>
> best,
> Colin
>
>
> On Thu, Jul 26, 2018, at 11:50, Vahid S Hashemian wrote:
> > Hi Jason,
> >
> > That makes sense.
> > I have updated the KIP based on the recent feedback.
> >
> > Thanks!
> > --Vahid
> >
> >
> >
> >
> > From: Jason Gustafson <ja...@confluent.io>
> > To: dev <de...@kafka.apache.org>
> > Date: 07/25/2018 02:23 PM
> > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > behavior in KafkaConsumer
> >
> >
> >
> > Hi Vahid,
> >
> > I was thinking we'd only use the old API version if we had to. That
is,
> > only if the user has explicitly configured "" as the group.id.
> Otherwise,
> > we'd just use the new one. Another option is to just drop support in
the
> > client for the empty group id, but usually we allow a deprecation
period
> > for changes like this.
> >
> > -Jason
> >
> > On Wed, Jul 25, 2018 at 12:49 PM, Vahid S Hashemian <
> > vahidhashemian@us.ibm.com> wrote:
> >
> > > Hi Jason,
> > >
> > > Thanks for additional clarification.
> > >
> > > So the next version of the OffsetCommit API will return an
> > > INVALID_GROUP_ID error for empty group ids; but on the client side
we
> > call
> > > the older version of the client until the next major release.
> > > The table below should summarize this.
> > >
> > > +-----------------------------------------------------+
> > > | Client (group.id="") |
> > > +-----------------------------------------------------+
> > > | pre-2.1 | 2.1 | 3.0 |
> > >
> > +-----+-----------+---------+------------------------+------
> ------------+
> > > | | V5 (cur.) | works | works | works |
> > > + API
> > +-----------+---------+------------------------+------------------+
> > > | | V6 | N/A | N/A (calls V5/warning) |
> INVALID_GROUP_ID
> > |
> > >
> > +-----+-----------+---------+------------------------+------
> ------------+
> > >
> > > Assumptions:
> > > * 2.1: The target release version for this KIP
> > > * 3.0: The next major release
> > >
> > > Please advise if you see an issue; otherwise, I'll update the KIP
> > > accordingly.
> > >
> > > Thanks!
> > > --Vahid
> > >
> > >
> > >
> > >
> > > From: Jason Gustafson <ja...@confluent.io>
> > > To: dev <de...@kafka.apache.org>
> > > Date: 07/25/2018 12:08 AM
> > > Subject: ***UNCHECKED*** Re: [DISCUSS] KIP-289: Improve the
> > default
> > > group id behavior in KafkaConsumer
> > >
> > >
> > >
> > > Hey Vahid,
> > >
> > > Sorry for the confusion. I think we all agree that going forward, we
> > > shouldn't support the empty group id, so the question is just around
> > > compatibility. I think we have to bump the OffsetCommit API version
so
> > > that
> > > old clients which are unknowingly depending on the default empty
group
> > id
> > > will continue to work with new brokers. For new versions of the
> client,
> > we
> > > can either drop support for the empty group id immediately or we can
> > give
> > > users a grace period. I was thinking we would do the latter. We can
> > change
> > > the default group.id, but in the case that a user has explicitly
> > > configured
> > > the empty group, then we can just use an old version of the
> OffsetCommit
> > > API which still supports it. In a future release, we can drop this
> > support
> > > and only use the latest OffsetCommit version. Does that make sense?
> > >
> > > Thanks,
> > > Jason
> > >
> > >
> > > On Tue, Jul 24, 2018 at 12:36 PM, Vahid S Hashemian <
> > > vahidhashemian@us.ibm.com> wrote:
> > >
> > > > Hi Jason,
> > > >
> > > > Thanks for clarifying.
> > > >
> > > > So if we are going to continue supporting the empty group id as
> before
> > > > (with only an addition of a deprecation warning), and disable
> > > > enable.auto.commit for the new default (null) group id on the
client
> > > side,
> > > > do we really need to bump up the OffsetCommit version?
> > > >
> > > > You mentioned "If an explicit empty string is configured for the
> group
> > > id,
> > > > then maybe we keep the current behavior for compatibility" which
> makes
> > > > sense to me, but I find it in conflict with your earlier
suggestion
> > "we
> > > > just need to bump the OffsetCommit request API and only accept the
> > > offset
> > > > commit for older versions.". Maybe I'm missing something?
> > > >
> > > > Thanks!
> > > > --Vahid
> > > >
> > > >
> > > >
> > > >
> > > > From: Jason Gustafson <ja...@confluent.io>
> > > > To: dev <de...@kafka.apache.org>
> > > > Date: 07/23/2018 10:52 PM
> > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group
id
> > > > behavior in KafkaConsumer
> > > >
> > > >
> > > >
> > > > Hey Vahid,
> > > >
> > > > Thanks for the updates. Just to clarify, I was suggesting that we
> > > disable
> > > > enable.auto.commit only if no explicit group.id is configured. If
an
> > > > explicit empty string is configured for the group id, then maybe
we
> > keep
> > > > the current behavior for compatibility. We can log a warning
> > mentioning
> > > > the
> > > > deprecation and we can use the old version of the OffsetCommit API
> > that
> > > > allows the empty group id. In a later release, we can drop this
> > support
> > > in
> > > > the client. Does that seem reasonable?
> > > >
> > > > By the way, instead of using the new ILLEGAL_OFFSET_COMMIT error
> code,
> > > > couldn't we use INVALID_GROUP_ID?
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > >
> > > >
> > > > On Mon, Jul 23, 2018 at 5:14 PM, Stanislav Kozlovski
> > > > <stanislav@confluent.io
> > > > > wrote:
> > > >
> > > > > Hey Vahid,
> > > > >
> > > > > No I don't see an issue with it. I believe it to be the best
> > approach.
> > > > >
> > > > > Best,
> > > > > Stanisav
> > > > >
> > > > > On Mon, Jul 23, 2018 at 12:41 PM Vahid S Hashemian <
> > > > > vahidhashemian@us.ibm.com> wrote:
> > > > >
> > > > > > Hi Stanislav,
> > > > > >
> > > > > > Thanks for the feedback.
> > > > > > Do you see an issue with using `null` as the default group id
(as
> > > > > > addressed by Jason in his response)?
> > > > > > This default group id would not support offset commits and
> > consumers
> > > > > would
> > > > > > use `auto.offset.reset` config when there is no current
offset.
> > > > > >
> > > > > > Thanks.
> > > > > > --Vahid
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > From: Stanislav Kozlovski <st...@confluent.io>
> > > > > > To: dev@kafka.apache.org
> > > > > > Date: 07/20/2018 11:09 AM
> > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default
group
> > id
> > > > > > behavior in KafkaConsumer
> > > > > >
> > > > > >
> > > > > >
> > > > > > I agree with Jason's notion that
> > > > > > > implicit use of the empty group.id to commit offsets is
more
> > > likely
> > > > > to
> > > > > > be causing users unexpected problems than actually providing a
> > > useful
> > > > > > capability.
> > > > > > I was initially confused that this is the behavior when
> > > investigating
> > > > a
> > > > > > new-ish JIRA issue <
> > > > > >
> > > >
> > >
> >
https://issues.apache.org/jira/browse/KAFKA-6758
> >
> > >
> > > >
> > > > > > > about
> > > > > > the same topic.
> > > > > > So, +1 to deprecating "" as a group.id
> > > > > >
> > > > > > The question after that becomes what the *default* value
should
> be
> > -
> > > > > > should
> > > > > > we:
> > > > > > a) treat an unconfigured group.id consumer as a sort of
> > intermittent
> > > > > > consumer where you don't store offsets at all (thereby making
the
> > > user
> > > > > > explicitly sign up for them)
> > > > > > b) have a default value which makes use of them? I sort of
like
> > the
> > > > > > former.
> > > > > >
> > > > > > @Dhruvil, thinking about it at a high-level - yes. I can't
think
> > of
> > > a
> > > > > > situation where it makes sense to name something an empty
string
> > as
> > > > far
> > > > > as
> > > > > > I'm aware - to me it seems like potential for confusion
> > > > > >
> > > > > >
> > > > > > On Fri, Jul 20, 2018 at 10:22 AM Rajini Sivaram
> > > > <rajinisivaram@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > +1 to deprecate use of "" as group.id since it is odd to
have
> a
> > > > > resource
> > > > > > > name that you cannot set ACLs for. Agree, we have to support
> > older
> > > > > > clients
> > > > > > > though.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Rajini
> > > > > > >
> > > > > > > On Fri, Jul 20, 2018 at 5:25 PM, Jason Gustafson
> > > > <ja...@confluent.io>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Vahid,
> > > > > > > >
> > > > > > > > Sorry for getting to this so late. I think there are two
> > things
> > > > here:
> > > > > > > >
> > > > > > > > 1. The use of "" as a groupId has always been a dubious
> > practice
> > > > at
> > > > > > best.
> > > > > > > > We definitely ought to deprecate its use in the client.
> > Perhaps
> > > in
> > > > > the
> > > > > > > next
> > > > > > > > major release, we can remove support completely. However,
> > since
> > > > older
> > > > > > > > clients depend on it, we may have to continue letting the
> > broker
> > > > > > support
> > > > > > > it
> > > > > > > > to some extent. Perhaps we just need to bump the
OffsetCommit
> > > > request
> > > > > > API
> > > > > > > > and only accept the offset commit for older versions. You
> > > probably
> > > > > > have
> > > > > > > to
> > > > > > > > do this anyway if you want to introduce the new error code
> > since
> > > > old
> > > > > > > > clients will not expect it.
> > > > > > > >
> > > > > > > > 2. There should be a way for the consumer to indicate that
it
> > > has
> > > > no
> > > > > > > group
> > > > > > > > id and will not commit offsets. This is an explicit
> > instruction
> > > > that
> > > > > > the
> > > > > > > > consumer should not bother with coordinator lookup and
such.
> > We
> > > > > > currently
> > > > > > > > have some brittle logic in place to let users avoid the
> > > > coordinator
> > > > > > > lookup,
> > > > > > > > but it is a bit error-prone. I was hoping that we could
> change
> > > the
> > > > > > > default
> > > > > > > > value of group.id to be null so that the user had to take
an
> > > > > explicit
> > > > > > > > action to opt into coordinator management (groups or
> offsets).
> > > > > > However,
> > > > > > > it
> > > > > > > > is true that some users may be unknowingly depending on
> offset
> > > > > storage
> > > > > > if
> > > > > > > > they are using both the default group.id and the default
> > > > > > > > enable.auto.commit. Perhaps one option is to disable
> > > > > > enable.auto.commit
> > > > > > > > automatically if no group.id is specified? I am not sure
if
> > > there
> > > > > are
> > > > > > > any
> > > > > > > > drawbacks, but my feeling is that implicit use of the
empty
> > > > group.id
> > > > > > to
> > > > > > > > commit offsets is more likely to be causing users
unexpected
> > > > problems
> > > > > > > than
> > > > > > > > actually providing a useful capability.
> > > > > > > >
> > > > > > > > Thoughts?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Jason
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, May 28, 2018 at 9:50 AM, Vahid S Hashemian <
> > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > >
> > > > > > > > > Hi Viktor,
> > > > > > > > >
> > > > > > > > > Thanks for sharing your opinion.
> > > > > > > > > So you're in favor of disallowing the empty ("") group
id
> > > > > altogether
> > > > > > > > (even
> > > > > > > > > for fetching).
> > > > > > > > > Given that ideally no one should be using the empty
group
> id
> > > (at
> > > > > > least
> > > > > > > in
> > > > > > > > > a production setting) I think the impact would be
minimal
> in
> > > > either
> > > > > > > case.
> > > > > > > > >
> > > > > > > > > But as you said, let's hear what others think and I'd be
> > happy
> > > > to
> > > > > > > modify
> > > > > > > > > the KIP if needed.
> > > > > > > > >
> > > > > > > > > Regards.
> > > > > > > > > --Vahid
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > > > > > > To: dev <de...@kafka.apache.org>
> > > > > > > > > Date: 05/28/2018 05:18 AM
> > > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the
default
> > > group
> > > > id
> > > > > > > > > behavior in KafkaConsumer
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi Vahid,
> > > > > > > > >
> > > > > > > > > (with the argument that using the default group id for
> > offset
> > > > > commit
> > > > > > > > > should not be the user's intention in practice).
> > > > > > > > >
> > > > > > > > > Yea, so in my opinion too this use case doesn't seem too
> > > > practical.
> > > > > > > Also
> > > > > > > > I
> > > > > > > > > think breaking the offset commit is not smaller from
this
> > > > > > perspective
> > > > > > > > than
> > > > > > > > > breaking fetch and offset fetch. If we suppose that
someone
> > > uses
> > > > > the
> > > > > > > > > default group id and we break the offset commit then
that
> > > might
> > > > be
> > > > > > > harder
> > > > > > > > > to detect than breaking the whole thing altogether. (If
we
> > > think
> > > > > > about
> > > > > > > an
> > > > > > > > > upgrade situation.)
> > > > > > > > > So since we think it is not a practical use case, I
think
> it
> > > > would
> > > > > > be
> > > > > > > > > better to break altogether but ofc that's just my 2
cents
> > :).
> > > > Let's
> > > > > > > > gather
> > > > > > > > > other's input as well.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Viktor
> > > > > > > > >
> > > > > > > > > On Fri, May 25, 2018 at 5:43 PM, Vahid S Hashemian <
> > > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > > >
> > > > > > > > > > Hi Victor,
> > > > > > > > > >
> > > > > > > > > > Thanks for reviewing the KIP.
> > > > > > > > > >
> > > > > > > > > > Yes, to minimize the backward compatibility impact,
there
> > > > would
> > > > > be
> > > > > > no
> > > > > > > > > harm
> > > > > > > > > > in letting a stand-alone consumer consume messages
under
> a
> > > ""
> > > > > > group
> > > > > > > id
> > > > > > > > > (as
> > > > > > > > > > long as there is no offset commit).
> > > > > > > > > > It would have to knowingly seek to an offset or rely
on
> > the
> > > > > > > > > > auto.offset.reset config for the starting offset.
> > > > > > > > > > This way the existing functionality would be preserved
> for
> > > the
> > > > > > most
> > > > > > > > part
> > > > > > > > > > (with the argument that using the default group id for
> > > offset
> > > > > > commit
> > > > > > > > > > should not be the user's intention in practice).
> > > > > > > > > >
> > > > > > > > > > Does it seem reasonable?
> > > > > > > > > >
> > > > > > > > > > Thanks.
> > > > > > > > > > --Vahid
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > > > > > > > To: dev <de...@kafka.apache.org>
> > > > > > > > > > Date: 05/25/2018 04:56 AM
> > > > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the
> default
> > > > group
> > > > > > id
> > > > > > > > > > behavior in KafkaConsumer
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hi Vahid,
> > > > > > > > > >
> > > > > > > > > > When reading your KIP I coldn't fully understand why
did
> > you
> > > > > > decide
> > > > > > > at
> > > > > > > > > > failing with "offset_commit" in case #2? Can't we fail
> > with
> > > an
> > > > > > empty
> > > > > > > > > group
> > > > > > > > > > id even in "fetch" or "fetch_offset"? What was the
reason
> > > for
> > > > > > > deciding
> > > > > > > > > to
> > > > > > > > > > fail at "offset_commit"? Was it because of upgrade
> > > > compatibility
> > > > > > > > > reasons?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Viktor
> > > > > > > > > >
> > > > > > > > > > On Thu, May 24, 2018 at 12:06 AM, Ted Yu
> > > <yu...@gmail.com>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Looks good to me.
> > > > > > > > > > > -------- Original message --------From: Vahid S
> > Hashemian
> > > <
> > > > > > > > > > > vahidhashemian@us.ibm.com> Date: 5/23/18 11:19 AM
> > > > > (GMT-08:00)
> > > > > > > To:
> > > > > > > > > > > dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-289:
> > > Improve
> > > > > the
> > > > > > > > > default
> > > > > > > > > > > group id behavior in KafkaConsumer
> > > > > > > > > > > Hi Ted,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for reviewing the KIP. I updated the KIP and
> > > > introduced
> > > > > > an
> > > > > > > > > error
> > > > > > > > > > > code for the scenario described.
> > > > > > > > > > >
> > > > > > > > > > > --Vahid
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > From: Ted Yu <yu...@gmail.com>
> > > > > > > > > > > To: dev@kafka.apache.org
> > > > > > > > > > > Date: 04/27/2018 04:31 PM
> > > > > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the
> > default
> > > > > group
> > > > > > id
> > > > > > > > > > > behavior in KafkaConsumer
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > bq. If they attempt an offset commit they will
receive
> > an
> > > > > error.
> > > > > > > > > > >
> > > > > > > > > > > Can you outline what specific error would be
> encountered
> > ?
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Apr 27, 2018 at 2:17 PM, Vahid S Hashemian <
> > > > > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > I have drafted a proposal for improving the
behavior
> > of
> > > > > > > > > KafkaConsumer
> > > > > > > > > > > when
> > > > > > > > > > > > using the default group id:
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > >
> > >
> >
https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >
> > >
> > > >
> > > > > >
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > 289%3A+Improve+the+default+group+id+behavior+in+
> > > > > KafkaConsumer
> > > > > > > > > > > > The proposal based on the issue and suggestion
> > reported
> > > in
> > > > > > > > > KAFKA-6774.
> > > > > > > > > > > >
> > > > > > > > > > > > Your feedback is welcome!
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks.
> > > > > > > > > > > > --Vahid
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best,
> > > > > > Stanislav
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Best,
> > > > > Stanislav
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer
Posted by Jason Gustafson <ja...@confluent.io>.
Hey Colin,
The problem is both that the empty group id is the default value and that
it is actually accepted by the broker for offset commits. Combine that with
the fact that auto commit is enabled by default and you users get
surprising behavior. If you look at a random Kafka cluster, you'll probably
find a bunch of inadvertent offset commits for the empty group id. I was
hoping we could distinguish between users who are using the empty group id
as an accident of the default configuration and those who use it
intentionally. By default, there will be no group id and the consumer will
not commit offsets. If a user has actually intentionally used the empty
group id, however, it will continue to work. I actually think there are
probably very few people doing this (maybe even no one), but I thought we
might err on the side of compatibility.
The big incompatible change here is having brokers reject using assign(...)
> with empty / null group.id.
This is not correct. In the proposal, the broker will only reject the empty
group id for the new version of OffsetCommit. Older clients, which cannot
be changed, will continue to work because the old versions of the
OffsetCommit API still accept the empty group id. The null group id is
different from the empty group id: it is not allowed in any version of the
API. It is basically a way to indicate that the consumer has no dependence
on the coordinator at all, which we actually have a surprising number of
use cases for. Furthermore, if a user has an actual need for the empty
group id, it will still be allowed. We are just deprecating it.
-Jason
On Fri, Jul 27, 2018 at 9:56 AM, Colin McCabe <cm...@apache.org> wrote:
> Sorry if this is a silly question, but what's the rationale for switching
> to using null for the default group id, rather than the empty string?
> Continuing to use the empty string seems like less churn. And after all,
> we're not using the empty string group name for anything else.
>
> The big incompatible change here is having brokers reject using
> assign(...) with empty / null group.id. If I understand correctly, the
> KIP proposes that this change be made on the brokers on the next
> incompatible Kafka release. But that has nothing to do with client
> versions. Why not just have a broker config which controls this? Maybe "
> allow.assign.empty.group.id", or something like that. At first, the
> default will be true, and then eventually we can flip it over to false.
>
> It seems like the main rationale for tying this behavior to the Kafka
> client version is to force people to stop using the empty group id so that
> they can upgrade their clients. But it's also possible that people will
> stop upgrading their Kafka clients instead. That would be pretty negative
> since they'd miss out on any efficiency and feature improvements in the
> new clients and eventually have to do more protocol downgrading, etc.
>
> best,
> Colin
>
>
> On Thu, Jul 26, 2018, at 11:50, Vahid S Hashemian wrote:
> > Hi Jason,
> >
> > That makes sense.
> > I have updated the KIP based on the recent feedback.
> >
> > Thanks!
> > --Vahid
> >
> >
> >
> >
> > From: Jason Gustafson <ja...@confluent.io>
> > To: dev <de...@kafka.apache.org>
> > Date: 07/25/2018 02:23 PM
> > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > behavior in KafkaConsumer
> >
> >
> >
> > Hi Vahid,
> >
> > I was thinking we'd only use the old API version if we had to. That is,
> > only if the user has explicitly configured "" as the group.id.
> Otherwise,
> > we'd just use the new one. Another option is to just drop support in the
> > client for the empty group id, but usually we allow a deprecation period
> > for changes like this.
> >
> > -Jason
> >
> > On Wed, Jul 25, 2018 at 12:49 PM, Vahid S Hashemian <
> > vahidhashemian@us.ibm.com> wrote:
> >
> > > Hi Jason,
> > >
> > > Thanks for additional clarification.
> > >
> > > So the next version of the OffsetCommit API will return an
> > > INVALID_GROUP_ID error for empty group ids; but on the client side we
> > call
> > > the older version of the client until the next major release.
> > > The table below should summarize this.
> > >
> > > +-----------------------------------------------------+
> > > | Client (group.id="") |
> > > +-----------------------------------------------------+
> > > | pre-2.1 | 2.1 | 3.0 |
> > >
> > +-----+-----------+---------+------------------------+------
> ------------+
> > > | | V5 (cur.) | works | works | works |
> > > + API
> > +-----------+---------+------------------------+------------------+
> > > | | V6 | N/A | N/A (calls V5/warning) |
> INVALID_GROUP_ID
> > |
> > >
> > +-----+-----------+---------+------------------------+------
> ------------+
> > >
> > > Assumptions:
> > > * 2.1: The target release version for this KIP
> > > * 3.0: The next major release
> > >
> > > Please advise if you see an issue; otherwise, I'll update the KIP
> > > accordingly.
> > >
> > > Thanks!
> > > --Vahid
> > >
> > >
> > >
> > >
> > > From: Jason Gustafson <ja...@confluent.io>
> > > To: dev <de...@kafka.apache.org>
> > > Date: 07/25/2018 12:08 AM
> > > Subject: ***UNCHECKED*** Re: [DISCUSS] KIP-289: Improve the
> > default
> > > group id behavior in KafkaConsumer
> > >
> > >
> > >
> > > Hey Vahid,
> > >
> > > Sorry for the confusion. I think we all agree that going forward, we
> > > shouldn't support the empty group id, so the question is just around
> > > compatibility. I think we have to bump the OffsetCommit API version so
> > > that
> > > old clients which are unknowingly depending on the default empty group
> > id
> > > will continue to work with new brokers. For new versions of the
> client,
> > we
> > > can either drop support for the empty group id immediately or we can
> > give
> > > users a grace period. I was thinking we would do the latter. We can
> > change
> > > the default group.id, but in the case that a user has explicitly
> > > configured
> > > the empty group, then we can just use an old version of the
> OffsetCommit
> > > API which still supports it. In a future release, we can drop this
> > support
> > > and only use the latest OffsetCommit version. Does that make sense?
> > >
> > > Thanks,
> > > Jason
> > >
> > >
> > > On Tue, Jul 24, 2018 at 12:36 PM, Vahid S Hashemian <
> > > vahidhashemian@us.ibm.com> wrote:
> > >
> > > > Hi Jason,
> > > >
> > > > Thanks for clarifying.
> > > >
> > > > So if we are going to continue supporting the empty group id as
> before
> > > > (with only an addition of a deprecation warning), and disable
> > > > enable.auto.commit for the new default (null) group id on the client
> > > side,
> > > > do we really need to bump up the OffsetCommit version?
> > > >
> > > > You mentioned "If an explicit empty string is configured for the
> group
> > > id,
> > > > then maybe we keep the current behavior for compatibility" which
> makes
> > > > sense to me, but I find it in conflict with your earlier suggestion
> > "we
> > > > just need to bump the OffsetCommit request API and only accept the
> > > offset
> > > > commit for older versions.". Maybe I'm missing something?
> > > >
> > > > Thanks!
> > > > --Vahid
> > > >
> > > >
> > > >
> > > >
> > > > From: Jason Gustafson <ja...@confluent.io>
> > > > To: dev <de...@kafka.apache.org>
> > > > Date: 07/23/2018 10:52 PM
> > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > > > behavior in KafkaConsumer
> > > >
> > > >
> > > >
> > > > Hey Vahid,
> > > >
> > > > Thanks for the updates. Just to clarify, I was suggesting that we
> > > disable
> > > > enable.auto.commit only if no explicit group.id is configured. If an
> > > > explicit empty string is configured for the group id, then maybe we
> > keep
> > > > the current behavior for compatibility. We can log a warning
> > mentioning
> > > > the
> > > > deprecation and we can use the old version of the OffsetCommit API
> > that
> > > > allows the empty group id. In a later release, we can drop this
> > support
> > > in
> > > > the client. Does that seem reasonable?
> > > >
> > > > By the way, instead of using the new ILLEGAL_OFFSET_COMMIT error
> code,
> > > > couldn't we use INVALID_GROUP_ID?
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > >
> > > >
> > > > On Mon, Jul 23, 2018 at 5:14 PM, Stanislav Kozlovski
> > > > <stanislav@confluent.io
> > > > > wrote:
> > > >
> > > > > Hey Vahid,
> > > > >
> > > > > No I don't see an issue with it. I believe it to be the best
> > approach.
> > > > >
> > > > > Best,
> > > > > Stanisav
> > > > >
> > > > > On Mon, Jul 23, 2018 at 12:41 PM Vahid S Hashemian <
> > > > > vahidhashemian@us.ibm.com> wrote:
> > > > >
> > > > > > Hi Stanislav,
> > > > > >
> > > > > > Thanks for the feedback.
> > > > > > Do you see an issue with using `null` as the default group id (as
> > > > > > addressed by Jason in his response)?
> > > > > > This default group id would not support offset commits and
> > consumers
> > > > > would
> > > > > > use `auto.offset.reset` config when there is no current offset.
> > > > > >
> > > > > > Thanks.
> > > > > > --Vahid
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > From: Stanislav Kozlovski <st...@confluent.io>
> > > > > > To: dev@kafka.apache.org
> > > > > > Date: 07/20/2018 11:09 AM
> > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group
> > id
> > > > > > behavior in KafkaConsumer
> > > > > >
> > > > > >
> > > > > >
> > > > > > I agree with Jason's notion that
> > > > > > > implicit use of the empty group.id to commit offsets is more
> > > likely
> > > > > to
> > > > > > be causing users unexpected problems than actually providing a
> > > useful
> > > > > > capability.
> > > > > > I was initially confused that this is the behavior when
> > > investigating
> > > > a
> > > > > > new-ish JIRA issue <
> > > > > >
> > > >
> > >
> > https://issues.apache.org/jira/browse/KAFKA-6758
> >
> > >
> > > >
> > > > > > > about
> > > > > > the same topic.
> > > > > > So, +1 to deprecating "" as a group.id
> > > > > >
> > > > > > The question after that becomes what the *default* value should
> be
> > -
> > > > > > should
> > > > > > we:
> > > > > > a) treat an unconfigured group.id consumer as a sort of
> > intermittent
> > > > > > consumer where you don't store offsets at all (thereby making the
> > > user
> > > > > > explicitly sign up for them)
> > > > > > b) have a default value which makes use of them? I sort of like
> > the
> > > > > > former.
> > > > > >
> > > > > > @Dhruvil, thinking about it at a high-level - yes. I can't think
> > of
> > > a
> > > > > > situation where it makes sense to name something an empty string
> > as
> > > > far
> > > > > as
> > > > > > I'm aware - to me it seems like potential for confusion
> > > > > >
> > > > > >
> > > > > > On Fri, Jul 20, 2018 at 10:22 AM Rajini Sivaram
> > > > <rajinisivaram@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > +1 to deprecate use of "" as group.id since it is odd to have
> a
> > > > > resource
> > > > > > > name that you cannot set ACLs for. Agree, we have to support
> > older
> > > > > > clients
> > > > > > > though.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Rajini
> > > > > > >
> > > > > > > On Fri, Jul 20, 2018 at 5:25 PM, Jason Gustafson
> > > > <ja...@confluent.io>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Vahid,
> > > > > > > >
> > > > > > > > Sorry for getting to this so late. I think there are two
> > things
> > > > here:
> > > > > > > >
> > > > > > > > 1. The use of "" as a groupId has always been a dubious
> > practice
> > > > at
> > > > > > best.
> > > > > > > > We definitely ought to deprecate its use in the client.
> > Perhaps
> > > in
> > > > > the
> > > > > > > next
> > > > > > > > major release, we can remove support completely. However,
> > since
> > > > older
> > > > > > > > clients depend on it, we may have to continue letting the
> > broker
> > > > > > support
> > > > > > > it
> > > > > > > > to some extent. Perhaps we just need to bump the OffsetCommit
> > > > request
> > > > > > API
> > > > > > > > and only accept the offset commit for older versions. You
> > > probably
> > > > > > have
> > > > > > > to
> > > > > > > > do this anyway if you want to introduce the new error code
> > since
> > > > old
> > > > > > > > clients will not expect it.
> > > > > > > >
> > > > > > > > 2. There should be a way for the consumer to indicate that it
> > > has
> > > > no
> > > > > > > group
> > > > > > > > id and will not commit offsets. This is an explicit
> > instruction
> > > > that
> > > > > > the
> > > > > > > > consumer should not bother with coordinator lookup and such.
> > We
> > > > > > currently
> > > > > > > > have some brittle logic in place to let users avoid the
> > > > coordinator
> > > > > > > lookup,
> > > > > > > > but it is a bit error-prone. I was hoping that we could
> change
> > > the
> > > > > > > default
> > > > > > > > value of group.id to be null so that the user had to take an
> > > > > explicit
> > > > > > > > action to opt into coordinator management (groups or
> offsets).
> > > > > > However,
> > > > > > > it
> > > > > > > > is true that some users may be unknowingly depending on
> offset
> > > > > storage
> > > > > > if
> > > > > > > > they are using both the default group.id and the default
> > > > > > > > enable.auto.commit. Perhaps one option is to disable
> > > > > > enable.auto.commit
> > > > > > > > automatically if no group.id is specified? I am not sure if
> > > there
> > > > > are
> > > > > > > any
> > > > > > > > drawbacks, but my feeling is that implicit use of the empty
> > > > group.id
> > > > > > to
> > > > > > > > commit offsets is more likely to be causing users unexpected
> > > > problems
> > > > > > > than
> > > > > > > > actually providing a useful capability.
> > > > > > > >
> > > > > > > > Thoughts?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Jason
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, May 28, 2018 at 9:50 AM, Vahid S Hashemian <
> > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > >
> > > > > > > > > Hi Viktor,
> > > > > > > > >
> > > > > > > > > Thanks for sharing your opinion.
> > > > > > > > > So you're in favor of disallowing the empty ("") group id
> > > > > altogether
> > > > > > > > (even
> > > > > > > > > for fetching).
> > > > > > > > > Given that ideally no one should be using the empty group
> id
> > > (at
> > > > > > least
> > > > > > > in
> > > > > > > > > a production setting) I think the impact would be minimal
> in
> > > > either
> > > > > > > case.
> > > > > > > > >
> > > > > > > > > But as you said, let's hear what others think and I'd be
> > happy
> > > > to
> > > > > > > modify
> > > > > > > > > the KIP if needed.
> > > > > > > > >
> > > > > > > > > Regards.
> > > > > > > > > --Vahid
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > > > > > > To: dev <de...@kafka.apache.org>
> > > > > > > > > Date: 05/28/2018 05:18 AM
> > > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default
> > > group
> > > > id
> > > > > > > > > behavior in KafkaConsumer
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi Vahid,
> > > > > > > > >
> > > > > > > > > (with the argument that using the default group id for
> > offset
> > > > > commit
> > > > > > > > > should not be the user's intention in practice).
> > > > > > > > >
> > > > > > > > > Yea, so in my opinion too this use case doesn't seem too
> > > > practical.
> > > > > > > Also
> > > > > > > > I
> > > > > > > > > think breaking the offset commit is not smaller from this
> > > > > > perspective
> > > > > > > > than
> > > > > > > > > breaking fetch and offset fetch. If we suppose that someone
> > > uses
> > > > > the
> > > > > > > > > default group id and we break the offset commit then that
> > > might
> > > > be
> > > > > > > harder
> > > > > > > > > to detect than breaking the whole thing altogether. (If we
> > > think
> > > > > > about
> > > > > > > an
> > > > > > > > > upgrade situation.)
> > > > > > > > > So since we think it is not a practical use case, I think
> it
> > > > would
> > > > > > be
> > > > > > > > > better to break altogether but ofc that's just my 2 cents
> > :).
> > > > Let's
> > > > > > > > gather
> > > > > > > > > other's input as well.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Viktor
> > > > > > > > >
> > > > > > > > > On Fri, May 25, 2018 at 5:43 PM, Vahid S Hashemian <
> > > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > > >
> > > > > > > > > > Hi Victor,
> > > > > > > > > >
> > > > > > > > > > Thanks for reviewing the KIP.
> > > > > > > > > >
> > > > > > > > > > Yes, to minimize the backward compatibility impact, there
> > > > would
> > > > > be
> > > > > > no
> > > > > > > > > harm
> > > > > > > > > > in letting a stand-alone consumer consume messages under
> a
> > > ""
> > > > > > group
> > > > > > > id
> > > > > > > > > (as
> > > > > > > > > > long as there is no offset commit).
> > > > > > > > > > It would have to knowingly seek to an offset or rely on
> > the
> > > > > > > > > > auto.offset.reset config for the starting offset.
> > > > > > > > > > This way the existing functionality would be preserved
> for
> > > the
> > > > > > most
> > > > > > > > part
> > > > > > > > > > (with the argument that using the default group id for
> > > offset
> > > > > > commit
> > > > > > > > > > should not be the user's intention in practice).
> > > > > > > > > >
> > > > > > > > > > Does it seem reasonable?
> > > > > > > > > >
> > > > > > > > > > Thanks.
> > > > > > > > > > --Vahid
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > > > > > > > To: dev <de...@kafka.apache.org>
> > > > > > > > > > Date: 05/25/2018 04:56 AM
> > > > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the
> default
> > > > group
> > > > > > id
> > > > > > > > > > behavior in KafkaConsumer
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hi Vahid,
> > > > > > > > > >
> > > > > > > > > > When reading your KIP I coldn't fully understand why did
> > you
> > > > > > decide
> > > > > > > at
> > > > > > > > > > failing with "offset_commit" in case #2? Can't we fail
> > with
> > > an
> > > > > > empty
> > > > > > > > > group
> > > > > > > > > > id even in "fetch" or "fetch_offset"? What was the reason
> > > for
> > > > > > > deciding
> > > > > > > > > to
> > > > > > > > > > fail at "offset_commit"? Was it because of upgrade
> > > > compatibility
> > > > > > > > > reasons?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Viktor
> > > > > > > > > >
> > > > > > > > > > On Thu, May 24, 2018 at 12:06 AM, Ted Yu
> > > <yu...@gmail.com>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Looks good to me.
> > > > > > > > > > > -------- Original message --------From: Vahid S
> > Hashemian
> > > <
> > > > > > > > > > > vahidhashemian@us.ibm.com> Date: 5/23/18 11:19 AM
> > > > > (GMT-08:00)
> > > > > > > To:
> > > > > > > > > > > dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-289:
> > > Improve
> > > > > the
> > > > > > > > > default
> > > > > > > > > > > group id behavior in KafkaConsumer
> > > > > > > > > > > Hi Ted,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for reviewing the KIP. I updated the KIP and
> > > > introduced
> > > > > > an
> > > > > > > > > error
> > > > > > > > > > > code for the scenario described.
> > > > > > > > > > >
> > > > > > > > > > > --Vahid
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > From: Ted Yu <yu...@gmail.com>
> > > > > > > > > > > To: dev@kafka.apache.org
> > > > > > > > > > > Date: 04/27/2018 04:31 PM
> > > > > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the
> > default
> > > > > group
> > > > > > id
> > > > > > > > > > > behavior in KafkaConsumer
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > bq. If they attempt an offset commit they will receive
> > an
> > > > > error.
> > > > > > > > > > >
> > > > > > > > > > > Can you outline what specific error would be
> encountered
> > ?
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Apr 27, 2018 at 2:17 PM, Vahid S Hashemian <
> > > > > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > I have drafted a proposal for improving the behavior
> > of
> > > > > > > > > KafkaConsumer
> > > > > > > > > > > when
> > > > > > > > > > > > using the default group id:
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >
> > >
> > > >
> > > > > >
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > 289%3A+Improve+the+default+group+id+behavior+in+
> > > > > KafkaConsumer
> > > > > > > > > > > > The proposal based on the issue and suggestion
> > reported
> > > in
> > > > > > > > > KAFKA-6774.
> > > > > > > > > > > >
> > > > > > > > > > > > Your feedback is welcome!
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks.
> > > > > > > > > > > > --Vahid
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best,
> > > > > > Stanislav
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Best,
> > > > > Stanislav
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
Re: [DISCUSS] KIP-289: Improve the default group id behavior in
KafkaConsumer
Posted by Colin McCabe <cm...@apache.org>.
Sorry if this is a silly question, but what's the rationale for switching to using null for the default group id, rather than the empty string? Continuing to use the empty string seems like less churn. And after all, we're not using the empty string group name for anything else.
The big incompatible change here is having brokers reject using assign(...) with empty / null group.id. If I understand correctly, the KIP proposes that this change be made on the brokers on the next incompatible Kafka release. But that has nothing to do with client versions. Why not just have a broker config which controls this? Maybe "allow.assign.empty.group.id", or something like that. At first, the default will be true, and then eventually we can flip it over to false.
It seems like the main rationale for tying this behavior to the Kafka client version is to force people to stop using the empty group id so that they can upgrade their clients. But it's also possible that people will stop upgrading their Kafka clients instead. That would be pretty negative since they'd miss out on any efficiency and feature improvements in the new clients and eventually have to do more protocol downgrading, etc.
best,
Colin
On Thu, Jul 26, 2018, at 11:50, Vahid S Hashemian wrote:
> Hi Jason,
>
> That makes sense.
> I have updated the KIP based on the recent feedback.
>
> Thanks!
> --Vahid
>
>
>
>
> From: Jason Gustafson <ja...@confluent.io>
> To: dev <de...@kafka.apache.org>
> Date: 07/25/2018 02:23 PM
> Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> behavior in KafkaConsumer
>
>
>
> Hi Vahid,
>
> I was thinking we'd only use the old API version if we had to. That is,
> only if the user has explicitly configured "" as the group.id. Otherwise,
> we'd just use the new one. Another option is to just drop support in the
> client for the empty group id, but usually we allow a deprecation period
> for changes like this.
>
> -Jason
>
> On Wed, Jul 25, 2018 at 12:49 PM, Vahid S Hashemian <
> vahidhashemian@us.ibm.com> wrote:
>
> > Hi Jason,
> >
> > Thanks for additional clarification.
> >
> > So the next version of the OffsetCommit API will return an
> > INVALID_GROUP_ID error for empty group ids; but on the client side we
> call
> > the older version of the client until the next major release.
> > The table below should summarize this.
> >
> > +-----------------------------------------------------+
> > | Client (group.id="") |
> > +-----------------------------------------------------+
> > | pre-2.1 | 2.1 | 3.0 |
> >
> +-----+-----------+---------+------------------------+------------------+
> > | | V5 (cur.) | works | works | works |
> > + API
> +-----------+---------+------------------------+------------------+
> > | | V6 | N/A | N/A (calls V5/warning) | INVALID_GROUP_ID
> |
> >
> +-----+-----------+---------+------------------------+------------------+
> >
> > Assumptions:
> > * 2.1: The target release version for this KIP
> > * 3.0: The next major release
> >
> > Please advise if you see an issue; otherwise, I'll update the KIP
> > accordingly.
> >
> > Thanks!
> > --Vahid
> >
> >
> >
> >
> > From: Jason Gustafson <ja...@confluent.io>
> > To: dev <de...@kafka.apache.org>
> > Date: 07/25/2018 12:08 AM
> > Subject: ***UNCHECKED*** Re: [DISCUSS] KIP-289: Improve the
> default
> > group id behavior in KafkaConsumer
> >
> >
> >
> > Hey Vahid,
> >
> > Sorry for the confusion. I think we all agree that going forward, we
> > shouldn't support the empty group id, so the question is just around
> > compatibility. I think we have to bump the OffsetCommit API version so
> > that
> > old clients which are unknowingly depending on the default empty group
> id
> > will continue to work with new brokers. For new versions of the client,
> we
> > can either drop support for the empty group id immediately or we can
> give
> > users a grace period. I was thinking we would do the latter. We can
> change
> > the default group.id, but in the case that a user has explicitly
> > configured
> > the empty group, then we can just use an old version of the OffsetCommit
> > API which still supports it. In a future release, we can drop this
> support
> > and only use the latest OffsetCommit version. Does that make sense?
> >
> > Thanks,
> > Jason
> >
> >
> > On Tue, Jul 24, 2018 at 12:36 PM, Vahid S Hashemian <
> > vahidhashemian@us.ibm.com> wrote:
> >
> > > Hi Jason,
> > >
> > > Thanks for clarifying.
> > >
> > > So if we are going to continue supporting the empty group id as before
> > > (with only an addition of a deprecation warning), and disable
> > > enable.auto.commit for the new default (null) group id on the client
> > side,
> > > do we really need to bump up the OffsetCommit version?
> > >
> > > You mentioned "If an explicit empty string is configured for the group
> > id,
> > > then maybe we keep the current behavior for compatibility" which makes
> > > sense to me, but I find it in conflict with your earlier suggestion
> "we
> > > just need to bump the OffsetCommit request API and only accept the
> > offset
> > > commit for older versions.". Maybe I'm missing something?
> > >
> > > Thanks!
> > > --Vahid
> > >
> > >
> > >
> > >
> > > From: Jason Gustafson <ja...@confluent.io>
> > > To: dev <de...@kafka.apache.org>
> > > Date: 07/23/2018 10:52 PM
> > > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > > behavior in KafkaConsumer
> > >
> > >
> > >
> > > Hey Vahid,
> > >
> > > Thanks for the updates. Just to clarify, I was suggesting that we
> > disable
> > > enable.auto.commit only if no explicit group.id is configured. If an
> > > explicit empty string is configured for the group id, then maybe we
> keep
> > > the current behavior for compatibility. We can log a warning
> mentioning
> > > the
> > > deprecation and we can use the old version of the OffsetCommit API
> that
> > > allows the empty group id. In a later release, we can drop this
> support
> > in
> > > the client. Does that seem reasonable?
> > >
> > > By the way, instead of using the new ILLEGAL_OFFSET_COMMIT error code,
> > > couldn't we use INVALID_GROUP_ID?
> > >
> > > Thanks,
> > > Jason
> > >
> > >
> > >
> > > On Mon, Jul 23, 2018 at 5:14 PM, Stanislav Kozlovski
> > > <stanislav@confluent.io
> > > > wrote:
> > >
> > > > Hey Vahid,
> > > >
> > > > No I don't see an issue with it. I believe it to be the best
> approach.
> > > >
> > > > Best,
> > > > Stanisav
> > > >
> > > > On Mon, Jul 23, 2018 at 12:41 PM Vahid S Hashemian <
> > > > vahidhashemian@us.ibm.com> wrote:
> > > >
> > > > > Hi Stanislav,
> > > > >
> > > > > Thanks for the feedback.
> > > > > Do you see an issue with using `null` as the default group id (as
> > > > > addressed by Jason in his response)?
> > > > > This default group id would not support offset commits and
> consumers
> > > > would
> > > > > use `auto.offset.reset` config when there is no current offset.
> > > > >
> > > > > Thanks.
> > > > > --Vahid
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > From: Stanislav Kozlovski <st...@confluent.io>
> > > > > To: dev@kafka.apache.org
> > > > > Date: 07/20/2018 11:09 AM
> > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group
> id
> > > > > behavior in KafkaConsumer
> > > > >
> > > > >
> > > > >
> > > > > I agree with Jason's notion that
> > > > > > implicit use of the empty group.id to commit offsets is more
> > likely
> > > > to
> > > > > be causing users unexpected problems than actually providing a
> > useful
> > > > > capability.
> > > > > I was initially confused that this is the behavior when
> > investigating
> > > a
> > > > > new-ish JIRA issue <
> > > > >
> > >
> >
> https://issues.apache.org/jira/browse/KAFKA-6758
>
> >
> > >
> > > > > > about
> > > > > the same topic.
> > > > > So, +1 to deprecating "" as a group.id
> > > > >
> > > > > The question after that becomes what the *default* value should be
> -
> > > > > should
> > > > > we:
> > > > > a) treat an unconfigured group.id consumer as a sort of
> intermittent
> > > > > consumer where you don't store offsets at all (thereby making the
> > user
> > > > > explicitly sign up for them)
> > > > > b) have a default value which makes use of them? I sort of like
> the
> > > > > former.
> > > > >
> > > > > @Dhruvil, thinking about it at a high-level - yes. I can't think
> of
> > a
> > > > > situation where it makes sense to name something an empty string
> as
> > > far
> > > > as
> > > > > I'm aware - to me it seems like potential for confusion
> > > > >
> > > > >
> > > > > On Fri, Jul 20, 2018 at 10:22 AM Rajini Sivaram
> > > <rajinisivaram@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > +1 to deprecate use of "" as group.id since it is odd to have a
> > > > resource
> > > > > > name that you cannot set ACLs for. Agree, we have to support
> older
> > > > > clients
> > > > > > though.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > > > On Fri, Jul 20, 2018 at 5:25 PM, Jason Gustafson
> > > <ja...@confluent.io>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Vahid,
> > > > > > >
> > > > > > > Sorry for getting to this so late. I think there are two
> things
> > > here:
> > > > > > >
> > > > > > > 1. The use of "" as a groupId has always been a dubious
> practice
> > > at
> > > > > best.
> > > > > > > We definitely ought to deprecate its use in the client.
> Perhaps
> > in
> > > > the
> > > > > > next
> > > > > > > major release, we can remove support completely. However,
> since
> > > older
> > > > > > > clients depend on it, we may have to continue letting the
> broker
> > > > > support
> > > > > > it
> > > > > > > to some extent. Perhaps we just need to bump the OffsetCommit
> > > request
> > > > > API
> > > > > > > and only accept the offset commit for older versions. You
> > probably
> > > > > have
> > > > > > to
> > > > > > > do this anyway if you want to introduce the new error code
> since
> > > old
> > > > > > > clients will not expect it.
> > > > > > >
> > > > > > > 2. There should be a way for the consumer to indicate that it
> > has
> > > no
> > > > > > group
> > > > > > > id and will not commit offsets. This is an explicit
> instruction
> > > that
> > > > > the
> > > > > > > consumer should not bother with coordinator lookup and such.
> We
> > > > > currently
> > > > > > > have some brittle logic in place to let users avoid the
> > > coordinator
> > > > > > lookup,
> > > > > > > but it is a bit error-prone. I was hoping that we could change
> > the
> > > > > > default
> > > > > > > value of group.id to be null so that the user had to take an
> > > > explicit
> > > > > > > action to opt into coordinator management (groups or offsets).
> > > > > However,
> > > > > > it
> > > > > > > is true that some users may be unknowingly depending on offset
> > > > storage
> > > > > if
> > > > > > > they are using both the default group.id and the default
> > > > > > > enable.auto.commit. Perhaps one option is to disable
> > > > > enable.auto.commit
> > > > > > > automatically if no group.id is specified? I am not sure if
> > there
> > > > are
> > > > > > any
> > > > > > > drawbacks, but my feeling is that implicit use of the empty
> > > group.id
> > > > > to
> > > > > > > commit offsets is more likely to be causing users unexpected
> > > problems
> > > > > > than
> > > > > > > actually providing a useful capability.
> > > > > > >
> > > > > > > Thoughts?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Jason
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, May 28, 2018 at 9:50 AM, Vahid S Hashemian <
> > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > >
> > > > > > > > Hi Viktor,
> > > > > > > >
> > > > > > > > Thanks for sharing your opinion.
> > > > > > > > So you're in favor of disallowing the empty ("") group id
> > > > altogether
> > > > > > > (even
> > > > > > > > for fetching).
> > > > > > > > Given that ideally no one should be using the empty group id
> > (at
> > > > > least
> > > > > > in
> > > > > > > > a production setting) I think the impact would be minimal in
> > > either
> > > > > > case.
> > > > > > > >
> > > > > > > > But as you said, let's hear what others think and I'd be
> happy
> > > to
> > > > > > modify
> > > > > > > > the KIP if needed.
> > > > > > > >
> > > > > > > > Regards.
> > > > > > > > --Vahid
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > > > > > To: dev <de...@kafka.apache.org>
> > > > > > > > Date: 05/28/2018 05:18 AM
> > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default
> > group
> > > id
> > > > > > > > behavior in KafkaConsumer
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Hi Vahid,
> > > > > > > >
> > > > > > > > (with the argument that using the default group id for
> offset
> > > > commit
> > > > > > > > should not be the user's intention in practice).
> > > > > > > >
> > > > > > > > Yea, so in my opinion too this use case doesn't seem too
> > > practical.
> > > > > > Also
> > > > > > > I
> > > > > > > > think breaking the offset commit is not smaller from this
> > > > > perspective
> > > > > > > than
> > > > > > > > breaking fetch and offset fetch. If we suppose that someone
> > uses
> > > > the
> > > > > > > > default group id and we break the offset commit then that
> > might
> > > be
> > > > > > harder
> > > > > > > > to detect than breaking the whole thing altogether. (If we
> > think
> > > > > about
> > > > > > an
> > > > > > > > upgrade situation.)
> > > > > > > > So since we think it is not a practical use case, I think it
> > > would
> > > > > be
> > > > > > > > better to break altogether but ofc that's just my 2 cents
> :).
> > > Let's
> > > > > > > gather
> > > > > > > > other's input as well.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Viktor
> > > > > > > >
> > > > > > > > On Fri, May 25, 2018 at 5:43 PM, Vahid S Hashemian <
> > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > >
> > > > > > > > > Hi Victor,
> > > > > > > > >
> > > > > > > > > Thanks for reviewing the KIP.
> > > > > > > > >
> > > > > > > > > Yes, to minimize the backward compatibility impact, there
> > > would
> > > > be
> > > > > no
> > > > > > > > harm
> > > > > > > > > in letting a stand-alone consumer consume messages under a
> > ""
> > > > > group
> > > > > > id
> > > > > > > > (as
> > > > > > > > > long as there is no offset commit).
> > > > > > > > > It would have to knowingly seek to an offset or rely on
> the
> > > > > > > > > auto.offset.reset config for the starting offset.
> > > > > > > > > This way the existing functionality would be preserved for
> > the
> > > > > most
> > > > > > > part
> > > > > > > > > (with the argument that using the default group id for
> > offset
> > > > > commit
> > > > > > > > > should not be the user's intention in practice).
> > > > > > > > >
> > > > > > > > > Does it seem reasonable?
> > > > > > > > >
> > > > > > > > > Thanks.
> > > > > > > > > --Vahid
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > > > > > > To: dev <de...@kafka.apache.org>
> > > > > > > > > Date: 05/25/2018 04:56 AM
> > > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default
> > > group
> > > > > id
> > > > > > > > > behavior in KafkaConsumer
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi Vahid,
> > > > > > > > >
> > > > > > > > > When reading your KIP I coldn't fully understand why did
> you
> > > > > decide
> > > > > > at
> > > > > > > > > failing with "offset_commit" in case #2? Can't we fail
> with
> > an
> > > > > empty
> > > > > > > > group
> > > > > > > > > id even in "fetch" or "fetch_offset"? What was the reason
> > for
> > > > > > deciding
> > > > > > > > to
> > > > > > > > > fail at "offset_commit"? Was it because of upgrade
> > > compatibility
> > > > > > > > reasons?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Viktor
> > > > > > > > >
> > > > > > > > > On Thu, May 24, 2018 at 12:06 AM, Ted Yu
> > <yu...@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Looks good to me.
> > > > > > > > > > -------- Original message --------From: Vahid S
> Hashemian
> > <
> > > > > > > > > > vahidhashemian@us.ibm.com> Date: 5/23/18 11:19 AM
> > > > (GMT-08:00)
> > > > > > To:
> > > > > > > > > > dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-289:
> > Improve
> > > > the
> > > > > > > > default
> > > > > > > > > > group id behavior in KafkaConsumer
> > > > > > > > > > Hi Ted,
> > > > > > > > > >
> > > > > > > > > > Thanks for reviewing the KIP. I updated the KIP and
> > > introduced
> > > > > an
> > > > > > > > error
> > > > > > > > > > code for the scenario described.
> > > > > > > > > >
> > > > > > > > > > --Vahid
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > From: Ted Yu <yu...@gmail.com>
> > > > > > > > > > To: dev@kafka.apache.org
> > > > > > > > > > Date: 04/27/2018 04:31 PM
> > > > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the
> default
> > > > group
> > > > > id
> > > > > > > > > > behavior in KafkaConsumer
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > bq. If they attempt an offset commit they will receive
> an
> > > > error.
> > > > > > > > > >
> > > > > > > > > > Can you outline what specific error would be encountered
> ?
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > >
> > > > > > > > > > On Fri, Apr 27, 2018 at 2:17 PM, Vahid S Hashemian <
> > > > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > I have drafted a proposal for improving the behavior
> of
> > > > > > > > KafkaConsumer
> > > > > > > > > > when
> > > > > > > > > > > using the default group id:
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>
> >
> > >
> > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > 289%3A+Improve+the+default+group+id+behavior+in+
> > > > KafkaConsumer
> > > > > > > > > > > The proposal based on the issue and suggestion
> reported
> > in
> > > > > > > > KAFKA-6774.
> > > > > > > > > > >
> > > > > > > > > > > Your feedback is welcome!
> > > > > > > > > > >
> > > > > > > > > > > Thanks.
> > > > > > > > > > > --Vahid
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best,
> > > > > Stanislav
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > Best,
> > > > Stanislav
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
Re: [DISCUSS] KIP-289: Improve the default group id behavior in
KafkaConsumer
Posted by Vahid S Hashemian <va...@us.ibm.com>.
Hi Jason,
That makes sense.
I have updated the KIP based on the recent feedback.
Thanks!
--Vahid
From: Jason Gustafson <ja...@confluent.io>
To: dev <de...@kafka.apache.org>
Date: 07/25/2018 02:23 PM
Subject: Re: [DISCUSS] KIP-289: Improve the default group id
behavior in KafkaConsumer
Hi Vahid,
I was thinking we'd only use the old API version if we had to. That is,
only if the user has explicitly configured "" as the group.id. Otherwise,
we'd just use the new one. Another option is to just drop support in the
client for the empty group id, but usually we allow a deprecation period
for changes like this.
-Jason
On Wed, Jul 25, 2018 at 12:49 PM, Vahid S Hashemian <
vahidhashemian@us.ibm.com> wrote:
> Hi Jason,
>
> Thanks for additional clarification.
>
> So the next version of the OffsetCommit API will return an
> INVALID_GROUP_ID error for empty group ids; but on the client side we
call
> the older version of the client until the next major release.
> The table below should summarize this.
>
> +-----------------------------------------------------+
> | Client (group.id="") |
> +-----------------------------------------------------+
> | pre-2.1 | 2.1 | 3.0 |
>
+-----+-----------+---------+------------------------+------------------+
> | | V5 (cur.) | works | works | works |
> + API
+-----------+---------+------------------------+------------------+
> | | V6 | N/A | N/A (calls V5/warning) | INVALID_GROUP_ID
|
>
+-----+-----------+---------+------------------------+------------------+
>
> Assumptions:
> * 2.1: The target release version for this KIP
> * 3.0: The next major release
>
> Please advise if you see an issue; otherwise, I'll update the KIP
> accordingly.
>
> Thanks!
> --Vahid
>
>
>
>
> From: Jason Gustafson <ja...@confluent.io>
> To: dev <de...@kafka.apache.org>
> Date: 07/25/2018 12:08 AM
> Subject: ***UNCHECKED*** Re: [DISCUSS] KIP-289: Improve the
default
> group id behavior in KafkaConsumer
>
>
>
> Hey Vahid,
>
> Sorry for the confusion. I think we all agree that going forward, we
> shouldn't support the empty group id, so the question is just around
> compatibility. I think we have to bump the OffsetCommit API version so
> that
> old clients which are unknowingly depending on the default empty group
id
> will continue to work with new brokers. For new versions of the client,
we
> can either drop support for the empty group id immediately or we can
give
> users a grace period. I was thinking we would do the latter. We can
change
> the default group.id, but in the case that a user has explicitly
> configured
> the empty group, then we can just use an old version of the OffsetCommit
> API which still supports it. In a future release, we can drop this
support
> and only use the latest OffsetCommit version. Does that make sense?
>
> Thanks,
> Jason
>
>
> On Tue, Jul 24, 2018 at 12:36 PM, Vahid S Hashemian <
> vahidhashemian@us.ibm.com> wrote:
>
> > Hi Jason,
> >
> > Thanks for clarifying.
> >
> > So if we are going to continue supporting the empty group id as before
> > (with only an addition of a deprecation warning), and disable
> > enable.auto.commit for the new default (null) group id on the client
> side,
> > do we really need to bump up the OffsetCommit version?
> >
> > You mentioned "If an explicit empty string is configured for the group
> id,
> > then maybe we keep the current behavior for compatibility" which makes
> > sense to me, but I find it in conflict with your earlier suggestion
"we
> > just need to bump the OffsetCommit request API and only accept the
> offset
> > commit for older versions.". Maybe I'm missing something?
> >
> > Thanks!
> > --Vahid
> >
> >
> >
> >
> > From: Jason Gustafson <ja...@confluent.io>
> > To: dev <de...@kafka.apache.org>
> > Date: 07/23/2018 10:52 PM
> > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > behavior in KafkaConsumer
> >
> >
> >
> > Hey Vahid,
> >
> > Thanks for the updates. Just to clarify, I was suggesting that we
> disable
> > enable.auto.commit only if no explicit group.id is configured. If an
> > explicit empty string is configured for the group id, then maybe we
keep
> > the current behavior for compatibility. We can log a warning
mentioning
> > the
> > deprecation and we can use the old version of the OffsetCommit API
that
> > allows the empty group id. In a later release, we can drop this
support
> in
> > the client. Does that seem reasonable?
> >
> > By the way, instead of using the new ILLEGAL_OFFSET_COMMIT error code,
> > couldn't we use INVALID_GROUP_ID?
> >
> > Thanks,
> > Jason
> >
> >
> >
> > On Mon, Jul 23, 2018 at 5:14 PM, Stanislav Kozlovski
> > <stanislav@confluent.io
> > > wrote:
> >
> > > Hey Vahid,
> > >
> > > No I don't see an issue with it. I believe it to be the best
approach.
> > >
> > > Best,
> > > Stanisav
> > >
> > > On Mon, Jul 23, 2018 at 12:41 PM Vahid S Hashemian <
> > > vahidhashemian@us.ibm.com> wrote:
> > >
> > > > Hi Stanislav,
> > > >
> > > > Thanks for the feedback.
> > > > Do you see an issue with using `null` as the default group id (as
> > > > addressed by Jason in his response)?
> > > > This default group id would not support offset commits and
consumers
> > > would
> > > > use `auto.offset.reset` config when there is no current offset.
> > > >
> > > > Thanks.
> > > > --Vahid
> > > >
> > > >
> > > >
> > > >
> > > > From: Stanislav Kozlovski <st...@confluent.io>
> > > > To: dev@kafka.apache.org
> > > > Date: 07/20/2018 11:09 AM
> > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group
id
> > > > behavior in KafkaConsumer
> > > >
> > > >
> > > >
> > > > I agree with Jason's notion that
> > > > > implicit use of the empty group.id to commit offsets is more
> likely
> > > to
> > > > be causing users unexpected problems than actually providing a
> useful
> > > > capability.
> > > > I was initially confused that this is the behavior when
> investigating
> > a
> > > > new-ish JIRA issue <
> > > >
> >
>
https://issues.apache.org/jira/browse/KAFKA-6758
>
> >
> > > > > about
> > > > the same topic.
> > > > So, +1 to deprecating "" as a group.id
> > > >
> > > > The question after that becomes what the *default* value should be
-
> > > > should
> > > > we:
> > > > a) treat an unconfigured group.id consumer as a sort of
intermittent
> > > > consumer where you don't store offsets at all (thereby making the
> user
> > > > explicitly sign up for them)
> > > > b) have a default value which makes use of them? I sort of like
the
> > > > former.
> > > >
> > > > @Dhruvil, thinking about it at a high-level - yes. I can't think
of
> a
> > > > situation where it makes sense to name something an empty string
as
> > far
> > > as
> > > > I'm aware - to me it seems like potential for confusion
> > > >
> > > >
> > > > On Fri, Jul 20, 2018 at 10:22 AM Rajini Sivaram
> > <rajinisivaram@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > +1 to deprecate use of "" as group.id since it is odd to have a
> > > resource
> > > > > name that you cannot set ACLs for. Agree, we have to support
older
> > > > clients
> > > > > though.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Rajini
> > > > >
> > > > > On Fri, Jul 20, 2018 at 5:25 PM, Jason Gustafson
> > <ja...@confluent.io>
> > > > > wrote:
> > > > >
> > > > > > Hi Vahid,
> > > > > >
> > > > > > Sorry for getting to this so late. I think there are two
things
> > here:
> > > > > >
> > > > > > 1. The use of "" as a groupId has always been a dubious
practice
> > at
> > > > best.
> > > > > > We definitely ought to deprecate its use in the client.
Perhaps
> in
> > > the
> > > > > next
> > > > > > major release, we can remove support completely. However,
since
> > older
> > > > > > clients depend on it, we may have to continue letting the
broker
> > > > support
> > > > > it
> > > > > > to some extent. Perhaps we just need to bump the OffsetCommit
> > request
> > > > API
> > > > > > and only accept the offset commit for older versions. You
> probably
> > > > have
> > > > > to
> > > > > > do this anyway if you want to introduce the new error code
since
> > old
> > > > > > clients will not expect it.
> > > > > >
> > > > > > 2. There should be a way for the consumer to indicate that it
> has
> > no
> > > > > group
> > > > > > id and will not commit offsets. This is an explicit
instruction
> > that
> > > > the
> > > > > > consumer should not bother with coordinator lookup and such.
We
> > > > currently
> > > > > > have some brittle logic in place to let users avoid the
> > coordinator
> > > > > lookup,
> > > > > > but it is a bit error-prone. I was hoping that we could change
> the
> > > > > default
> > > > > > value of group.id to be null so that the user had to take an
> > > explicit
> > > > > > action to opt into coordinator management (groups or offsets).
> > > > However,
> > > > > it
> > > > > > is true that some users may be unknowingly depending on offset
> > > storage
> > > > if
> > > > > > they are using both the default group.id and the default
> > > > > > enable.auto.commit. Perhaps one option is to disable
> > > > enable.auto.commit
> > > > > > automatically if no group.id is specified? I am not sure if
> there
> > > are
> > > > > any
> > > > > > drawbacks, but my feeling is that implicit use of the empty
> > group.id
> > > > to
> > > > > > commit offsets is more likely to be causing users unexpected
> > problems
> > > > > than
> > > > > > actually providing a useful capability.
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > Thanks,
> > > > > > Jason
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, May 28, 2018 at 9:50 AM, Vahid S Hashemian <
> > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > >
> > > > > > > Hi Viktor,
> > > > > > >
> > > > > > > Thanks for sharing your opinion.
> > > > > > > So you're in favor of disallowing the empty ("") group id
> > > altogether
> > > > > > (even
> > > > > > > for fetching).
> > > > > > > Given that ideally no one should be using the empty group id
> (at
> > > > least
> > > > > in
> > > > > > > a production setting) I think the impact would be minimal in
> > either
> > > > > case.
> > > > > > >
> > > > > > > But as you said, let's hear what others think and I'd be
happy
> > to
> > > > > modify
> > > > > > > the KIP if needed.
> > > > > > >
> > > > > > > Regards.
> > > > > > > --Vahid
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > > > > To: dev <de...@kafka.apache.org>
> > > > > > > Date: 05/28/2018 05:18 AM
> > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default
> group
> > id
> > > > > > > behavior in KafkaConsumer
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Hi Vahid,
> > > > > > >
> > > > > > > (with the argument that using the default group id for
offset
> > > commit
> > > > > > > should not be the user's intention in practice).
> > > > > > >
> > > > > > > Yea, so in my opinion too this use case doesn't seem too
> > practical.
> > > > > Also
> > > > > > I
> > > > > > > think breaking the offset commit is not smaller from this
> > > > perspective
> > > > > > than
> > > > > > > breaking fetch and offset fetch. If we suppose that someone
> uses
> > > the
> > > > > > > default group id and we break the offset commit then that
> might
> > be
> > > > > harder
> > > > > > > to detect than breaking the whole thing altogether. (If we
> think
> > > > about
> > > > > an
> > > > > > > upgrade situation.)
> > > > > > > So since we think it is not a practical use case, I think it
> > would
> > > > be
> > > > > > > better to break altogether but ofc that's just my 2 cents
:).
> > Let's
> > > > > > gather
> > > > > > > other's input as well.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Viktor
> > > > > > >
> > > > > > > On Fri, May 25, 2018 at 5:43 PM, Vahid S Hashemian <
> > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > >
> > > > > > > > Hi Victor,
> > > > > > > >
> > > > > > > > Thanks for reviewing the KIP.
> > > > > > > >
> > > > > > > > Yes, to minimize the backward compatibility impact, there
> > would
> > > be
> > > > no
> > > > > > > harm
> > > > > > > > in letting a stand-alone consumer consume messages under a
> ""
> > > > group
> > > > > id
> > > > > > > (as
> > > > > > > > long as there is no offset commit).
> > > > > > > > It would have to knowingly seek to an offset or rely on
the
> > > > > > > > auto.offset.reset config for the starting offset.
> > > > > > > > This way the existing functionality would be preserved for
> the
> > > > most
> > > > > > part
> > > > > > > > (with the argument that using the default group id for
> offset
> > > > commit
> > > > > > > > should not be the user's intention in practice).
> > > > > > > >
> > > > > > > > Does it seem reasonable?
> > > > > > > >
> > > > > > > > Thanks.
> > > > > > > > --Vahid
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > > > > > To: dev <de...@kafka.apache.org>
> > > > > > > > Date: 05/25/2018 04:56 AM
> > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default
> > group
> > > > id
> > > > > > > > behavior in KafkaConsumer
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Hi Vahid,
> > > > > > > >
> > > > > > > > When reading your KIP I coldn't fully understand why did
you
> > > > decide
> > > > > at
> > > > > > > > failing with "offset_commit" in case #2? Can't we fail
with
> an
> > > > empty
> > > > > > > group
> > > > > > > > id even in "fetch" or "fetch_offset"? What was the reason
> for
> > > > > deciding
> > > > > > > to
> > > > > > > > fail at "offset_commit"? Was it because of upgrade
> > compatibility
> > > > > > > reasons?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Viktor
> > > > > > > >
> > > > > > > > On Thu, May 24, 2018 at 12:06 AM, Ted Yu
> <yu...@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Looks good to me.
> > > > > > > > > -------- Original message --------From: Vahid S
Hashemian
> <
> > > > > > > > > vahidhashemian@us.ibm.com> Date: 5/23/18 11:19 AM
> > > (GMT-08:00)
> > > > > To:
> > > > > > > > > dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-289:
> Improve
> > > the
> > > > > > > default
> > > > > > > > > group id behavior in KafkaConsumer
> > > > > > > > > Hi Ted,
> > > > > > > > >
> > > > > > > > > Thanks for reviewing the KIP. I updated the KIP and
> > introduced
> > > > an
> > > > > > > error
> > > > > > > > > code for the scenario described.
> > > > > > > > >
> > > > > > > > > --Vahid
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > From: Ted Yu <yu...@gmail.com>
> > > > > > > > > To: dev@kafka.apache.org
> > > > > > > > > Date: 04/27/2018 04:31 PM
> > > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the
default
> > > group
> > > > id
> > > > > > > > > behavior in KafkaConsumer
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > bq. If they attempt an offset commit they will receive
an
> > > error.
> > > > > > > > >
> > > > > > > > > Can you outline what specific error would be encountered
?
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > >
> > > > > > > > > On Fri, Apr 27, 2018 at 2:17 PM, Vahid S Hashemian <
> > > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > I have drafted a proposal for improving the behavior
of
> > > > > > > KafkaConsumer
> > > > > > > > > when
> > > > > > > > > > using the default group id:
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > >
> >
>
https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>
> >
> > > >
> > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > > 289%3A+Improve+the+default+group+id+behavior+in+
> > > KafkaConsumer
> > > > > > > > > > The proposal based on the issue and suggestion
reported
> in
> > > > > > > KAFKA-6774.
> > > > > > > > > >
> > > > > > > > > > Your feedback is welcome!
> > > > > > > > > >
> > > > > > > > > > Thanks.
> > > > > > > > > > --Vahid
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Best,
> > > > Stanislav
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > > --
> > > Best,
> > > Stanislav
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer
Posted by Jason Gustafson <ja...@confluent.io>.
Hi Vahid,
I was thinking we'd only use the old API version if we had to. That is,
only if the user has explicitly configured "" as the group.id. Otherwise,
we'd just use the new one. Another option is to just drop support in the
client for the empty group id, but usually we allow a deprecation period
for changes like this.
-Jason
On Wed, Jul 25, 2018 at 12:49 PM, Vahid S Hashemian <
vahidhashemian@us.ibm.com> wrote:
> Hi Jason,
>
> Thanks for additional clarification.
>
> So the next version of the OffsetCommit API will return an
> INVALID_GROUP_ID error for empty group ids; but on the client side we call
> the older version of the client until the next major release.
> The table below should summarize this.
>
> +-----------------------------------------------------+
> | Client (group.id="") |
> +-----------------------------------------------------+
> | pre-2.1 | 2.1 | 3.0 |
> +-----+-----------+---------+------------------------+------------------+
> | | V5 (cur.) | works | works | works |
> + API +-----------+---------+------------------------+------------------+
> | | V6 | N/A | N/A (calls V5/warning) | INVALID_GROUP_ID |
> +-----+-----------+---------+------------------------+------------------+
>
> Assumptions:
> * 2.1: The target release version for this KIP
> * 3.0: The next major release
>
> Please advise if you see an issue; otherwise, I'll update the KIP
> accordingly.
>
> Thanks!
> --Vahid
>
>
>
>
> From: Jason Gustafson <ja...@confluent.io>
> To: dev <de...@kafka.apache.org>
> Date: 07/25/2018 12:08 AM
> Subject: ***UNCHECKED*** Re: [DISCUSS] KIP-289: Improve the default
> group id behavior in KafkaConsumer
>
>
>
> Hey Vahid,
>
> Sorry for the confusion. I think we all agree that going forward, we
> shouldn't support the empty group id, so the question is just around
> compatibility. I think we have to bump the OffsetCommit API version so
> that
> old clients which are unknowingly depending on the default empty group id
> will continue to work with new brokers. For new versions of the client, we
> can either drop support for the empty group id immediately or we can give
> users a grace period. I was thinking we would do the latter. We can change
> the default group.id, but in the case that a user has explicitly
> configured
> the empty group, then we can just use an old version of the OffsetCommit
> API which still supports it. In a future release, we can drop this support
> and only use the latest OffsetCommit version. Does that make sense?
>
> Thanks,
> Jason
>
>
> On Tue, Jul 24, 2018 at 12:36 PM, Vahid S Hashemian <
> vahidhashemian@us.ibm.com> wrote:
>
> > Hi Jason,
> >
> > Thanks for clarifying.
> >
> > So if we are going to continue supporting the empty group id as before
> > (with only an addition of a deprecation warning), and disable
> > enable.auto.commit for the new default (null) group id on the client
> side,
> > do we really need to bump up the OffsetCommit version?
> >
> > You mentioned "If an explicit empty string is configured for the group
> id,
> > then maybe we keep the current behavior for compatibility" which makes
> > sense to me, but I find it in conflict with your earlier suggestion "we
> > just need to bump the OffsetCommit request API and only accept the
> offset
> > commit for older versions.". Maybe I'm missing something?
> >
> > Thanks!
> > --Vahid
> >
> >
> >
> >
> > From: Jason Gustafson <ja...@confluent.io>
> > To: dev <de...@kafka.apache.org>
> > Date: 07/23/2018 10:52 PM
> > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > behavior in KafkaConsumer
> >
> >
> >
> > Hey Vahid,
> >
> > Thanks for the updates. Just to clarify, I was suggesting that we
> disable
> > enable.auto.commit only if no explicit group.id is configured. If an
> > explicit empty string is configured for the group id, then maybe we keep
> > the current behavior for compatibility. We can log a warning mentioning
> > the
> > deprecation and we can use the old version of the OffsetCommit API that
> > allows the empty group id. In a later release, we can drop this support
> in
> > the client. Does that seem reasonable?
> >
> > By the way, instead of using the new ILLEGAL_OFFSET_COMMIT error code,
> > couldn't we use INVALID_GROUP_ID?
> >
> > Thanks,
> > Jason
> >
> >
> >
> > On Mon, Jul 23, 2018 at 5:14 PM, Stanislav Kozlovski
> > <stanislav@confluent.io
> > > wrote:
> >
> > > Hey Vahid,
> > >
> > > No I don't see an issue with it. I believe it to be the best approach.
> > >
> > > Best,
> > > Stanisav
> > >
> > > On Mon, Jul 23, 2018 at 12:41 PM Vahid S Hashemian <
> > > vahidhashemian@us.ibm.com> wrote:
> > >
> > > > Hi Stanislav,
> > > >
> > > > Thanks for the feedback.
> > > > Do you see an issue with using `null` as the default group id (as
> > > > addressed by Jason in his response)?
> > > > This default group id would not support offset commits and consumers
> > > would
> > > > use `auto.offset.reset` config when there is no current offset.
> > > >
> > > > Thanks.
> > > > --Vahid
> > > >
> > > >
> > > >
> > > >
> > > > From: Stanislav Kozlovski <st...@confluent.io>
> > > > To: dev@kafka.apache.org
> > > > Date: 07/20/2018 11:09 AM
> > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > > > behavior in KafkaConsumer
> > > >
> > > >
> > > >
> > > > I agree with Jason's notion that
> > > > > implicit use of the empty group.id to commit offsets is more
> likely
> > > to
> > > > be causing users unexpected problems than actually providing a
> useful
> > > > capability.
> > > > I was initially confused that this is the behavior when
> investigating
> > a
> > > > new-ish JIRA issue <
> > > >
> >
> https://issues.apache.org/jira/browse/KAFKA-6758
>
> >
> > > > > about
> > > > the same topic.
> > > > So, +1 to deprecating "" as a group.id
> > > >
> > > > The question after that becomes what the *default* value should be -
> > > > should
> > > > we:
> > > > a) treat an unconfigured group.id consumer as a sort of intermittent
> > > > consumer where you don't store offsets at all (thereby making the
> user
> > > > explicitly sign up for them)
> > > > b) have a default value which makes use of them? I sort of like the
> > > > former.
> > > >
> > > > @Dhruvil, thinking about it at a high-level - yes. I can't think of
> a
> > > > situation where it makes sense to name something an empty string as
> > far
> > > as
> > > > I'm aware - to me it seems like potential for confusion
> > > >
> > > >
> > > > On Fri, Jul 20, 2018 at 10:22 AM Rajini Sivaram
> > <rajinisivaram@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > +1 to deprecate use of "" as group.id since it is odd to have a
> > > resource
> > > > > name that you cannot set ACLs for. Agree, we have to support older
> > > > clients
> > > > > though.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Rajini
> > > > >
> > > > > On Fri, Jul 20, 2018 at 5:25 PM, Jason Gustafson
> > <ja...@confluent.io>
> > > > > wrote:
> > > > >
> > > > > > Hi Vahid,
> > > > > >
> > > > > > Sorry for getting to this so late. I think there are two things
> > here:
> > > > > >
> > > > > > 1. The use of "" as a groupId has always been a dubious practice
> > at
> > > > best.
> > > > > > We definitely ought to deprecate its use in the client. Perhaps
> in
> > > the
> > > > > next
> > > > > > major release, we can remove support completely. However, since
> > older
> > > > > > clients depend on it, we may have to continue letting the broker
> > > > support
> > > > > it
> > > > > > to some extent. Perhaps we just need to bump the OffsetCommit
> > request
> > > > API
> > > > > > and only accept the offset commit for older versions. You
> probably
> > > > have
> > > > > to
> > > > > > do this anyway if you want to introduce the new error code since
> > old
> > > > > > clients will not expect it.
> > > > > >
> > > > > > 2. There should be a way for the consumer to indicate that it
> has
> > no
> > > > > group
> > > > > > id and will not commit offsets. This is an explicit instruction
> > that
> > > > the
> > > > > > consumer should not bother with coordinator lookup and such. We
> > > > currently
> > > > > > have some brittle logic in place to let users avoid the
> > coordinator
> > > > > lookup,
> > > > > > but it is a bit error-prone. I was hoping that we could change
> the
> > > > > default
> > > > > > value of group.id to be null so that the user had to take an
> > > explicit
> > > > > > action to opt into coordinator management (groups or offsets).
> > > > However,
> > > > > it
> > > > > > is true that some users may be unknowingly depending on offset
> > > storage
> > > > if
> > > > > > they are using both the default group.id and the default
> > > > > > enable.auto.commit. Perhaps one option is to disable
> > > > enable.auto.commit
> > > > > > automatically if no group.id is specified? I am not sure if
> there
> > > are
> > > > > any
> > > > > > drawbacks, but my feeling is that implicit use of the empty
> > group.id
> > > > to
> > > > > > commit offsets is more likely to be causing users unexpected
> > problems
> > > > > than
> > > > > > actually providing a useful capability.
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > Thanks,
> > > > > > Jason
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, May 28, 2018 at 9:50 AM, Vahid S Hashemian <
> > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > >
> > > > > > > Hi Viktor,
> > > > > > >
> > > > > > > Thanks for sharing your opinion.
> > > > > > > So you're in favor of disallowing the empty ("") group id
> > > altogether
> > > > > > (even
> > > > > > > for fetching).
> > > > > > > Given that ideally no one should be using the empty group id
> (at
> > > > least
> > > > > in
> > > > > > > a production setting) I think the impact would be minimal in
> > either
> > > > > case.
> > > > > > >
> > > > > > > But as you said, let's hear what others think and I'd be happy
> > to
> > > > > modify
> > > > > > > the KIP if needed.
> > > > > > >
> > > > > > > Regards.
> > > > > > > --Vahid
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > > > > To: dev <de...@kafka.apache.org>
> > > > > > > Date: 05/28/2018 05:18 AM
> > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default
> group
> > id
> > > > > > > behavior in KafkaConsumer
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Hi Vahid,
> > > > > > >
> > > > > > > (with the argument that using the default group id for offset
> > > commit
> > > > > > > should not be the user's intention in practice).
> > > > > > >
> > > > > > > Yea, so in my opinion too this use case doesn't seem too
> > practical.
> > > > > Also
> > > > > > I
> > > > > > > think breaking the offset commit is not smaller from this
> > > > perspective
> > > > > > than
> > > > > > > breaking fetch and offset fetch. If we suppose that someone
> uses
> > > the
> > > > > > > default group id and we break the offset commit then that
> might
> > be
> > > > > harder
> > > > > > > to detect than breaking the whole thing altogether. (If we
> think
> > > > about
> > > > > an
> > > > > > > upgrade situation.)
> > > > > > > So since we think it is not a practical use case, I think it
> > would
> > > > be
> > > > > > > better to break altogether but ofc that's just my 2 cents :).
> > Let's
> > > > > > gather
> > > > > > > other's input as well.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Viktor
> > > > > > >
> > > > > > > On Fri, May 25, 2018 at 5:43 PM, Vahid S Hashemian <
> > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > >
> > > > > > > > Hi Victor,
> > > > > > > >
> > > > > > > > Thanks for reviewing the KIP.
> > > > > > > >
> > > > > > > > Yes, to minimize the backward compatibility impact, there
> > would
> > > be
> > > > no
> > > > > > > harm
> > > > > > > > in letting a stand-alone consumer consume messages under a
> ""
> > > > group
> > > > > id
> > > > > > > (as
> > > > > > > > long as there is no offset commit).
> > > > > > > > It would have to knowingly seek to an offset or rely on the
> > > > > > > > auto.offset.reset config for the starting offset.
> > > > > > > > This way the existing functionality would be preserved for
> the
> > > > most
> > > > > > part
> > > > > > > > (with the argument that using the default group id for
> offset
> > > > commit
> > > > > > > > should not be the user's intention in practice).
> > > > > > > >
> > > > > > > > Does it seem reasonable?
> > > > > > > >
> > > > > > > > Thanks.
> > > > > > > > --Vahid
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > > > > > To: dev <de...@kafka.apache.org>
> > > > > > > > Date: 05/25/2018 04:56 AM
> > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default
> > group
> > > > id
> > > > > > > > behavior in KafkaConsumer
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Hi Vahid,
> > > > > > > >
> > > > > > > > When reading your KIP I coldn't fully understand why did you
> > > > decide
> > > > > at
> > > > > > > > failing with "offset_commit" in case #2? Can't we fail with
> an
> > > > empty
> > > > > > > group
> > > > > > > > id even in "fetch" or "fetch_offset"? What was the reason
> for
> > > > > deciding
> > > > > > > to
> > > > > > > > fail at "offset_commit"? Was it because of upgrade
> > compatibility
> > > > > > > reasons?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Viktor
> > > > > > > >
> > > > > > > > On Thu, May 24, 2018 at 12:06 AM, Ted Yu
> <yu...@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Looks good to me.
> > > > > > > > > -------- Original message --------From: Vahid S Hashemian
> <
> > > > > > > > > vahidhashemian@us.ibm.com> Date: 5/23/18 11:19 AM
> > > (GMT-08:00)
> > > > > To:
> > > > > > > > > dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-289:
> Improve
> > > the
> > > > > > > default
> > > > > > > > > group id behavior in KafkaConsumer
> > > > > > > > > Hi Ted,
> > > > > > > > >
> > > > > > > > > Thanks for reviewing the KIP. I updated the KIP and
> > introduced
> > > > an
> > > > > > > error
> > > > > > > > > code for the scenario described.
> > > > > > > > >
> > > > > > > > > --Vahid
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > From: Ted Yu <yu...@gmail.com>
> > > > > > > > > To: dev@kafka.apache.org
> > > > > > > > > Date: 04/27/2018 04:31 PM
> > > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default
> > > group
> > > > id
> > > > > > > > > behavior in KafkaConsumer
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > bq. If they attempt an offset commit they will receive an
> > > error.
> > > > > > > > >
> > > > > > > > > Can you outline what specific error would be encountered ?
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > >
> > > > > > > > > On Fri, Apr 27, 2018 at 2:17 PM, Vahid S Hashemian <
> > > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > I have drafted a proposal for improving the behavior of
> > > > > > > KafkaConsumer
> > > > > > > > > when
> > > > > > > > > > using the default group id:
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>
> >
> > > >
> > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > > 289%3A+Improve+the+default+group+id+behavior+in+
> > > KafkaConsumer
> > > > > > > > > > The proposal based on the issue and suggestion reported
> in
> > > > > > > KAFKA-6774.
> > > > > > > > > >
> > > > > > > > > > Your feedback is welcome!
> > > > > > > > > >
> > > > > > > > > > Thanks.
> > > > > > > > > > --Vahid
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Best,
> > > > Stanislav
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > > --
> > > Best,
> > > Stanislav
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
Re: [DISCUSS] KIP-289: Improve the default group id behavior in
KafkaConsumer
Posted by Vahid S Hashemian <va...@us.ibm.com>.
Hi Jason,
Thanks for additional clarification.
So the next version of the OffsetCommit API will return an
INVALID_GROUP_ID error for empty group ids; but on the client side we call
the older version of the client until the next major release.
The table below should summarize this.
+-----------------------------------------------------+
| Client (group.id="") |
+-----------------------------------------------------+
| pre-2.1 | 2.1 | 3.0 |
+-----+-----------+---------+------------------------+------------------+
| | V5 (cur.) | works | works | works |
+ API +-----------+---------+------------------------+------------------+
| | V6 | N/A | N/A (calls V5/warning) | INVALID_GROUP_ID |
+-----+-----------+---------+------------------------+------------------+
Assumptions:
* 2.1: The target release version for this KIP
* 3.0: The next major release
Please advise if you see an issue; otherwise, I'll update the KIP
accordingly.
Thanks!
--Vahid
From: Jason Gustafson <ja...@confluent.io>
To: dev <de...@kafka.apache.org>
Date: 07/25/2018 12:08 AM
Subject: ***UNCHECKED*** Re: [DISCUSS] KIP-289: Improve the default
group id behavior in KafkaConsumer
Hey Vahid,
Sorry for the confusion. I think we all agree that going forward, we
shouldn't support the empty group id, so the question is just around
compatibility. I think we have to bump the OffsetCommit API version so
that
old clients which are unknowingly depending on the default empty group id
will continue to work with new brokers. For new versions of the client, we
can either drop support for the empty group id immediately or we can give
users a grace period. I was thinking we would do the latter. We can change
the default group.id, but in the case that a user has explicitly
configured
the empty group, then we can just use an old version of the OffsetCommit
API which still supports it. In a future release, we can drop this support
and only use the latest OffsetCommit version. Does that make sense?
Thanks,
Jason
On Tue, Jul 24, 2018 at 12:36 PM, Vahid S Hashemian <
vahidhashemian@us.ibm.com> wrote:
> Hi Jason,
>
> Thanks for clarifying.
>
> So if we are going to continue supporting the empty group id as before
> (with only an addition of a deprecation warning), and disable
> enable.auto.commit for the new default (null) group id on the client
side,
> do we really need to bump up the OffsetCommit version?
>
> You mentioned "If an explicit empty string is configured for the group
id,
> then maybe we keep the current behavior for compatibility" which makes
> sense to me, but I find it in conflict with your earlier suggestion "we
> just need to bump the OffsetCommit request API and only accept the
offset
> commit for older versions.". Maybe I'm missing something?
>
> Thanks!
> --Vahid
>
>
>
>
> From: Jason Gustafson <ja...@confluent.io>
> To: dev <de...@kafka.apache.org>
> Date: 07/23/2018 10:52 PM
> Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> behavior in KafkaConsumer
>
>
>
> Hey Vahid,
>
> Thanks for the updates. Just to clarify, I was suggesting that we
disable
> enable.auto.commit only if no explicit group.id is configured. If an
> explicit empty string is configured for the group id, then maybe we keep
> the current behavior for compatibility. We can log a warning mentioning
> the
> deprecation and we can use the old version of the OffsetCommit API that
> allows the empty group id. In a later release, we can drop this support
in
> the client. Does that seem reasonable?
>
> By the way, instead of using the new ILLEGAL_OFFSET_COMMIT error code,
> couldn't we use INVALID_GROUP_ID?
>
> Thanks,
> Jason
>
>
>
> On Mon, Jul 23, 2018 at 5:14 PM, Stanislav Kozlovski
> <stanislav@confluent.io
> > wrote:
>
> > Hey Vahid,
> >
> > No I don't see an issue with it. I believe it to be the best approach.
> >
> > Best,
> > Stanisav
> >
> > On Mon, Jul 23, 2018 at 12:41 PM Vahid S Hashemian <
> > vahidhashemian@us.ibm.com> wrote:
> >
> > > Hi Stanislav,
> > >
> > > Thanks for the feedback.
> > > Do you see an issue with using `null` as the default group id (as
> > > addressed by Jason in his response)?
> > > This default group id would not support offset commits and consumers
> > would
> > > use `auto.offset.reset` config when there is no current offset.
> > >
> > > Thanks.
> > > --Vahid
> > >
> > >
> > >
> > >
> > > From: Stanislav Kozlovski <st...@confluent.io>
> > > To: dev@kafka.apache.org
> > > Date: 07/20/2018 11:09 AM
> > > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > > behavior in KafkaConsumer
> > >
> > >
> > >
> > > I agree with Jason's notion that
> > > > implicit use of the empty group.id to commit offsets is more
likely
> > to
> > > be causing users unexpected problems than actually providing a
useful
> > > capability.
> > > I was initially confused that this is the behavior when
investigating
> a
> > > new-ish JIRA issue <
> > >
>
https://issues.apache.org/jira/browse/KAFKA-6758
>
> > > > about
> > > the same topic.
> > > So, +1 to deprecating "" as a group.id
> > >
> > > The question after that becomes what the *default* value should be -
> > > should
> > > we:
> > > a) treat an unconfigured group.id consumer as a sort of intermittent
> > > consumer where you don't store offsets at all (thereby making the
user
> > > explicitly sign up for them)
> > > b) have a default value which makes use of them? I sort of like the
> > > former.
> > >
> > > @Dhruvil, thinking about it at a high-level - yes. I can't think of
a
> > > situation where it makes sense to name something an empty string as
> far
> > as
> > > I'm aware - to me it seems like potential for confusion
> > >
> > >
> > > On Fri, Jul 20, 2018 at 10:22 AM Rajini Sivaram
> <rajinisivaram@gmail.com
> > >
> > > wrote:
> > >
> > > > +1 to deprecate use of "" as group.id since it is odd to have a
> > resource
> > > > name that you cannot set ACLs for. Agree, we have to support older
> > > clients
> > > > though.
> > > >
> > > > Thanks,
> > > >
> > > > Rajini
> > > >
> > > > On Fri, Jul 20, 2018 at 5:25 PM, Jason Gustafson
> <ja...@confluent.io>
> > > > wrote:
> > > >
> > > > > Hi Vahid,
> > > > >
> > > > > Sorry for getting to this so late. I think there are two things
> here:
> > > > >
> > > > > 1. The use of "" as a groupId has always been a dubious practice
> at
> > > best.
> > > > > We definitely ought to deprecate its use in the client. Perhaps
in
> > the
> > > > next
> > > > > major release, we can remove support completely. However, since
> older
> > > > > clients depend on it, we may have to continue letting the broker
> > > support
> > > > it
> > > > > to some extent. Perhaps we just need to bump the OffsetCommit
> request
> > > API
> > > > > and only accept the offset commit for older versions. You
probably
> > > have
> > > > to
> > > > > do this anyway if you want to introduce the new error code since
> old
> > > > > clients will not expect it.
> > > > >
> > > > > 2. There should be a way for the consumer to indicate that it
has
> no
> > > > group
> > > > > id and will not commit offsets. This is an explicit instruction
> that
> > > the
> > > > > consumer should not bother with coordinator lookup and such. We
> > > currently
> > > > > have some brittle logic in place to let users avoid the
> coordinator
> > > > lookup,
> > > > > but it is a bit error-prone. I was hoping that we could change
the
> > > > default
> > > > > value of group.id to be null so that the user had to take an
> > explicit
> > > > > action to opt into coordinator management (groups or offsets).
> > > However,
> > > > it
> > > > > is true that some users may be unknowingly depending on offset
> > storage
> > > if
> > > > > they are using both the default group.id and the default
> > > > > enable.auto.commit. Perhaps one option is to disable
> > > enable.auto.commit
> > > > > automatically if no group.id is specified? I am not sure if
there
> > are
> > > > any
> > > > > drawbacks, but my feeling is that implicit use of the empty
> group.id
> > > to
> > > > > commit offsets is more likely to be causing users unexpected
> problems
> > > > than
> > > > > actually providing a useful capability.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > Thanks,
> > > > > Jason
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, May 28, 2018 at 9:50 AM, Vahid S Hashemian <
> > > > > vahidhashemian@us.ibm.com> wrote:
> > > > >
> > > > > > Hi Viktor,
> > > > > >
> > > > > > Thanks for sharing your opinion.
> > > > > > So you're in favor of disallowing the empty ("") group id
> > altogether
> > > > > (even
> > > > > > for fetching).
> > > > > > Given that ideally no one should be using the empty group id
(at
> > > least
> > > > in
> > > > > > a production setting) I think the impact would be minimal in
> either
> > > > case.
> > > > > >
> > > > > > But as you said, let's hear what others think and I'd be happy
> to
> > > > modify
> > > > > > the KIP if needed.
> > > > > >
> > > > > > Regards.
> > > > > > --Vahid
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > > > To: dev <de...@kafka.apache.org>
> > > > > > Date: 05/28/2018 05:18 AM
> > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default
group
> id
> > > > > > behavior in KafkaConsumer
> > > > > >
> > > > > >
> > > > > >
> > > > > > Hi Vahid,
> > > > > >
> > > > > > (with the argument that using the default group id for offset
> > commit
> > > > > > should not be the user's intention in practice).
> > > > > >
> > > > > > Yea, so in my opinion too this use case doesn't seem too
> practical.
> > > > Also
> > > > > I
> > > > > > think breaking the offset commit is not smaller from this
> > > perspective
> > > > > than
> > > > > > breaking fetch and offset fetch. If we suppose that someone
uses
> > the
> > > > > > default group id and we break the offset commit then that
might
> be
> > > > harder
> > > > > > to detect than breaking the whole thing altogether. (If we
think
> > > about
> > > > an
> > > > > > upgrade situation.)
> > > > > > So since we think it is not a practical use case, I think it
> would
> > > be
> > > > > > better to break altogether but ofc that's just my 2 cents :).
> Let's
> > > > > gather
> > > > > > other's input as well.
> > > > > >
> > > > > > Cheers,
> > > > > > Viktor
> > > > > >
> > > > > > On Fri, May 25, 2018 at 5:43 PM, Vahid S Hashemian <
> > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > >
> > > > > > > Hi Victor,
> > > > > > >
> > > > > > > Thanks for reviewing the KIP.
> > > > > > >
> > > > > > > Yes, to minimize the backward compatibility impact, there
> would
> > be
> > > no
> > > > > > harm
> > > > > > > in letting a stand-alone consumer consume messages under a
""
> > > group
> > > > id
> > > > > > (as
> > > > > > > long as there is no offset commit).
> > > > > > > It would have to knowingly seek to an offset or rely on the
> > > > > > > auto.offset.reset config for the starting offset.
> > > > > > > This way the existing functionality would be preserved for
the
> > > most
> > > > > part
> > > > > > > (with the argument that using the default group id for
offset
> > > commit
> > > > > > > should not be the user's intention in practice).
> > > > > > >
> > > > > > > Does it seem reasonable?
> > > > > > >
> > > > > > > Thanks.
> > > > > > > --Vahid
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > > > > To: dev <de...@kafka.apache.org>
> > > > > > > Date: 05/25/2018 04:56 AM
> > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default
> group
> > > id
> > > > > > > behavior in KafkaConsumer
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Hi Vahid,
> > > > > > >
> > > > > > > When reading your KIP I coldn't fully understand why did you
> > > decide
> > > > at
> > > > > > > failing with "offset_commit" in case #2? Can't we fail with
an
> > > empty
> > > > > > group
> > > > > > > id even in "fetch" or "fetch_offset"? What was the reason
for
> > > > deciding
> > > > > > to
> > > > > > > fail at "offset_commit"? Was it because of upgrade
> compatibility
> > > > > > reasons?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Viktor
> > > > > > >
> > > > > > > On Thu, May 24, 2018 at 12:06 AM, Ted Yu
<yu...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > Looks good to me.
> > > > > > > > -------- Original message --------From: Vahid S Hashemian
<
> > > > > > > > vahidhashemian@us.ibm.com> Date: 5/23/18 11:19 AM
> > (GMT-08:00)
> > > > To:
> > > > > > > > dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-289:
Improve
> > the
> > > > > > default
> > > > > > > > group id behavior in KafkaConsumer
> > > > > > > > Hi Ted,
> > > > > > > >
> > > > > > > > Thanks for reviewing the KIP. I updated the KIP and
> introduced
> > > an
> > > > > > error
> > > > > > > > code for the scenario described.
> > > > > > > >
> > > > > > > > --Vahid
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > From: Ted Yu <yu...@gmail.com>
> > > > > > > > To: dev@kafka.apache.org
> > > > > > > > Date: 04/27/2018 04:31 PM
> > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default
> > group
> > > id
> > > > > > > > behavior in KafkaConsumer
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > bq. If they attempt an offset commit they will receive an
> > error.
> > > > > > > >
> > > > > > > > Can you outline what specific error would be encountered ?
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > > On Fri, Apr 27, 2018 at 2:17 PM, Vahid S Hashemian <
> > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > I have drafted a proposal for improving the behavior of
> > > > > > KafkaConsumer
> > > > > > > > when
> > > > > > > > > using the default group id:
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > >
>
https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>
> > >
> > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > > 289%3A+Improve+the+default+group+id+behavior+in+
> > KafkaConsumer
> > > > > > > > > The proposal based on the issue and suggestion reported
in
> > > > > > KAFKA-6774.
> > > > > > > > >
> > > > > > > > > Your feedback is welcome!
> > > > > > > > >
> > > > > > > > > Thanks.
> > > > > > > > > --Vahid
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > Best,
> > > Stanislav
> > >
> > >
> > >
> > >
> > >
> >
> > --
> > Best,
> > Stanislav
> >
>
>
>
>
>
***UNCHECKED*** Re: [DISCUSS] KIP-289: Improve the default group id
behavior in KafkaConsumer
Posted by Jason Gustafson <ja...@confluent.io>.
Hey Vahid,
Sorry for the confusion. I think we all agree that going forward, we
shouldn't support the empty group id, so the question is just around
compatibility. I think we have to bump the OffsetCommit API version so that
old clients which are unknowingly depending on the default empty group id
will continue to work with new brokers. For new versions of the client, we
can either drop support for the empty group id immediately or we can give
users a grace period. I was thinking we would do the latter. We can change
the default group.id, but in the case that a user has explicitly configured
the empty group, then we can just use an old version of the OffsetCommit
API which still supports it. In a future release, we can drop this support
and only use the latest OffsetCommit version. Does that make sense?
Thanks,
Jason
On Tue, Jul 24, 2018 at 12:36 PM, Vahid S Hashemian <
vahidhashemian@us.ibm.com> wrote:
> Hi Jason,
>
> Thanks for clarifying.
>
> So if we are going to continue supporting the empty group id as before
> (with only an addition of a deprecation warning), and disable
> enable.auto.commit for the new default (null) group id on the client side,
> do we really need to bump up the OffsetCommit version?
>
> You mentioned "If an explicit empty string is configured for the group id,
> then maybe we keep the current behavior for compatibility" which makes
> sense to me, but I find it in conflict with your earlier suggestion "we
> just need to bump the OffsetCommit request API and only accept the offset
> commit for older versions.". Maybe I'm missing something?
>
> Thanks!
> --Vahid
>
>
>
>
> From: Jason Gustafson <ja...@confluent.io>
> To: dev <de...@kafka.apache.org>
> Date: 07/23/2018 10:52 PM
> Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> behavior in KafkaConsumer
>
>
>
> Hey Vahid,
>
> Thanks for the updates. Just to clarify, I was suggesting that we disable
> enable.auto.commit only if no explicit group.id is configured. If an
> explicit empty string is configured for the group id, then maybe we keep
> the current behavior for compatibility. We can log a warning mentioning
> the
> deprecation and we can use the old version of the OffsetCommit API that
> allows the empty group id. In a later release, we can drop this support in
> the client. Does that seem reasonable?
>
> By the way, instead of using the new ILLEGAL_OFFSET_COMMIT error code,
> couldn't we use INVALID_GROUP_ID?
>
> Thanks,
> Jason
>
>
>
> On Mon, Jul 23, 2018 at 5:14 PM, Stanislav Kozlovski
> <stanislav@confluent.io
> > wrote:
>
> > Hey Vahid,
> >
> > No I don't see an issue with it. I believe it to be the best approach.
> >
> > Best,
> > Stanisav
> >
> > On Mon, Jul 23, 2018 at 12:41 PM Vahid S Hashemian <
> > vahidhashemian@us.ibm.com> wrote:
> >
> > > Hi Stanislav,
> > >
> > > Thanks for the feedback.
> > > Do you see an issue with using `null` as the default group id (as
> > > addressed by Jason in his response)?
> > > This default group id would not support offset commits and consumers
> > would
> > > use `auto.offset.reset` config when there is no current offset.
> > >
> > > Thanks.
> > > --Vahid
> > >
> > >
> > >
> > >
> > > From: Stanislav Kozlovski <st...@confluent.io>
> > > To: dev@kafka.apache.org
> > > Date: 07/20/2018 11:09 AM
> > > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > > behavior in KafkaConsumer
> > >
> > >
> > >
> > > I agree with Jason's notion that
> > > > implicit use of the empty group.id to commit offsets is more likely
> > to
> > > be causing users unexpected problems than actually providing a useful
> > > capability.
> > > I was initially confused that this is the behavior when investigating
> a
> > > new-ish JIRA issue <
> > >
> https://issues.apache.org/jira/browse/KAFKA-6758
>
> > > > about
> > > the same topic.
> > > So, +1 to deprecating "" as a group.id
> > >
> > > The question after that becomes what the *default* value should be -
> > > should
> > > we:
> > > a) treat an unconfigured group.id consumer as a sort of intermittent
> > > consumer where you don't store offsets at all (thereby making the user
> > > explicitly sign up for them)
> > > b) have a default value which makes use of them? I sort of like the
> > > former.
> > >
> > > @Dhruvil, thinking about it at a high-level - yes. I can't think of a
> > > situation where it makes sense to name something an empty string as
> far
> > as
> > > I'm aware - to me it seems like potential for confusion
> > >
> > >
> > > On Fri, Jul 20, 2018 at 10:22 AM Rajini Sivaram
> <rajinisivaram@gmail.com
> > >
> > > wrote:
> > >
> > > > +1 to deprecate use of "" as group.id since it is odd to have a
> > resource
> > > > name that you cannot set ACLs for. Agree, we have to support older
> > > clients
> > > > though.
> > > >
> > > > Thanks,
> > > >
> > > > Rajini
> > > >
> > > > On Fri, Jul 20, 2018 at 5:25 PM, Jason Gustafson
> <ja...@confluent.io>
> > > > wrote:
> > > >
> > > > > Hi Vahid,
> > > > >
> > > > > Sorry for getting to this so late. I think there are two things
> here:
> > > > >
> > > > > 1. The use of "" as a groupId has always been a dubious practice
> at
> > > best.
> > > > > We definitely ought to deprecate its use in the client. Perhaps in
> > the
> > > > next
> > > > > major release, we can remove support completely. However, since
> older
> > > > > clients depend on it, we may have to continue letting the broker
> > > support
> > > > it
> > > > > to some extent. Perhaps we just need to bump the OffsetCommit
> request
> > > API
> > > > > and only accept the offset commit for older versions. You probably
> > > have
> > > > to
> > > > > do this anyway if you want to introduce the new error code since
> old
> > > > > clients will not expect it.
> > > > >
> > > > > 2. There should be a way for the consumer to indicate that it has
> no
> > > > group
> > > > > id and will not commit offsets. This is an explicit instruction
> that
> > > the
> > > > > consumer should not bother with coordinator lookup and such. We
> > > currently
> > > > > have some brittle logic in place to let users avoid the
> coordinator
> > > > lookup,
> > > > > but it is a bit error-prone. I was hoping that we could change the
> > > > default
> > > > > value of group.id to be null so that the user had to take an
> > explicit
> > > > > action to opt into coordinator management (groups or offsets).
> > > However,
> > > > it
> > > > > is true that some users may be unknowingly depending on offset
> > storage
> > > if
> > > > > they are using both the default group.id and the default
> > > > > enable.auto.commit. Perhaps one option is to disable
> > > enable.auto.commit
> > > > > automatically if no group.id is specified? I am not sure if there
> > are
> > > > any
> > > > > drawbacks, but my feeling is that implicit use of the empty
> group.id
> > > to
> > > > > commit offsets is more likely to be causing users unexpected
> problems
> > > > than
> > > > > actually providing a useful capability.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > Thanks,
> > > > > Jason
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, May 28, 2018 at 9:50 AM, Vahid S Hashemian <
> > > > > vahidhashemian@us.ibm.com> wrote:
> > > > >
> > > > > > Hi Viktor,
> > > > > >
> > > > > > Thanks for sharing your opinion.
> > > > > > So you're in favor of disallowing the empty ("") group id
> > altogether
> > > > > (even
> > > > > > for fetching).
> > > > > > Given that ideally no one should be using the empty group id (at
> > > least
> > > > in
> > > > > > a production setting) I think the impact would be minimal in
> either
> > > > case.
> > > > > >
> > > > > > But as you said, let's hear what others think and I'd be happy
> to
> > > > modify
> > > > > > the KIP if needed.
> > > > > >
> > > > > > Regards.
> > > > > > --Vahid
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > > > To: dev <de...@kafka.apache.org>
> > > > > > Date: 05/28/2018 05:18 AM
> > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group
> id
> > > > > > behavior in KafkaConsumer
> > > > > >
> > > > > >
> > > > > >
> > > > > > Hi Vahid,
> > > > > >
> > > > > > (with the argument that using the default group id for offset
> > commit
> > > > > > should not be the user's intention in practice).
> > > > > >
> > > > > > Yea, so in my opinion too this use case doesn't seem too
> practical.
> > > > Also
> > > > > I
> > > > > > think breaking the offset commit is not smaller from this
> > > perspective
> > > > > than
> > > > > > breaking fetch and offset fetch. If we suppose that someone uses
> > the
> > > > > > default group id and we break the offset commit then that might
> be
> > > > harder
> > > > > > to detect than breaking the whole thing altogether. (If we think
> > > about
> > > > an
> > > > > > upgrade situation.)
> > > > > > So since we think it is not a practical use case, I think it
> would
> > > be
> > > > > > better to break altogether but ofc that's just my 2 cents :).
> Let's
> > > > > gather
> > > > > > other's input as well.
> > > > > >
> > > > > > Cheers,
> > > > > > Viktor
> > > > > >
> > > > > > On Fri, May 25, 2018 at 5:43 PM, Vahid S Hashemian <
> > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > >
> > > > > > > Hi Victor,
> > > > > > >
> > > > > > > Thanks for reviewing the KIP.
> > > > > > >
> > > > > > > Yes, to minimize the backward compatibility impact, there
> would
> > be
> > > no
> > > > > > harm
> > > > > > > in letting a stand-alone consumer consume messages under a ""
> > > group
> > > > id
> > > > > > (as
> > > > > > > long as there is no offset commit).
> > > > > > > It would have to knowingly seek to an offset or rely on the
> > > > > > > auto.offset.reset config for the starting offset.
> > > > > > > This way the existing functionality would be preserved for the
> > > most
> > > > > part
> > > > > > > (with the argument that using the default group id for offset
> > > commit
> > > > > > > should not be the user's intention in practice).
> > > > > > >
> > > > > > > Does it seem reasonable?
> > > > > > >
> > > > > > > Thanks.
> > > > > > > --Vahid
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > > > > To: dev <de...@kafka.apache.org>
> > > > > > > Date: 05/25/2018 04:56 AM
> > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default
> group
> > > id
> > > > > > > behavior in KafkaConsumer
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Hi Vahid,
> > > > > > >
> > > > > > > When reading your KIP I coldn't fully understand why did you
> > > decide
> > > > at
> > > > > > > failing with "offset_commit" in case #2? Can't we fail with an
> > > empty
> > > > > > group
> > > > > > > id even in "fetch" or "fetch_offset"? What was the reason for
> > > > deciding
> > > > > > to
> > > > > > > fail at "offset_commit"? Was it because of upgrade
> compatibility
> > > > > > reasons?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Viktor
> > > > > > >
> > > > > > > On Thu, May 24, 2018 at 12:06 AM, Ted Yu <yu...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > Looks good to me.
> > > > > > > > -------- Original message --------From: Vahid S Hashemian <
> > > > > > > > vahidhashemian@us.ibm.com> Date: 5/23/18 11:19 AM
> > (GMT-08:00)
> > > > To:
> > > > > > > > dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-289: Improve
> > the
> > > > > > default
> > > > > > > > group id behavior in KafkaConsumer
> > > > > > > > Hi Ted,
> > > > > > > >
> > > > > > > > Thanks for reviewing the KIP. I updated the KIP and
> introduced
> > > an
> > > > > > error
> > > > > > > > code for the scenario described.
> > > > > > > >
> > > > > > > > --Vahid
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > From: Ted Yu <yu...@gmail.com>
> > > > > > > > To: dev@kafka.apache.org
> > > > > > > > Date: 04/27/2018 04:31 PM
> > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default
> > group
> > > id
> > > > > > > > behavior in KafkaConsumer
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > bq. If they attempt an offset commit they will receive an
> > error.
> > > > > > > >
> > > > > > > > Can you outline what specific error would be encountered ?
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > > On Fri, Apr 27, 2018 at 2:17 PM, Vahid S Hashemian <
> > > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > I have drafted a proposal for improving the behavior of
> > > > > > KafkaConsumer
> > > > > > > > when
> > > > > > > > > using the default group id:
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>
> > >
> > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > > 289%3A+Improve+the+default+group+id+behavior+in+
> > KafkaConsumer
> > > > > > > > > The proposal based on the issue and suggestion reported in
> > > > > > KAFKA-6774.
> > > > > > > > >
> > > > > > > > > Your feedback is welcome!
> > > > > > > > >
> > > > > > > > > Thanks.
> > > > > > > > > --Vahid
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > Best,
> > > Stanislav
> > >
> > >
> > >
> > >
> > >
> >
> > --
> > Best,
> > Stanislav
> >
>
>
>
>
>
Re: [DISCUSS] KIP-289: Improve the default group id behavior in
KafkaConsumer
Posted by Vahid S Hashemian <va...@us.ibm.com>.
Hi Jason,
Thanks for clarifying.
So if we are going to continue supporting the empty group id as before
(with only an addition of a deprecation warning), and disable
enable.auto.commit for the new default (null) group id on the client side,
do we really need to bump up the OffsetCommit version?
You mentioned "If an explicit empty string is configured for the group id,
then maybe we keep the current behavior for compatibility" which makes
sense to me, but I find it in conflict with your earlier suggestion "we
just need to bump the OffsetCommit request API and only accept the offset
commit for older versions.". Maybe I'm missing something?
Thanks!
--Vahid
From: Jason Gustafson <ja...@confluent.io>
To: dev <de...@kafka.apache.org>
Date: 07/23/2018 10:52 PM
Subject: Re: [DISCUSS] KIP-289: Improve the default group id
behavior in KafkaConsumer
Hey Vahid,
Thanks for the updates. Just to clarify, I was suggesting that we disable
enable.auto.commit only if no explicit group.id is configured. If an
explicit empty string is configured for the group id, then maybe we keep
the current behavior for compatibility. We can log a warning mentioning
the
deprecation and we can use the old version of the OffsetCommit API that
allows the empty group id. In a later release, we can drop this support in
the client. Does that seem reasonable?
By the way, instead of using the new ILLEGAL_OFFSET_COMMIT error code,
couldn't we use INVALID_GROUP_ID?
Thanks,
Jason
On Mon, Jul 23, 2018 at 5:14 PM, Stanislav Kozlovski
<stanislav@confluent.io
> wrote:
> Hey Vahid,
>
> No I don't see an issue with it. I believe it to be the best approach.
>
> Best,
> Stanisav
>
> On Mon, Jul 23, 2018 at 12:41 PM Vahid S Hashemian <
> vahidhashemian@us.ibm.com> wrote:
>
> > Hi Stanislav,
> >
> > Thanks for the feedback.
> > Do you see an issue with using `null` as the default group id (as
> > addressed by Jason in his response)?
> > This default group id would not support offset commits and consumers
> would
> > use `auto.offset.reset` config when there is no current offset.
> >
> > Thanks.
> > --Vahid
> >
> >
> >
> >
> > From: Stanislav Kozlovski <st...@confluent.io>
> > To: dev@kafka.apache.org
> > Date: 07/20/2018 11:09 AM
> > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > behavior in KafkaConsumer
> >
> >
> >
> > I agree with Jason's notion that
> > > implicit use of the empty group.id to commit offsets is more likely
> to
> > be causing users unexpected problems than actually providing a useful
> > capability.
> > I was initially confused that this is the behavior when investigating
a
> > new-ish JIRA issue <
> >
https://issues.apache.org/jira/browse/KAFKA-6758
> > > about
> > the same topic.
> > So, +1 to deprecating "" as a group.id
> >
> > The question after that becomes what the *default* value should be -
> > should
> > we:
> > a) treat an unconfigured group.id consumer as a sort of intermittent
> > consumer where you don't store offsets at all (thereby making the user
> > explicitly sign up for them)
> > b) have a default value which makes use of them? I sort of like the
> > former.
> >
> > @Dhruvil, thinking about it at a high-level - yes. I can't think of a
> > situation where it makes sense to name something an empty string as
far
> as
> > I'm aware - to me it seems like potential for confusion
> >
> >
> > On Fri, Jul 20, 2018 at 10:22 AM Rajini Sivaram
<rajinisivaram@gmail.com
> >
> > wrote:
> >
> > > +1 to deprecate use of "" as group.id since it is odd to have a
> resource
> > > name that you cannot set ACLs for. Agree, we have to support older
> > clients
> > > though.
> > >
> > > Thanks,
> > >
> > > Rajini
> > >
> > > On Fri, Jul 20, 2018 at 5:25 PM, Jason Gustafson
<ja...@confluent.io>
> > > wrote:
> > >
> > > > Hi Vahid,
> > > >
> > > > Sorry for getting to this so late. I think there are two things
here:
> > > >
> > > > 1. The use of "" as a groupId has always been a dubious practice
at
> > best.
> > > > We definitely ought to deprecate its use in the client. Perhaps in
> the
> > > next
> > > > major release, we can remove support completely. However, since
older
> > > > clients depend on it, we may have to continue letting the broker
> > support
> > > it
> > > > to some extent. Perhaps we just need to bump the OffsetCommit
request
> > API
> > > > and only accept the offset commit for older versions. You probably
> > have
> > > to
> > > > do this anyway if you want to introduce the new error code since
old
> > > > clients will not expect it.
> > > >
> > > > 2. There should be a way for the consumer to indicate that it has
no
> > > group
> > > > id and will not commit offsets. This is an explicit instruction
that
> > the
> > > > consumer should not bother with coordinator lookup and such. We
> > currently
> > > > have some brittle logic in place to let users avoid the
coordinator
> > > lookup,
> > > > but it is a bit error-prone. I was hoping that we could change the
> > > default
> > > > value of group.id to be null so that the user had to take an
> explicit
> > > > action to opt into coordinator management (groups or offsets).
> > However,
> > > it
> > > > is true that some users may be unknowingly depending on offset
> storage
> > if
> > > > they are using both the default group.id and the default
> > > > enable.auto.commit. Perhaps one option is to disable
> > enable.auto.commit
> > > > automatically if no group.id is specified? I am not sure if there
> are
> > > any
> > > > drawbacks, but my feeling is that implicit use of the empty
group.id
> > to
> > > > commit offsets is more likely to be causing users unexpected
problems
> > > than
> > > > actually providing a useful capability.
> > > >
> > > > Thoughts?
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, May 28, 2018 at 9:50 AM, Vahid S Hashemian <
> > > > vahidhashemian@us.ibm.com> wrote:
> > > >
> > > > > Hi Viktor,
> > > > >
> > > > > Thanks for sharing your opinion.
> > > > > So you're in favor of disallowing the empty ("") group id
> altogether
> > > > (even
> > > > > for fetching).
> > > > > Given that ideally no one should be using the empty group id (at
> > least
> > > in
> > > > > a production setting) I think the impact would be minimal in
either
> > > case.
> > > > >
> > > > > But as you said, let's hear what others think and I'd be happy
to
> > > modify
> > > > > the KIP if needed.
> > > > >
> > > > > Regards.
> > > > > --Vahid
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > > To: dev <de...@kafka.apache.org>
> > > > > Date: 05/28/2018 05:18 AM
> > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group
id
> > > > > behavior in KafkaConsumer
> > > > >
> > > > >
> > > > >
> > > > > Hi Vahid,
> > > > >
> > > > > (with the argument that using the default group id for offset
> commit
> > > > > should not be the user's intention in practice).
> > > > >
> > > > > Yea, so in my opinion too this use case doesn't seem too
practical.
> > > Also
> > > > I
> > > > > think breaking the offset commit is not smaller from this
> > perspective
> > > > than
> > > > > breaking fetch and offset fetch. If we suppose that someone uses
> the
> > > > > default group id and we break the offset commit then that might
be
> > > harder
> > > > > to detect than breaking the whole thing altogether. (If we think
> > about
> > > an
> > > > > upgrade situation.)
> > > > > So since we think it is not a practical use case, I think it
would
> > be
> > > > > better to break altogether but ofc that's just my 2 cents :).
Let's
> > > > gather
> > > > > other's input as well.
> > > > >
> > > > > Cheers,
> > > > > Viktor
> > > > >
> > > > > On Fri, May 25, 2018 at 5:43 PM, Vahid S Hashemian <
> > > > > vahidhashemian@us.ibm.com> wrote:
> > > > >
> > > > > > Hi Victor,
> > > > > >
> > > > > > Thanks for reviewing the KIP.
> > > > > >
> > > > > > Yes, to minimize the backward compatibility impact, there
would
> be
> > no
> > > > > harm
> > > > > > in letting a stand-alone consumer consume messages under a ""
> > group
> > > id
> > > > > (as
> > > > > > long as there is no offset commit).
> > > > > > It would have to knowingly seek to an offset or rely on the
> > > > > > auto.offset.reset config for the starting offset.
> > > > > > This way the existing functionality would be preserved for the
> > most
> > > > part
> > > > > > (with the argument that using the default group id for offset
> > commit
> > > > > > should not be the user's intention in practice).
> > > > > >
> > > > > > Does it seem reasonable?
> > > > > >
> > > > > > Thanks.
> > > > > > --Vahid
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > > > To: dev <de...@kafka.apache.org>
> > > > > > Date: 05/25/2018 04:56 AM
> > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default
group
> > id
> > > > > > behavior in KafkaConsumer
> > > > > >
> > > > > >
> > > > > >
> > > > > > Hi Vahid,
> > > > > >
> > > > > > When reading your KIP I coldn't fully understand why did you
> > decide
> > > at
> > > > > > failing with "offset_commit" in case #2? Can't we fail with an
> > empty
> > > > > group
> > > > > > id even in "fetch" or "fetch_offset"? What was the reason for
> > > deciding
> > > > > to
> > > > > > fail at "offset_commit"? Was it because of upgrade
compatibility
> > > > > reasons?
> > > > > >
> > > > > > Thanks,
> > > > > > Viktor
> > > > > >
> > > > > > On Thu, May 24, 2018 at 12:06 AM, Ted Yu <yu...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > Looks good to me.
> > > > > > > -------- Original message --------From: Vahid S Hashemian <
> > > > > > > vahidhashemian@us.ibm.com> Date: 5/23/18 11:19 AM
> (GMT-08:00)
> > > To:
> > > > > > > dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-289: Improve
> the
> > > > > default
> > > > > > > group id behavior in KafkaConsumer
> > > > > > > Hi Ted,
> > > > > > >
> > > > > > > Thanks for reviewing the KIP. I updated the KIP and
introduced
> > an
> > > > > error
> > > > > > > code for the scenario described.
> > > > > > >
> > > > > > > --Vahid
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > From: Ted Yu <yu...@gmail.com>
> > > > > > > To: dev@kafka.apache.org
> > > > > > > Date: 04/27/2018 04:31 PM
> > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default
> group
> > id
> > > > > > > behavior in KafkaConsumer
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > bq. If they attempt an offset commit they will receive an
> error.
> > > > > > >
> > > > > > > Can you outline what specific error would be encountered ?
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > On Fri, Apr 27, 2018 at 2:17 PM, Vahid S Hashemian <
> > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I have drafted a proposal for improving the behavior of
> > > > > KafkaConsumer
> > > > > > > when
> > > > > > > > using the default group id:
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> >
https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >
> > > > >
> > > > > >
> > > > > > >
> > > > > > > > 289%3A+Improve+the+default+group+id+behavior+in+
> KafkaConsumer
> > > > > > > > The proposal based on the issue and suggestion reported in
> > > > > KAFKA-6774.
> > > > > > > >
> > > > > > > > Your feedback is welcome!
> > > > > > > >
> > > > > > > > Thanks.
> > > > > > > > --Vahid
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > Best,
> > Stanislav
> >
> >
> >
> >
> >
>
> --
> Best,
> Stanislav
>
Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer
Posted by Jason Gustafson <ja...@confluent.io>.
Hey Vahid,
Thanks for the updates. Just to clarify, I was suggesting that we disable
enable.auto.commit only if no explicit group.id is configured. If an
explicit empty string is configured for the group id, then maybe we keep
the current behavior for compatibility. We can log a warning mentioning the
deprecation and we can use the old version of the OffsetCommit API that
allows the empty group id. In a later release, we can drop this support in
the client. Does that seem reasonable?
By the way, instead of using the new ILLEGAL_OFFSET_COMMIT error code,
couldn't we use INVALID_GROUP_ID?
Thanks,
Jason
On Mon, Jul 23, 2018 at 5:14 PM, Stanislav Kozlovski <stanislav@confluent.io
> wrote:
> Hey Vahid,
>
> No I don't see an issue with it. I believe it to be the best approach.
>
> Best,
> Stanisav
>
> On Mon, Jul 23, 2018 at 12:41 PM Vahid S Hashemian <
> vahidhashemian@us.ibm.com> wrote:
>
> > Hi Stanislav,
> >
> > Thanks for the feedback.
> > Do you see an issue with using `null` as the default group id (as
> > addressed by Jason in his response)?
> > This default group id would not support offset commits and consumers
> would
> > use `auto.offset.reset` config when there is no current offset.
> >
> > Thanks.
> > --Vahid
> >
> >
> >
> >
> > From: Stanislav Kozlovski <st...@confluent.io>
> > To: dev@kafka.apache.org
> > Date: 07/20/2018 11:09 AM
> > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > behavior in KafkaConsumer
> >
> >
> >
> > I agree with Jason's notion that
> > > implicit use of the empty group.id to commit offsets is more likely
> to
> > be causing users unexpected problems than actually providing a useful
> > capability.
> > I was initially confused that this is the behavior when investigating a
> > new-ish JIRA issue <
> > https://issues.apache.org/jira/browse/KAFKA-6758
> > > about
> > the same topic.
> > So, +1 to deprecating "" as a group.id
> >
> > The question after that becomes what the *default* value should be -
> > should
> > we:
> > a) treat an unconfigured group.id consumer as a sort of intermittent
> > consumer where you don't store offsets at all (thereby making the user
> > explicitly sign up for them)
> > b) have a default value which makes use of them? I sort of like the
> > former.
> >
> > @Dhruvil, thinking about it at a high-level - yes. I can't think of a
> > situation where it makes sense to name something an empty string as far
> as
> > I'm aware - to me it seems like potential for confusion
> >
> >
> > On Fri, Jul 20, 2018 at 10:22 AM Rajini Sivaram <rajinisivaram@gmail.com
> >
> > wrote:
> >
> > > +1 to deprecate use of "" as group.id since it is odd to have a
> resource
> > > name that you cannot set ACLs for. Agree, we have to support older
> > clients
> > > though.
> > >
> > > Thanks,
> > >
> > > Rajini
> > >
> > > On Fri, Jul 20, 2018 at 5:25 PM, Jason Gustafson <ja...@confluent.io>
> > > wrote:
> > >
> > > > Hi Vahid,
> > > >
> > > > Sorry for getting to this so late. I think there are two things here:
> > > >
> > > > 1. The use of "" as a groupId has always been a dubious practice at
> > best.
> > > > We definitely ought to deprecate its use in the client. Perhaps in
> the
> > > next
> > > > major release, we can remove support completely. However, since older
> > > > clients depend on it, we may have to continue letting the broker
> > support
> > > it
> > > > to some extent. Perhaps we just need to bump the OffsetCommit request
> > API
> > > > and only accept the offset commit for older versions. You probably
> > have
> > > to
> > > > do this anyway if you want to introduce the new error code since old
> > > > clients will not expect it.
> > > >
> > > > 2. There should be a way for the consumer to indicate that it has no
> > > group
> > > > id and will not commit offsets. This is an explicit instruction that
> > the
> > > > consumer should not bother with coordinator lookup and such. We
> > currently
> > > > have some brittle logic in place to let users avoid the coordinator
> > > lookup,
> > > > but it is a bit error-prone. I was hoping that we could change the
> > > default
> > > > value of group.id to be null so that the user had to take an
> explicit
> > > > action to opt into coordinator management (groups or offsets).
> > However,
> > > it
> > > > is true that some users may be unknowingly depending on offset
> storage
> > if
> > > > they are using both the default group.id and the default
> > > > enable.auto.commit. Perhaps one option is to disable
> > enable.auto.commit
> > > > automatically if no group.id is specified? I am not sure if there
> are
> > > any
> > > > drawbacks, but my feeling is that implicit use of the empty group.id
> > to
> > > > commit offsets is more likely to be causing users unexpected problems
> > > than
> > > > actually providing a useful capability.
> > > >
> > > > Thoughts?
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, May 28, 2018 at 9:50 AM, Vahid S Hashemian <
> > > > vahidhashemian@us.ibm.com> wrote:
> > > >
> > > > > Hi Viktor,
> > > > >
> > > > > Thanks for sharing your opinion.
> > > > > So you're in favor of disallowing the empty ("") group id
> altogether
> > > > (even
> > > > > for fetching).
> > > > > Given that ideally no one should be using the empty group id (at
> > least
> > > in
> > > > > a production setting) I think the impact would be minimal in either
> > > case.
> > > > >
> > > > > But as you said, let's hear what others think and I'd be happy to
> > > modify
> > > > > the KIP if needed.
> > > > >
> > > > > Regards.
> > > > > --Vahid
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > > To: dev <de...@kafka.apache.org>
> > > > > Date: 05/28/2018 05:18 AM
> > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > > > > behavior in KafkaConsumer
> > > > >
> > > > >
> > > > >
> > > > > Hi Vahid,
> > > > >
> > > > > (with the argument that using the default group id for offset
> commit
> > > > > should not be the user's intention in practice).
> > > > >
> > > > > Yea, so in my opinion too this use case doesn't seem too practical.
> > > Also
> > > > I
> > > > > think breaking the offset commit is not smaller from this
> > perspective
> > > > than
> > > > > breaking fetch and offset fetch. If we suppose that someone uses
> the
> > > > > default group id and we break the offset commit then that might be
> > > harder
> > > > > to detect than breaking the whole thing altogether. (If we think
> > about
> > > an
> > > > > upgrade situation.)
> > > > > So since we think it is not a practical use case, I think it would
> > be
> > > > > better to break altogether but ofc that's just my 2 cents :). Let's
> > > > gather
> > > > > other's input as well.
> > > > >
> > > > > Cheers,
> > > > > Viktor
> > > > >
> > > > > On Fri, May 25, 2018 at 5:43 PM, Vahid S Hashemian <
> > > > > vahidhashemian@us.ibm.com> wrote:
> > > > >
> > > > > > Hi Victor,
> > > > > >
> > > > > > Thanks for reviewing the KIP.
> > > > > >
> > > > > > Yes, to minimize the backward compatibility impact, there would
> be
> > no
> > > > > harm
> > > > > > in letting a stand-alone consumer consume messages under a ""
> > group
> > > id
> > > > > (as
> > > > > > long as there is no offset commit).
> > > > > > It would have to knowingly seek to an offset or rely on the
> > > > > > auto.offset.reset config for the starting offset.
> > > > > > This way the existing functionality would be preserved for the
> > most
> > > > part
> > > > > > (with the argument that using the default group id for offset
> > commit
> > > > > > should not be the user's intention in practice).
> > > > > >
> > > > > > Does it seem reasonable?
> > > > > >
> > > > > > Thanks.
> > > > > > --Vahid
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > > > To: dev <de...@kafka.apache.org>
> > > > > > Date: 05/25/2018 04:56 AM
> > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group
> > id
> > > > > > behavior in KafkaConsumer
> > > > > >
> > > > > >
> > > > > >
> > > > > > Hi Vahid,
> > > > > >
> > > > > > When reading your KIP I coldn't fully understand why did you
> > decide
> > > at
> > > > > > failing with "offset_commit" in case #2? Can't we fail with an
> > empty
> > > > > group
> > > > > > id even in "fetch" or "fetch_offset"? What was the reason for
> > > deciding
> > > > > to
> > > > > > fail at "offset_commit"? Was it because of upgrade compatibility
> > > > > reasons?
> > > > > >
> > > > > > Thanks,
> > > > > > Viktor
> > > > > >
> > > > > > On Thu, May 24, 2018 at 12:06 AM, Ted Yu <yu...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > Looks good to me.
> > > > > > > -------- Original message --------From: Vahid S Hashemian <
> > > > > > > vahidhashemian@us.ibm.com> Date: 5/23/18 11:19 AM
> (GMT-08:00)
> > > To:
> > > > > > > dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-289: Improve
> the
> > > > > default
> > > > > > > group id behavior in KafkaConsumer
> > > > > > > Hi Ted,
> > > > > > >
> > > > > > > Thanks for reviewing the KIP. I updated the KIP and introduced
> > an
> > > > > error
> > > > > > > code for the scenario described.
> > > > > > >
> > > > > > > --Vahid
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > From: Ted Yu <yu...@gmail.com>
> > > > > > > To: dev@kafka.apache.org
> > > > > > > Date: 04/27/2018 04:31 PM
> > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default
> group
> > id
> > > > > > > behavior in KafkaConsumer
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > bq. If they attempt an offset commit they will receive an
> error.
> > > > > > >
> > > > > > > Can you outline what specific error would be encountered ?
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > On Fri, Apr 27, 2018 at 2:17 PM, Vahid S Hashemian <
> > > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I have drafted a proposal for improving the behavior of
> > > > > KafkaConsumer
> > > > > > > when
> > > > > > > > using the default group id:
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >
> > > > >
> > > > > >
> > > > > > >
> > > > > > > > 289%3A+Improve+the+default+group+id+behavior+in+
> KafkaConsumer
> > > > > > > > The proposal based on the issue and suggestion reported in
> > > > > KAFKA-6774.
> > > > > > > >
> > > > > > > > Your feedback is welcome!
> > > > > > > >
> > > > > > > > Thanks.
> > > > > > > > --Vahid
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > Best,
> > Stanislav
> >
> >
> >
> >
> >
>
> --
> Best,
> Stanislav
>
Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer
Posted by Stanislav Kozlovski <st...@confluent.io>.
Hey Vahid,
No I don't see an issue with it. I believe it to be the best approach.
Best,
Stanisav
On Mon, Jul 23, 2018 at 12:41 PM Vahid S Hashemian <
vahidhashemian@us.ibm.com> wrote:
> Hi Stanislav,
>
> Thanks for the feedback.
> Do you see an issue with using `null` as the default group id (as
> addressed by Jason in his response)?
> This default group id would not support offset commits and consumers would
> use `auto.offset.reset` config when there is no current offset.
>
> Thanks.
> --Vahid
>
>
>
>
> From: Stanislav Kozlovski <st...@confluent.io>
> To: dev@kafka.apache.org
> Date: 07/20/2018 11:09 AM
> Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> behavior in KafkaConsumer
>
>
>
> I agree with Jason's notion that
> > implicit use of the empty group.id to commit offsets is more likely to
> be causing users unexpected problems than actually providing a useful
> capability.
> I was initially confused that this is the behavior when investigating a
> new-ish JIRA issue <
> https://issues.apache.org/jira/browse/KAFKA-6758
> > about
> the same topic.
> So, +1 to deprecating "" as a group.id
>
> The question after that becomes what the *default* value should be -
> should
> we:
> a) treat an unconfigured group.id consumer as a sort of intermittent
> consumer where you don't store offsets at all (thereby making the user
> explicitly sign up for them)
> b) have a default value which makes use of them? I sort of like the
> former.
>
> @Dhruvil, thinking about it at a high-level - yes. I can't think of a
> situation where it makes sense to name something an empty string as far as
> I'm aware - to me it seems like potential for confusion
>
>
> On Fri, Jul 20, 2018 at 10:22 AM Rajini Sivaram <ra...@gmail.com>
> wrote:
>
> > +1 to deprecate use of "" as group.id since it is odd to have a resource
> > name that you cannot set ACLs for. Agree, we have to support older
> clients
> > though.
> >
> > Thanks,
> >
> > Rajini
> >
> > On Fri, Jul 20, 2018 at 5:25 PM, Jason Gustafson <ja...@confluent.io>
> > wrote:
> >
> > > Hi Vahid,
> > >
> > > Sorry for getting to this so late. I think there are two things here:
> > >
> > > 1. The use of "" as a groupId has always been a dubious practice at
> best.
> > > We definitely ought to deprecate its use in the client. Perhaps in the
> > next
> > > major release, we can remove support completely. However, since older
> > > clients depend on it, we may have to continue letting the broker
> support
> > it
> > > to some extent. Perhaps we just need to bump the OffsetCommit request
> API
> > > and only accept the offset commit for older versions. You probably
> have
> > to
> > > do this anyway if you want to introduce the new error code since old
> > > clients will not expect it.
> > >
> > > 2. There should be a way for the consumer to indicate that it has no
> > group
> > > id and will not commit offsets. This is an explicit instruction that
> the
> > > consumer should not bother with coordinator lookup and such. We
> currently
> > > have some brittle logic in place to let users avoid the coordinator
> > lookup,
> > > but it is a bit error-prone. I was hoping that we could change the
> > default
> > > value of group.id to be null so that the user had to take an explicit
> > > action to opt into coordinator management (groups or offsets).
> However,
> > it
> > > is true that some users may be unknowingly depending on offset storage
> if
> > > they are using both the default group.id and the default
> > > enable.auto.commit. Perhaps one option is to disable
> enable.auto.commit
> > > automatically if no group.id is specified? I am not sure if there are
> > any
> > > drawbacks, but my feeling is that implicit use of the empty group.id
> to
> > > commit offsets is more likely to be causing users unexpected problems
> > than
> > > actually providing a useful capability.
> > >
> > > Thoughts?
> > >
> > > Thanks,
> > > Jason
> > >
> > >
> > >
> > >
> > > On Mon, May 28, 2018 at 9:50 AM, Vahid S Hashemian <
> > > vahidhashemian@us.ibm.com> wrote:
> > >
> > > > Hi Viktor,
> > > >
> > > > Thanks for sharing your opinion.
> > > > So you're in favor of disallowing the empty ("") group id altogether
> > > (even
> > > > for fetching).
> > > > Given that ideally no one should be using the empty group id (at
> least
> > in
> > > > a production setting) I think the impact would be minimal in either
> > case.
> > > >
> > > > But as you said, let's hear what others think and I'd be happy to
> > modify
> > > > the KIP if needed.
> > > >
> > > > Regards.
> > > > --Vahid
> > > >
> > > >
> > > >
> > > >
> > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > To: dev <de...@kafka.apache.org>
> > > > Date: 05/28/2018 05:18 AM
> > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > > > behavior in KafkaConsumer
> > > >
> > > >
> > > >
> > > > Hi Vahid,
> > > >
> > > > (with the argument that using the default group id for offset commit
> > > > should not be the user's intention in practice).
> > > >
> > > > Yea, so in my opinion too this use case doesn't seem too practical.
> > Also
> > > I
> > > > think breaking the offset commit is not smaller from this
> perspective
> > > than
> > > > breaking fetch and offset fetch. If we suppose that someone uses the
> > > > default group id and we break the offset commit then that might be
> > harder
> > > > to detect than breaking the whole thing altogether. (If we think
> about
> > an
> > > > upgrade situation.)
> > > > So since we think it is not a practical use case, I think it would
> be
> > > > better to break altogether but ofc that's just my 2 cents :). Let's
> > > gather
> > > > other's input as well.
> > > >
> > > > Cheers,
> > > > Viktor
> > > >
> > > > On Fri, May 25, 2018 at 5:43 PM, Vahid S Hashemian <
> > > > vahidhashemian@us.ibm.com> wrote:
> > > >
> > > > > Hi Victor,
> > > > >
> > > > > Thanks for reviewing the KIP.
> > > > >
> > > > > Yes, to minimize the backward compatibility impact, there would be
> no
> > > > harm
> > > > > in letting a stand-alone consumer consume messages under a ""
> group
> > id
> > > > (as
> > > > > long as there is no offset commit).
> > > > > It would have to knowingly seek to an offset or rely on the
> > > > > auto.offset.reset config for the starting offset.
> > > > > This way the existing functionality would be preserved for the
> most
> > > part
> > > > > (with the argument that using the default group id for offset
> commit
> > > > > should not be the user's intention in practice).
> > > > >
> > > > > Does it seem reasonable?
> > > > >
> > > > > Thanks.
> > > > > --Vahid
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > > To: dev <de...@kafka.apache.org>
> > > > > Date: 05/25/2018 04:56 AM
> > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group
> id
> > > > > behavior in KafkaConsumer
> > > > >
> > > > >
> > > > >
> > > > > Hi Vahid,
> > > > >
> > > > > When reading your KIP I coldn't fully understand why did you
> decide
> > at
> > > > > failing with "offset_commit" in case #2? Can't we fail with an
> empty
> > > > group
> > > > > id even in "fetch" or "fetch_offset"? What was the reason for
> > deciding
> > > > to
> > > > > fail at "offset_commit"? Was it because of upgrade compatibility
> > > > reasons?
> > > > >
> > > > > Thanks,
> > > > > Viktor
> > > > >
> > > > > On Thu, May 24, 2018 at 12:06 AM, Ted Yu <yu...@gmail.com>
> > wrote:
> > > > >
> > > > > > Looks good to me.
> > > > > > -------- Original message --------From: Vahid S Hashemian <
> > > > > > vahidhashemian@us.ibm.com> Date: 5/23/18 11:19 AM (GMT-08:00)
> > To:
> > > > > > dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-289: Improve the
> > > > default
> > > > > > group id behavior in KafkaConsumer
> > > > > > Hi Ted,
> > > > > >
> > > > > > Thanks for reviewing the KIP. I updated the KIP and introduced
> an
> > > > error
> > > > > > code for the scenario described.
> > > > > >
> > > > > > --Vahid
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > From: Ted Yu <yu...@gmail.com>
> > > > > > To: dev@kafka.apache.org
> > > > > > Date: 04/27/2018 04:31 PM
> > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group
> id
> > > > > > behavior in KafkaConsumer
> > > > > >
> > > > > >
> > > > > >
> > > > > > bq. If they attempt an offset commit they will receive an error.
> > > > > >
> > > > > > Can you outline what specific error would be encountered ?
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Fri, Apr 27, 2018 at 2:17 PM, Vahid S Hashemian <
> > > > > > vahidhashemian@us.ibm.com> wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I have drafted a proposal for improving the behavior of
> > > > KafkaConsumer
> > > > > > when
> > > > > > > using the default group id:
> > > > > > >
> > > > > >
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>
> > > >
> > > > >
> > > > > >
> > > > > > > 289%3A+Improve+the+default+group+id+behavior+in+KafkaConsumer
> > > > > > > The proposal based on the issue and suggestion reported in
> > > > KAFKA-6774.
> > > > > > >
> > > > > > > Your feedback is welcome!
> > > > > > >
> > > > > > > Thanks.
> > > > > > > --Vahid
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
>
>
> --
> Best,
> Stanislav
>
>
>
>
>
--
Best,
Stanislav
Re: [DISCUSS] KIP-289: Improve the default group id behavior in
KafkaConsumer
Posted by Vahid S Hashemian <va...@us.ibm.com>.
Hi Stanislav,
Thanks for the feedback.
Do you see an issue with using `null` as the default group id (as
addressed by Jason in his response)?
This default group id would not support offset commits and consumers would
use `auto.offset.reset` config when there is no current offset.
Thanks.
--Vahid
From: Stanislav Kozlovski <st...@confluent.io>
To: dev@kafka.apache.org
Date: 07/20/2018 11:09 AM
Subject: Re: [DISCUSS] KIP-289: Improve the default group id
behavior in KafkaConsumer
I agree with Jason's notion that
> implicit use of the empty group.id to commit offsets is more likely to
be causing users unexpected problems than actually providing a useful
capability.
I was initially confused that this is the behavior when investigating a
new-ish JIRA issue <
https://issues.apache.org/jira/browse/KAFKA-6758
> about
the same topic.
So, +1 to deprecating "" as a group.id
The question after that becomes what the *default* value should be -
should
we:
a) treat an unconfigured group.id consumer as a sort of intermittent
consumer where you don't store offsets at all (thereby making the user
explicitly sign up for them)
b) have a default value which makes use of them? I sort of like the
former.
@Dhruvil, thinking about it at a high-level - yes. I can't think of a
situation where it makes sense to name something an empty string as far as
I'm aware - to me it seems like potential for confusion
On Fri, Jul 20, 2018 at 10:22 AM Rajini Sivaram <ra...@gmail.com>
wrote:
> +1 to deprecate use of "" as group.id since it is odd to have a resource
> name that you cannot set ACLs for. Agree, we have to support older
clients
> though.
>
> Thanks,
>
> Rajini
>
> On Fri, Jul 20, 2018 at 5:25 PM, Jason Gustafson <ja...@confluent.io>
> wrote:
>
> > Hi Vahid,
> >
> > Sorry for getting to this so late. I think there are two things here:
> >
> > 1. The use of "" as a groupId has always been a dubious practice at
best.
> > We definitely ought to deprecate its use in the client. Perhaps in the
> next
> > major release, we can remove support completely. However, since older
> > clients depend on it, we may have to continue letting the broker
support
> it
> > to some extent. Perhaps we just need to bump the OffsetCommit request
API
> > and only accept the offset commit for older versions. You probably
have
> to
> > do this anyway if you want to introduce the new error code since old
> > clients will not expect it.
> >
> > 2. There should be a way for the consumer to indicate that it has no
> group
> > id and will not commit offsets. This is an explicit instruction that
the
> > consumer should not bother with coordinator lookup and such. We
currently
> > have some brittle logic in place to let users avoid the coordinator
> lookup,
> > but it is a bit error-prone. I was hoping that we could change the
> default
> > value of group.id to be null so that the user had to take an explicit
> > action to opt into coordinator management (groups or offsets).
However,
> it
> > is true that some users may be unknowingly depending on offset storage
if
> > they are using both the default group.id and the default
> > enable.auto.commit. Perhaps one option is to disable
enable.auto.commit
> > automatically if no group.id is specified? I am not sure if there are
> any
> > drawbacks, but my feeling is that implicit use of the empty group.id
to
> > commit offsets is more likely to be causing users unexpected problems
> than
> > actually providing a useful capability.
> >
> > Thoughts?
> >
> > Thanks,
> > Jason
> >
> >
> >
> >
> > On Mon, May 28, 2018 at 9:50 AM, Vahid S Hashemian <
> > vahidhashemian@us.ibm.com> wrote:
> >
> > > Hi Viktor,
> > >
> > > Thanks for sharing your opinion.
> > > So you're in favor of disallowing the empty ("") group id altogether
> > (even
> > > for fetching).
> > > Given that ideally no one should be using the empty group id (at
least
> in
> > > a production setting) I think the impact would be minimal in either
> case.
> > >
> > > But as you said, let's hear what others think and I'd be happy to
> modify
> > > the KIP if needed.
> > >
> > > Regards.
> > > --Vahid
> > >
> > >
> > >
> > >
> > > From: Viktor Somogyi <vi...@gmail.com>
> > > To: dev <de...@kafka.apache.org>
> > > Date: 05/28/2018 05:18 AM
> > > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > > behavior in KafkaConsumer
> > >
> > >
> > >
> > > Hi Vahid,
> > >
> > > (with the argument that using the default group id for offset commit
> > > should not be the user's intention in practice).
> > >
> > > Yea, so in my opinion too this use case doesn't seem too practical.
> Also
> > I
> > > think breaking the offset commit is not smaller from this
perspective
> > than
> > > breaking fetch and offset fetch. If we suppose that someone uses the
> > > default group id and we break the offset commit then that might be
> harder
> > > to detect than breaking the whole thing altogether. (If we think
about
> an
> > > upgrade situation.)
> > > So since we think it is not a practical use case, I think it would
be
> > > better to break altogether but ofc that's just my 2 cents :). Let's
> > gather
> > > other's input as well.
> > >
> > > Cheers,
> > > Viktor
> > >
> > > On Fri, May 25, 2018 at 5:43 PM, Vahid S Hashemian <
> > > vahidhashemian@us.ibm.com> wrote:
> > >
> > > > Hi Victor,
> > > >
> > > > Thanks for reviewing the KIP.
> > > >
> > > > Yes, to minimize the backward compatibility impact, there would be
no
> > > harm
> > > > in letting a stand-alone consumer consume messages under a ""
group
> id
> > > (as
> > > > long as there is no offset commit).
> > > > It would have to knowingly seek to an offset or rely on the
> > > > auto.offset.reset config for the starting offset.
> > > > This way the existing functionality would be preserved for the
most
> > part
> > > > (with the argument that using the default group id for offset
commit
> > > > should not be the user's intention in practice).
> > > >
> > > > Does it seem reasonable?
> > > >
> > > > Thanks.
> > > > --Vahid
> > > >
> > > >
> > > >
> > > >
> > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > To: dev <de...@kafka.apache.org>
> > > > Date: 05/25/2018 04:56 AM
> > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group
id
> > > > behavior in KafkaConsumer
> > > >
> > > >
> > > >
> > > > Hi Vahid,
> > > >
> > > > When reading your KIP I coldn't fully understand why did you
decide
> at
> > > > failing with "offset_commit" in case #2? Can't we fail with an
empty
> > > group
> > > > id even in "fetch" or "fetch_offset"? What was the reason for
> deciding
> > > to
> > > > fail at "offset_commit"? Was it because of upgrade compatibility
> > > reasons?
> > > >
> > > > Thanks,
> > > > Viktor
> > > >
> > > > On Thu, May 24, 2018 at 12:06 AM, Ted Yu <yu...@gmail.com>
> wrote:
> > > >
> > > > > Looks good to me.
> > > > > -------- Original message --------From: Vahid S Hashemian <
> > > > > vahidhashemian@us.ibm.com> Date: 5/23/18 11:19 AM (GMT-08:00)
> To:
> > > > > dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-289: Improve the
> > > default
> > > > > group id behavior in KafkaConsumer
> > > > > Hi Ted,
> > > > >
> > > > > Thanks for reviewing the KIP. I updated the KIP and introduced
an
> > > error
> > > > > code for the scenario described.
> > > > >
> > > > > --Vahid
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > From: Ted Yu <yu...@gmail.com>
> > > > > To: dev@kafka.apache.org
> > > > > Date: 04/27/2018 04:31 PM
> > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group
id
> > > > > behavior in KafkaConsumer
> > > > >
> > > > >
> > > > >
> > > > > bq. If they attempt an offset commit they will receive an error.
> > > > >
> > > > > Can you outline what specific error would be encountered ?
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Fri, Apr 27, 2018 at 2:17 PM, Vahid S Hashemian <
> > > > > vahidhashemian@us.ibm.com> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I have drafted a proposal for improving the behavior of
> > > KafkaConsumer
> > > > > when
> > > > > > using the default group id:
> > > > > >
> > > > >
> > > >
> > >
https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >
> > > >
> > > > >
> > > > > > 289%3A+Improve+the+default+group+id+behavior+in+KafkaConsumer
> > > > > > The proposal based on the issue and suggestion reported in
> > > KAFKA-6774.
> > > > > >
> > > > > > Your feedback is welcome!
> > > > > >
> > > > > > Thanks.
> > > > > > --Vahid
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
>
--
Best,
Stanislav
Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer
Posted by Stanislav Kozlovski <st...@confluent.io>.
I agree with Jason's notion that
> implicit use of the empty group.id to commit offsets is more likely to
be causing users unexpected problems than actually providing a useful
capability.
I was initially confused that this is the behavior when investigating a
new-ish JIRA issue <https://issues.apache.org/jira/browse/KAFKA-6758> about
the same topic.
So, +1 to deprecating "" as a group.id
The question after that becomes what the *default* value should be - should
we:
a) treat an unconfigured group.id consumer as a sort of intermittent
consumer where you don't store offsets at all (thereby making the user
explicitly sign up for them)
b) have a default value which makes use of them? I sort of like the former.
@Dhruvil, thinking about it at a high-level - yes. I can't think of a
situation where it makes sense to name something an empty string as far as
I'm aware - to me it seems like potential for confusion
On Fri, Jul 20, 2018 at 10:22 AM Rajini Sivaram <ra...@gmail.com>
wrote:
> +1 to deprecate use of "" as group.id since it is odd to have a resource
> name that you cannot set ACLs for. Agree, we have to support older clients
> though.
>
> Thanks,
>
> Rajini
>
> On Fri, Jul 20, 2018 at 5:25 PM, Jason Gustafson <ja...@confluent.io>
> wrote:
>
> > Hi Vahid,
> >
> > Sorry for getting to this so late. I think there are two things here:
> >
> > 1. The use of "" as a groupId has always been a dubious practice at best.
> > We definitely ought to deprecate its use in the client. Perhaps in the
> next
> > major release, we can remove support completely. However, since older
> > clients depend on it, we may have to continue letting the broker support
> it
> > to some extent. Perhaps we just need to bump the OffsetCommit request API
> > and only accept the offset commit for older versions. You probably have
> to
> > do this anyway if you want to introduce the new error code since old
> > clients will not expect it.
> >
> > 2. There should be a way for the consumer to indicate that it has no
> group
> > id and will not commit offsets. This is an explicit instruction that the
> > consumer should not bother with coordinator lookup and such. We currently
> > have some brittle logic in place to let users avoid the coordinator
> lookup,
> > but it is a bit error-prone. I was hoping that we could change the
> default
> > value of group.id to be null so that the user had to take an explicit
> > action to opt into coordinator management (groups or offsets). However,
> it
> > is true that some users may be unknowingly depending on offset storage if
> > they are using both the default group.id and the default
> > enable.auto.commit. Perhaps one option is to disable enable.auto.commit
> > automatically if no group.id is specified? I am not sure if there are
> any
> > drawbacks, but my feeling is that implicit use of the empty group.id to
> > commit offsets is more likely to be causing users unexpected problems
> than
> > actually providing a useful capability.
> >
> > Thoughts?
> >
> > Thanks,
> > Jason
> >
> >
> >
> >
> > On Mon, May 28, 2018 at 9:50 AM, Vahid S Hashemian <
> > vahidhashemian@us.ibm.com> wrote:
> >
> > > Hi Viktor,
> > >
> > > Thanks for sharing your opinion.
> > > So you're in favor of disallowing the empty ("") group id altogether
> > (even
> > > for fetching).
> > > Given that ideally no one should be using the empty group id (at least
> in
> > > a production setting) I think the impact would be minimal in either
> case.
> > >
> > > But as you said, let's hear what others think and I'd be happy to
> modify
> > > the KIP if needed.
> > >
> > > Regards.
> > > --Vahid
> > >
> > >
> > >
> > >
> > > From: Viktor Somogyi <vi...@gmail.com>
> > > To: dev <de...@kafka.apache.org>
> > > Date: 05/28/2018 05:18 AM
> > > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > > behavior in KafkaConsumer
> > >
> > >
> > >
> > > Hi Vahid,
> > >
> > > (with the argument that using the default group id for offset commit
> > > should not be the user's intention in practice).
> > >
> > > Yea, so in my opinion too this use case doesn't seem too practical.
> Also
> > I
> > > think breaking the offset commit is not smaller from this perspective
> > than
> > > breaking fetch and offset fetch. If we suppose that someone uses the
> > > default group id and we break the offset commit then that might be
> harder
> > > to detect than breaking the whole thing altogether. (If we think about
> an
> > > upgrade situation.)
> > > So since we think it is not a practical use case, I think it would be
> > > better to break altogether but ofc that's just my 2 cents :). Let's
> > gather
> > > other's input as well.
> > >
> > > Cheers,
> > > Viktor
> > >
> > > On Fri, May 25, 2018 at 5:43 PM, Vahid S Hashemian <
> > > vahidhashemian@us.ibm.com> wrote:
> > >
> > > > Hi Victor,
> > > >
> > > > Thanks for reviewing the KIP.
> > > >
> > > > Yes, to minimize the backward compatibility impact, there would be no
> > > harm
> > > > in letting a stand-alone consumer consume messages under a "" group
> id
> > > (as
> > > > long as there is no offset commit).
> > > > It would have to knowingly seek to an offset or rely on the
> > > > auto.offset.reset config for the starting offset.
> > > > This way the existing functionality would be preserved for the most
> > part
> > > > (with the argument that using the default group id for offset commit
> > > > should not be the user's intention in practice).
> > > >
> > > > Does it seem reasonable?
> > > >
> > > > Thanks.
> > > > --Vahid
> > > >
> > > >
> > > >
> > > >
> > > > From: Viktor Somogyi <vi...@gmail.com>
> > > > To: dev <de...@kafka.apache.org>
> > > > Date: 05/25/2018 04:56 AM
> > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > > > behavior in KafkaConsumer
> > > >
> > > >
> > > >
> > > > Hi Vahid,
> > > >
> > > > When reading your KIP I coldn't fully understand why did you decide
> at
> > > > failing with "offset_commit" in case #2? Can't we fail with an empty
> > > group
> > > > id even in "fetch" or "fetch_offset"? What was the reason for
> deciding
> > > to
> > > > fail at "offset_commit"? Was it because of upgrade compatibility
> > > reasons?
> > > >
> > > > Thanks,
> > > > Viktor
> > > >
> > > > On Thu, May 24, 2018 at 12:06 AM, Ted Yu <yu...@gmail.com>
> wrote:
> > > >
> > > > > Looks good to me.
> > > > > -------- Original message --------From: Vahid S Hashemian <
> > > > > vahidhashemian@us.ibm.com> Date: 5/23/18 11:19 AM (GMT-08:00)
> To:
> > > > > dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-289: Improve the
> > > default
> > > > > group id behavior in KafkaConsumer
> > > > > Hi Ted,
> > > > >
> > > > > Thanks for reviewing the KIP. I updated the KIP and introduced an
> > > error
> > > > > code for the scenario described.
> > > > >
> > > > > --Vahid
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > From: Ted Yu <yu...@gmail.com>
> > > > > To: dev@kafka.apache.org
> > > > > Date: 04/27/2018 04:31 PM
> > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > > > > behavior in KafkaConsumer
> > > > >
> > > > >
> > > > >
> > > > > bq. If they attempt an offset commit they will receive an error.
> > > > >
> > > > > Can you outline what specific error would be encountered ?
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Fri, Apr 27, 2018 at 2:17 PM, Vahid S Hashemian <
> > > > > vahidhashemian@us.ibm.com> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I have drafted a proposal for improving the behavior of
> > > KafkaConsumer
> > > > > when
> > > > > > using the default group id:
> > > > > >
> > > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >
> > > >
> > > > >
> > > > > > 289%3A+Improve+the+default+group+id+behavior+in+KafkaConsumer
> > > > > > The proposal based on the issue and suggestion reported in
> > > KAFKA-6774.
> > > > > >
> > > > > > Your feedback is welcome!
> > > > > >
> > > > > > Thanks.
> > > > > > --Vahid
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
>
--
Best,
Stanislav
Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer
Posted by Rajini Sivaram <ra...@gmail.com>.
+1 to deprecate use of "" as group.id since it is odd to have a resource
name that you cannot set ACLs for. Agree, we have to support older clients
though.
Thanks,
Rajini
On Fri, Jul 20, 2018 at 5:25 PM, Jason Gustafson <ja...@confluent.io> wrote:
> Hi Vahid,
>
> Sorry for getting to this so late. I think there are two things here:
>
> 1. The use of "" as a groupId has always been a dubious practice at best.
> We definitely ought to deprecate its use in the client. Perhaps in the next
> major release, we can remove support completely. However, since older
> clients depend on it, we may have to continue letting the broker support it
> to some extent. Perhaps we just need to bump the OffsetCommit request API
> and only accept the offset commit for older versions. You probably have to
> do this anyway if you want to introduce the new error code since old
> clients will not expect it.
>
> 2. There should be a way for the consumer to indicate that it has no group
> id and will not commit offsets. This is an explicit instruction that the
> consumer should not bother with coordinator lookup and such. We currently
> have some brittle logic in place to let users avoid the coordinator lookup,
> but it is a bit error-prone. I was hoping that we could change the default
> value of group.id to be null so that the user had to take an explicit
> action to opt into coordinator management (groups or offsets). However, it
> is true that some users may be unknowingly depending on offset storage if
> they are using both the default group.id and the default
> enable.auto.commit. Perhaps one option is to disable enable.auto.commit
> automatically if no group.id is specified? I am not sure if there are any
> drawbacks, but my feeling is that implicit use of the empty group.id to
> commit offsets is more likely to be causing users unexpected problems than
> actually providing a useful capability.
>
> Thoughts?
>
> Thanks,
> Jason
>
>
>
>
> On Mon, May 28, 2018 at 9:50 AM, Vahid S Hashemian <
> vahidhashemian@us.ibm.com> wrote:
>
> > Hi Viktor,
> >
> > Thanks for sharing your opinion.
> > So you're in favor of disallowing the empty ("") group id altogether
> (even
> > for fetching).
> > Given that ideally no one should be using the empty group id (at least in
> > a production setting) I think the impact would be minimal in either case.
> >
> > But as you said, let's hear what others think and I'd be happy to modify
> > the KIP if needed.
> >
> > Regards.
> > --Vahid
> >
> >
> >
> >
> > From: Viktor Somogyi <vi...@gmail.com>
> > To: dev <de...@kafka.apache.org>
> > Date: 05/28/2018 05:18 AM
> > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > behavior in KafkaConsumer
> >
> >
> >
> > Hi Vahid,
> >
> > (with the argument that using the default group id for offset commit
> > should not be the user's intention in practice).
> >
> > Yea, so in my opinion too this use case doesn't seem too practical. Also
> I
> > think breaking the offset commit is not smaller from this perspective
> than
> > breaking fetch and offset fetch. If we suppose that someone uses the
> > default group id and we break the offset commit then that might be harder
> > to detect than breaking the whole thing altogether. (If we think about an
> > upgrade situation.)
> > So since we think it is not a practical use case, I think it would be
> > better to break altogether but ofc that's just my 2 cents :). Let's
> gather
> > other's input as well.
> >
> > Cheers,
> > Viktor
> >
> > On Fri, May 25, 2018 at 5:43 PM, Vahid S Hashemian <
> > vahidhashemian@us.ibm.com> wrote:
> >
> > > Hi Victor,
> > >
> > > Thanks for reviewing the KIP.
> > >
> > > Yes, to minimize the backward compatibility impact, there would be no
> > harm
> > > in letting a stand-alone consumer consume messages under a "" group id
> > (as
> > > long as there is no offset commit).
> > > It would have to knowingly seek to an offset or rely on the
> > > auto.offset.reset config for the starting offset.
> > > This way the existing functionality would be preserved for the most
> part
> > > (with the argument that using the default group id for offset commit
> > > should not be the user's intention in practice).
> > >
> > > Does it seem reasonable?
> > >
> > > Thanks.
> > > --Vahid
> > >
> > >
> > >
> > >
> > > From: Viktor Somogyi <vi...@gmail.com>
> > > To: dev <de...@kafka.apache.org>
> > > Date: 05/25/2018 04:56 AM
> > > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > > behavior in KafkaConsumer
> > >
> > >
> > >
> > > Hi Vahid,
> > >
> > > When reading your KIP I coldn't fully understand why did you decide at
> > > failing with "offset_commit" in case #2? Can't we fail with an empty
> > group
> > > id even in "fetch" or "fetch_offset"? What was the reason for deciding
> > to
> > > fail at "offset_commit"? Was it because of upgrade compatibility
> > reasons?
> > >
> > > Thanks,
> > > Viktor
> > >
> > > On Thu, May 24, 2018 at 12:06 AM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > Looks good to me.
> > > > -------- Original message --------From: Vahid S Hashemian <
> > > > vahidhashemian@us.ibm.com> Date: 5/23/18 11:19 AM (GMT-08:00) To:
> > > > dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-289: Improve the
> > default
> > > > group id behavior in KafkaConsumer
> > > > Hi Ted,
> > > >
> > > > Thanks for reviewing the KIP. I updated the KIP and introduced an
> > error
> > > > code for the scenario described.
> > > >
> > > > --Vahid
> > > >
> > > >
> > > >
> > > >
> > > > From: Ted Yu <yu...@gmail.com>
> > > > To: dev@kafka.apache.org
> > > > Date: 04/27/2018 04:31 PM
> > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > > > behavior in KafkaConsumer
> > > >
> > > >
> > > >
> > > > bq. If they attempt an offset commit they will receive an error.
> > > >
> > > > Can you outline what specific error would be encountered ?
> > > >
> > > > Thanks
> > > >
> > > > On Fri, Apr 27, 2018 at 2:17 PM, Vahid S Hashemian <
> > > > vahidhashemian@us.ibm.com> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I have drafted a proposal for improving the behavior of
> > KafkaConsumer
> > > > when
> > > > > using the default group id:
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >
> > >
> > > >
> > > > > 289%3A+Improve+the+default+group+id+behavior+in+KafkaConsumer
> > > > > The proposal based on the issue and suggestion reported in
> > KAFKA-6774.
> > > > >
> > > > > Your feedback is welcome!
> > > > >
> > > > > Thanks.
> > > > > --Vahid
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
Re: [DISCUSS] KIP-289: Improve the default group id behavior in
KafkaConsumer
Posted by Vahid S Hashemian <va...@us.ibm.com>.
Hi Jason,
Thanks for the feedback.
1. I think the current KIP follows your first bullet suggestion; except
for the addition of a OffsetCommit version and continuing to support old
versions (I will add these).
2. Regarding your suggestion, using `null` as default group id seems
reasonable. Could you please elaborate on disabling `enable.auto.commit`?
They way I see it, the new version of the OffsetCommit will reject
committing offsets for empty ("") and null group ids. Are you suggesting
to avoid committing offsets (by default) on the client side for these
group ids?
Thanks!
--Vahid
From: Jason Gustafson <ja...@confluent.io>
To: dev <de...@kafka.apache.org>
Date: 07/20/2018 09:25 AM
Subject: Re: [DISCUSS] KIP-289: Improve the default group id
behavior in KafkaConsumer
Hi Vahid,
Sorry for getting to this so late. I think there are two things here:
1. The use of "" as a groupId has always been a dubious practice at best.
We definitely ought to deprecate its use in the client. Perhaps in the
next
major release, we can remove support completely. However, since older
clients depend on it, we may have to continue letting the broker support
it
to some extent. Perhaps we just need to bump the OffsetCommit request API
and only accept the offset commit for older versions. You probably have to
do this anyway if you want to introduce the new error code since old
clients will not expect it.
2. There should be a way for the consumer to indicate that it has no group
id and will not commit offsets. This is an explicit instruction that the
consumer should not bother with coordinator lookup and such. We currently
have some brittle logic in place to let users avoid the coordinator
lookup,
but it is a bit error-prone. I was hoping that we could change the default
value of group.id to be null so that the user had to take an explicit
action to opt into coordinator management (groups or offsets). However, it
is true that some users may be unknowingly depending on offset storage if
they are using both the default group.id and the default
enable.auto.commit. Perhaps one option is to disable enable.auto.commit
automatically if no group.id is specified? I am not sure if there are any
drawbacks, but my feeling is that implicit use of the empty group.id to
commit offsets is more likely to be causing users unexpected problems than
actually providing a useful capability.
Thoughts?
Thanks,
Jason
On Mon, May 28, 2018 at 9:50 AM, Vahid S Hashemian <
vahidhashemian@us.ibm.com> wrote:
> Hi Viktor,
>
> Thanks for sharing your opinion.
> So you're in favor of disallowing the empty ("") group id altogether
(even
> for fetching).
> Given that ideally no one should be using the empty group id (at least
in
> a production setting) I think the impact would be minimal in either
case.
>
> But as you said, let's hear what others think and I'd be happy to modify
> the KIP if needed.
>
> Regards.
> --Vahid
>
>
>
>
> From: Viktor Somogyi <vi...@gmail.com>
> To: dev <de...@kafka.apache.org>
> Date: 05/28/2018 05:18 AM
> Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> behavior in KafkaConsumer
>
>
>
> Hi Vahid,
>
> (with the argument that using the default group id for offset commit
> should not be the user's intention in practice).
>
> Yea, so in my opinion too this use case doesn't seem too practical. Also
I
> think breaking the offset commit is not smaller from this perspective
than
> breaking fetch and offset fetch. If we suppose that someone uses the
> default group id and we break the offset commit then that might be
harder
> to detect than breaking the whole thing altogether. (If we think about
an
> upgrade situation.)
> So since we think it is not a practical use case, I think it would be
> better to break altogether but ofc that's just my 2 cents :). Let's
gather
> other's input as well.
>
> Cheers,
> Viktor
>
> On Fri, May 25, 2018 at 5:43 PM, Vahid S Hashemian <
> vahidhashemian@us.ibm.com> wrote:
>
> > Hi Victor,
> >
> > Thanks for reviewing the KIP.
> >
> > Yes, to minimize the backward compatibility impact, there would be no
> harm
> > in letting a stand-alone consumer consume messages under a "" group id
> (as
> > long as there is no offset commit).
> > It would have to knowingly seek to an offset or rely on the
> > auto.offset.reset config for the starting offset.
> > This way the existing functionality would be preserved for the most
part
> > (with the argument that using the default group id for offset commit
> > should not be the user's intention in practice).
> >
> > Does it seem reasonable?
> >
> > Thanks.
> > --Vahid
> >
> >
> >
> >
> > From: Viktor Somogyi <vi...@gmail.com>
> > To: dev <de...@kafka.apache.org>
> > Date: 05/25/2018 04:56 AM
> > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > behavior in KafkaConsumer
> >
> >
> >
> > Hi Vahid,
> >
> > When reading your KIP I coldn't fully understand why did you decide at
> > failing with "offset_commit" in case #2? Can't we fail with an empty
> group
> > id even in "fetch" or "fetch_offset"? What was the reason for deciding
> to
> > fail at "offset_commit"? Was it because of upgrade compatibility
> reasons?
> >
> > Thanks,
> > Viktor
> >
> > On Thu, May 24, 2018 at 12:06 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > Looks good to me.
> > > -------- Original message --------From: Vahid S Hashemian <
> > > vahidhashemian@us.ibm.com> Date: 5/23/18 11:19 AM (GMT-08:00) To:
> > > dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-289: Improve the
> default
> > > group id behavior in KafkaConsumer
> > > Hi Ted,
> > >
> > > Thanks for reviewing the KIP. I updated the KIP and introduced an
> error
> > > code for the scenario described.
> > >
> > > --Vahid
> > >
> > >
> > >
> > >
> > > From: Ted Yu <yu...@gmail.com>
> > > To: dev@kafka.apache.org
> > > Date: 04/27/2018 04:31 PM
> > > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > > behavior in KafkaConsumer
> > >
> > >
> > >
> > > bq. If they attempt an offset commit they will receive an error.
> > >
> > > Can you outline what specific error would be encountered ?
> > >
> > > Thanks
> > >
> > > On Fri, Apr 27, 2018 at 2:17 PM, Vahid S Hashemian <
> > > vahidhashemian@us.ibm.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I have drafted a proposal for improving the behavior of
> KafkaConsumer
> > > when
> > > > using the default group id:
> > > >
> > >
> >
>
https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>
> >
> > >
> > > > 289%3A+Improve+the+default+group+id+behavior+in+KafkaConsumer
> > > > The proposal based on the issue and suggestion reported in
> KAFKA-6774.
> > > >
> > > > Your feedback is welcome!
> > > >
> > > > Thanks.
> > > > --Vahid
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer
Posted by Jason Gustafson <ja...@confluent.io>.
Hi Vahid,
Sorry for getting to this so late. I think there are two things here:
1. The use of "" as a groupId has always been a dubious practice at best.
We definitely ought to deprecate its use in the client. Perhaps in the next
major release, we can remove support completely. However, since older
clients depend on it, we may have to continue letting the broker support it
to some extent. Perhaps we just need to bump the OffsetCommit request API
and only accept the offset commit for older versions. You probably have to
do this anyway if you want to introduce the new error code since old
clients will not expect it.
2. There should be a way for the consumer to indicate that it has no group
id and will not commit offsets. This is an explicit instruction that the
consumer should not bother with coordinator lookup and such. We currently
have some brittle logic in place to let users avoid the coordinator lookup,
but it is a bit error-prone. I was hoping that we could change the default
value of group.id to be null so that the user had to take an explicit
action to opt into coordinator management (groups or offsets). However, it
is true that some users may be unknowingly depending on offset storage if
they are using both the default group.id and the default
enable.auto.commit. Perhaps one option is to disable enable.auto.commit
automatically if no group.id is specified? I am not sure if there are any
drawbacks, but my feeling is that implicit use of the empty group.id to
commit offsets is more likely to be causing users unexpected problems than
actually providing a useful capability.
Thoughts?
Thanks,
Jason
On Mon, May 28, 2018 at 9:50 AM, Vahid S Hashemian <
vahidhashemian@us.ibm.com> wrote:
> Hi Viktor,
>
> Thanks for sharing your opinion.
> So you're in favor of disallowing the empty ("") group id altogether (even
> for fetching).
> Given that ideally no one should be using the empty group id (at least in
> a production setting) I think the impact would be minimal in either case.
>
> But as you said, let's hear what others think and I'd be happy to modify
> the KIP if needed.
>
> Regards.
> --Vahid
>
>
>
>
> From: Viktor Somogyi <vi...@gmail.com>
> To: dev <de...@kafka.apache.org>
> Date: 05/28/2018 05:18 AM
> Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> behavior in KafkaConsumer
>
>
>
> Hi Vahid,
>
> (with the argument that using the default group id for offset commit
> should not be the user's intention in practice).
>
> Yea, so in my opinion too this use case doesn't seem too practical. Also I
> think breaking the offset commit is not smaller from this perspective than
> breaking fetch and offset fetch. If we suppose that someone uses the
> default group id and we break the offset commit then that might be harder
> to detect than breaking the whole thing altogether. (If we think about an
> upgrade situation.)
> So since we think it is not a practical use case, I think it would be
> better to break altogether but ofc that's just my 2 cents :). Let's gather
> other's input as well.
>
> Cheers,
> Viktor
>
> On Fri, May 25, 2018 at 5:43 PM, Vahid S Hashemian <
> vahidhashemian@us.ibm.com> wrote:
>
> > Hi Victor,
> >
> > Thanks for reviewing the KIP.
> >
> > Yes, to minimize the backward compatibility impact, there would be no
> harm
> > in letting a stand-alone consumer consume messages under a "" group id
> (as
> > long as there is no offset commit).
> > It would have to knowingly seek to an offset or rely on the
> > auto.offset.reset config for the starting offset.
> > This way the existing functionality would be preserved for the most part
> > (with the argument that using the default group id for offset commit
> > should not be the user's intention in practice).
> >
> > Does it seem reasonable?
> >
> > Thanks.
> > --Vahid
> >
> >
> >
> >
> > From: Viktor Somogyi <vi...@gmail.com>
> > To: dev <de...@kafka.apache.org>
> > Date: 05/25/2018 04:56 AM
> > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > behavior in KafkaConsumer
> >
> >
> >
> > Hi Vahid,
> >
> > When reading your KIP I coldn't fully understand why did you decide at
> > failing with "offset_commit" in case #2? Can't we fail with an empty
> group
> > id even in "fetch" or "fetch_offset"? What was the reason for deciding
> to
> > fail at "offset_commit"? Was it because of upgrade compatibility
> reasons?
> >
> > Thanks,
> > Viktor
> >
> > On Thu, May 24, 2018 at 12:06 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > Looks good to me.
> > > -------- Original message --------From: Vahid S Hashemian <
> > > vahidhashemian@us.ibm.com> Date: 5/23/18 11:19 AM (GMT-08:00) To:
> > > dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-289: Improve the
> default
> > > group id behavior in KafkaConsumer
> > > Hi Ted,
> > >
> > > Thanks for reviewing the KIP. I updated the KIP and introduced an
> error
> > > code for the scenario described.
> > >
> > > --Vahid
> > >
> > >
> > >
> > >
> > > From: Ted Yu <yu...@gmail.com>
> > > To: dev@kafka.apache.org
> > > Date: 04/27/2018 04:31 PM
> > > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > > behavior in KafkaConsumer
> > >
> > >
> > >
> > > bq. If they attempt an offset commit they will receive an error.
> > >
> > > Can you outline what specific error would be encountered ?
> > >
> > > Thanks
> > >
> > > On Fri, Apr 27, 2018 at 2:17 PM, Vahid S Hashemian <
> > > vahidhashemian@us.ibm.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I have drafted a proposal for improving the behavior of
> KafkaConsumer
> > > when
> > > > using the default group id:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>
> >
> > >
> > > > 289%3A+Improve+the+default+group+id+behavior+in+KafkaConsumer
> > > > The proposal based on the issue and suggestion reported in
> KAFKA-6774.
> > > >
> > > > Your feedback is welcome!
> > > >
> > > > Thanks.
> > > > --Vahid
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
Re: [DISCUSS] KIP-289: Improve the default group id behavior in
KafkaConsumer
Posted by Vahid S Hashemian <va...@us.ibm.com>.
Hi Viktor,
Thanks for sharing your opinion.
So you're in favor of disallowing the empty ("") group id altogether (even
for fetching).
Given that ideally no one should be using the empty group id (at least in
a production setting) I think the impact would be minimal in either case.
But as you said, let's hear what others think and I'd be happy to modify
the KIP if needed.
Regards.
--Vahid
From: Viktor Somogyi <vi...@gmail.com>
To: dev <de...@kafka.apache.org>
Date: 05/28/2018 05:18 AM
Subject: Re: [DISCUSS] KIP-289: Improve the default group id
behavior in KafkaConsumer
Hi Vahid,
(with the argument that using the default group id for offset commit
should not be the user's intention in practice).
Yea, so in my opinion too this use case doesn't seem too practical. Also I
think breaking the offset commit is not smaller from this perspective than
breaking fetch and offset fetch. If we suppose that someone uses the
default group id and we break the offset commit then that might be harder
to detect than breaking the whole thing altogether. (If we think about an
upgrade situation.)
So since we think it is not a practical use case, I think it would be
better to break altogether but ofc that's just my 2 cents :). Let's gather
other's input as well.
Cheers,
Viktor
On Fri, May 25, 2018 at 5:43 PM, Vahid S Hashemian <
vahidhashemian@us.ibm.com> wrote:
> Hi Victor,
>
> Thanks for reviewing the KIP.
>
> Yes, to minimize the backward compatibility impact, there would be no
harm
> in letting a stand-alone consumer consume messages under a "" group id
(as
> long as there is no offset commit).
> It would have to knowingly seek to an offset or rely on the
> auto.offset.reset config for the starting offset.
> This way the existing functionality would be preserved for the most part
> (with the argument that using the default group id for offset commit
> should not be the user's intention in practice).
>
> Does it seem reasonable?
>
> Thanks.
> --Vahid
>
>
>
>
> From: Viktor Somogyi <vi...@gmail.com>
> To: dev <de...@kafka.apache.org>
> Date: 05/25/2018 04:56 AM
> Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> behavior in KafkaConsumer
>
>
>
> Hi Vahid,
>
> When reading your KIP I coldn't fully understand why did you decide at
> failing with "offset_commit" in case #2? Can't we fail with an empty
group
> id even in "fetch" or "fetch_offset"? What was the reason for deciding
to
> fail at "offset_commit"? Was it because of upgrade compatibility
reasons?
>
> Thanks,
> Viktor
>
> On Thu, May 24, 2018 at 12:06 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > Looks good to me.
> > -------- Original message --------From: Vahid S Hashemian <
> > vahidhashemian@us.ibm.com> Date: 5/23/18 11:19 AM (GMT-08:00) To:
> > dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-289: Improve the
default
> > group id behavior in KafkaConsumer
> > Hi Ted,
> >
> > Thanks for reviewing the KIP. I updated the KIP and introduced an
error
> > code for the scenario described.
> >
> > --Vahid
> >
> >
> >
> >
> > From: Ted Yu <yu...@gmail.com>
> > To: dev@kafka.apache.org
> > Date: 04/27/2018 04:31 PM
> > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > behavior in KafkaConsumer
> >
> >
> >
> > bq. If they attempt an offset commit they will receive an error.
> >
> > Can you outline what specific error would be encountered ?
> >
> > Thanks
> >
> > On Fri, Apr 27, 2018 at 2:17 PM, Vahid S Hashemian <
> > vahidhashemian@us.ibm.com> wrote:
> >
> > > Hi all,
> > >
> > > I have drafted a proposal for improving the behavior of
KafkaConsumer
> > when
> > > using the default group id:
> > >
> >
>
https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>
> >
> > > 289%3A+Improve+the+default+group+id+behavior+in+KafkaConsumer
> > > The proposal based on the issue and suggestion reported in
KAFKA-6774.
> > >
> > > Your feedback is welcome!
> > >
> > > Thanks.
> > > --Vahid
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer
Posted by Viktor Somogyi <vi...@gmail.com>.
Hi Vahid,
(with the argument that using the default group id for offset commit
should not be the user's intention in practice).
Yea, so in my opinion too this use case doesn't seem too practical. Also I
think breaking the offset commit is not smaller from this perspective than
breaking fetch and offset fetch. If we suppose that someone uses the
default group id and we break the offset commit then that might be harder
to detect than breaking the whole thing altogether. (If we think about an
upgrade situation.)
So since we think it is not a practical use case, I think it would be
better to break altogether but ofc that's just my 2 cents :). Let's gather
other's input as well.
Cheers,
Viktor
On Fri, May 25, 2018 at 5:43 PM, Vahid S Hashemian <
vahidhashemian@us.ibm.com> wrote:
> Hi Victor,
>
> Thanks for reviewing the KIP.
>
> Yes, to minimize the backward compatibility impact, there would be no harm
> in letting a stand-alone consumer consume messages under a "" group id (as
> long as there is no offset commit).
> It would have to knowingly seek to an offset or rely on the
> auto.offset.reset config for the starting offset.
> This way the existing functionality would be preserved for the most part
> (with the argument that using the default group id for offset commit
> should not be the user's intention in practice).
>
> Does it seem reasonable?
>
> Thanks.
> --Vahid
>
>
>
>
> From: Viktor Somogyi <vi...@gmail.com>
> To: dev <de...@kafka.apache.org>
> Date: 05/25/2018 04:56 AM
> Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> behavior in KafkaConsumer
>
>
>
> Hi Vahid,
>
> When reading your KIP I coldn't fully understand why did you decide at
> failing with "offset_commit" in case #2? Can't we fail with an empty group
> id even in "fetch" or "fetch_offset"? What was the reason for deciding to
> fail at "offset_commit"? Was it because of upgrade compatibility reasons?
>
> Thanks,
> Viktor
>
> On Thu, May 24, 2018 at 12:06 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > Looks good to me.
> > -------- Original message --------From: Vahid S Hashemian <
> > vahidhashemian@us.ibm.com> Date: 5/23/18 11:19 AM (GMT-08:00) To:
> > dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-289: Improve the default
> > group id behavior in KafkaConsumer
> > Hi Ted,
> >
> > Thanks for reviewing the KIP. I updated the KIP and introduced an error
> > code for the scenario described.
> >
> > --Vahid
> >
> >
> >
> >
> > From: Ted Yu <yu...@gmail.com>
> > To: dev@kafka.apache.org
> > Date: 04/27/2018 04:31 PM
> > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > behavior in KafkaConsumer
> >
> >
> >
> > bq. If they attempt an offset commit they will receive an error.
> >
> > Can you outline what specific error would be encountered ?
> >
> > Thanks
> >
> > On Fri, Apr 27, 2018 at 2:17 PM, Vahid S Hashemian <
> > vahidhashemian@us.ibm.com> wrote:
> >
> > > Hi all,
> > >
> > > I have drafted a proposal for improving the behavior of KafkaConsumer
> > when
> > > using the default group id:
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>
> >
> > > 289%3A+Improve+the+default+group+id+behavior+in+KafkaConsumer
> > > The proposal based on the issue and suggestion reported in KAFKA-6774.
> > >
> > > Your feedback is welcome!
> > >
> > > Thanks.
> > > --Vahid
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
Re: [DISCUSS] KIP-289: Improve the default group id behavior in
KafkaConsumer
Posted by Vahid S Hashemian <va...@us.ibm.com>.
Hi Victor,
Thanks for reviewing the KIP.
Yes, to minimize the backward compatibility impact, there would be no harm
in letting a stand-alone consumer consume messages under a "" group id (as
long as there is no offset commit).
It would have to knowingly seek to an offset or rely on the
auto.offset.reset config for the starting offset.
This way the existing functionality would be preserved for the most part
(with the argument that using the default group id for offset commit
should not be the user's intention in practice).
Does it seem reasonable?
Thanks.
--Vahid
From: Viktor Somogyi <vi...@gmail.com>
To: dev <de...@kafka.apache.org>
Date: 05/25/2018 04:56 AM
Subject: Re: [DISCUSS] KIP-289: Improve the default group id
behavior in KafkaConsumer
Hi Vahid,
When reading your KIP I coldn't fully understand why did you decide at
failing with "offset_commit" in case #2? Can't we fail with an empty group
id even in "fetch" or "fetch_offset"? What was the reason for deciding to
fail at "offset_commit"? Was it because of upgrade compatibility reasons?
Thanks,
Viktor
On Thu, May 24, 2018 at 12:06 AM, Ted Yu <yu...@gmail.com> wrote:
> Looks good to me.
> -------- Original message --------From: Vahid S Hashemian <
> vahidhashemian@us.ibm.com> Date: 5/23/18 11:19 AM (GMT-08:00) To:
> dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-289: Improve the default
> group id behavior in KafkaConsumer
> Hi Ted,
>
> Thanks for reviewing the KIP. I updated the KIP and introduced an error
> code for the scenario described.
>
> --Vahid
>
>
>
>
> From: Ted Yu <yu...@gmail.com>
> To: dev@kafka.apache.org
> Date: 04/27/2018 04:31 PM
> Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> behavior in KafkaConsumer
>
>
>
> bq. If they attempt an offset commit they will receive an error.
>
> Can you outline what specific error would be encountered ?
>
> Thanks
>
> On Fri, Apr 27, 2018 at 2:17 PM, Vahid S Hashemian <
> vahidhashemian@us.ibm.com> wrote:
>
> > Hi all,
> >
> > I have drafted a proposal for improving the behavior of KafkaConsumer
> when
> > using the default group id:
> >
>
https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>
> > 289%3A+Improve+the+default+group+id+behavior+in+KafkaConsumer
> > The proposal based on the issue and suggestion reported in KAFKA-6774.
> >
> > Your feedback is welcome!
> >
> > Thanks.
> > --Vahid
> >
> >
>
>
>
>
>
Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer
Posted by Viktor Somogyi <vi...@gmail.com>.
Hi Vahid,
When reading your KIP I coldn't fully understand why did you decide at
failing with "offset_commit" in case #2? Can't we fail with an empty group
id even in "fetch" or "fetch_offset"? What was the reason for deciding to
fail at "offset_commit"? Was it because of upgrade compatibility reasons?
Thanks,
Viktor
On Thu, May 24, 2018 at 12:06 AM, Ted Yu <yu...@gmail.com> wrote:
> Looks good to me.
> -------- Original message --------From: Vahid S Hashemian <
> vahidhashemian@us.ibm.com> Date: 5/23/18 11:19 AM (GMT-08:00) To:
> dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-289: Improve the default
> group id behavior in KafkaConsumer
> Hi Ted,
>
> Thanks for reviewing the KIP. I updated the KIP and introduced an error
> code for the scenario described.
>
> --Vahid
>
>
>
>
> From: Ted Yu <yu...@gmail.com>
> To: dev@kafka.apache.org
> Date: 04/27/2018 04:31 PM
> Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> behavior in KafkaConsumer
>
>
>
> bq. If they attempt an offset commit they will receive an error.
>
> Can you outline what specific error would be encountered ?
>
> Thanks
>
> On Fri, Apr 27, 2018 at 2:17 PM, Vahid S Hashemian <
> vahidhashemian@us.ibm.com> wrote:
>
> > Hi all,
> >
> > I have drafted a proposal for improving the behavior of KafkaConsumer
> when
> > using the default group id:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>
> > 289%3A+Improve+the+default+group+id+behavior+in+KafkaConsumer
> > The proposal based on the issue and suggestion reported in KAFKA-6774.
> >
> > Your feedback is welcome!
> >
> > Thanks.
> > --Vahid
> >
> >
>
>
>
>
>
Re: [DISCUSS] KIP-289: Improve the default group id behavior in
KafkaConsumer
Posted by Ted Yu <yu...@gmail.com>.
Looks good to me.
-------- Original message --------From: Vahid S Hashemian <va...@us.ibm.com> Date: 5/23/18 11:19 AM (GMT-08:00) To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer
Hi Ted,
Thanks for reviewing the KIP. I updated the KIP and introduced an error
code for the scenario described.
--Vahid
From: Ted Yu <yu...@gmail.com>
To: dev@kafka.apache.org
Date: 04/27/2018 04:31 PM
Subject: Re: [DISCUSS] KIP-289: Improve the default group id
behavior in KafkaConsumer
bq. If they attempt an offset commit they will receive an error.
Can you outline what specific error would be encountered ?
Thanks
On Fri, Apr 27, 2018 at 2:17 PM, Vahid S Hashemian <
vahidhashemian@us.ibm.com> wrote:
> Hi all,
>
> I have drafted a proposal for improving the behavior of KafkaConsumer
when
> using the default group id:
>
https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 289%3A+Improve+the+default+group+id+behavior+in+KafkaConsumer
> The proposal based on the issue and suggestion reported in KAFKA-6774.
>
> Your feedback is welcome!
>
> Thanks.
> --Vahid
>
>