You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by feyman2009 <fe...@aliyun.com.INVALID> on 2020/04/08 01:59:08 UTC
回复：回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in StreamsResetter
Hi, Boyang&Matthias
I think Matthias's proposal makes sense, but we can use the admin tool for this scenario as Boyang mentioned or follow up later, so I prefer to keep this KIP unchanged to minimize the scope.
Calling for vote ~
Thanks!
Feyman
------------------------------------------------------------------
发件人：Boyang Chen <re...@gmail.com>
发送时间：2020年4月8日(星期三) 02:15
收件人：dev <de...@kafka.apache.org>
主 题：Re: 回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in StreamsResetter
Hey Feyman,
I think Matthias' suggestion is optional, and we could just use admin tool
to remove single static members as well.
Boyang
On Tue, Apr 7, 2020 at 11:00 AM Matthias J. Sax <mj...@apache.org> wrote:
> > Would you mind to elaborate why we still need that
>
> Sure.
>
> For static memership, the session timeout it usually set quite high.
> This make scaling in an application tricky: if you shut down one
> instance, no rebalance would happen until `session.timeout.ms` hits.
> This is specific to Kafka Streams, because when a Kafka Stream client is
> closed, it does _not_ send a `LeaveGroupRequest`. Hence, the
> corresponding partitions would not be processed for a long time and
> thus, fall back.
>
> Given that each instance will have a unique `instance.id` provided by
> the user, we could allow users to remove the instance they want to
> decommission from the consumer group without the need to wait for
> `session.timeout.ms`.
>
> Hence, it's not an application reset scenario for which one wants to
> remove all members, but a scaling-in scenario. For dynamic membership,
> this issue usually does not occur because the `session.timeout.ms` is
> set to a fairly low value and a rebalance would happen quickly after an
> instance is decommissioned.
>
> Does this make sense?
>
> As said before, we may or may not include this in this KIP. It's up to
> you if you want to address it or not.
>
>
> -Matthias
>
>
>
> On 4/7/20 7:12 AM, feyman2009 wrote:
> > Hi, Matthias
> > Thanks a lot!
> > So you do not plan so support removing a _single static_ member via
> `StreamsResetter`?
> > =>
> > Would you mind to elaborate why we still need that if we are
> able to batch remove active members with adminClient?
> >
> > Thanks!
> >
> > Feyman
> > ------------------------------------------------------------------
> > 发件人：Matthias J. Sax <mj...@apache.org>
> > 发送时间：2020年4月7日(星期二) 08:25
> > 收件人：dev <de...@kafka.apache.org>
> > 主 题：Re: 回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members
> in StreamsResetter
> >
> > Overall LGTM.
> >
> > +1 (binding)
> >
> > So you do not plan so support removing a _single static_ member via
> > `StreamsResetter`? We can of course still add this as a follow up but it
> > might be nice to just add it to this KIP right away. Up to you if you
> > want to include it or not.
> >
> >
> > -Matthias
> >
> >
> >
> > On 3/30/20 8:16 AM, feyman2009 wrote:
> >> Hi, Boyang
> >> Thanks a lot, that make sense, we should not expose member.id in
> the MemberToRemove anymore, I have fixed it in the KIP.
> >>
> >>
> >> Feyman
> >> ------------------------------------------------------------------
> >> 发件人：Boyang Chen <re...@gmail.com>
> >> 发送时间：2020年3月29日(星期日) 01:45
> >> 收件人：dev <de...@kafka.apache.org>; feyman2009 <fe...@aliyun.com>
> >> 主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove members in
> StreamsResetter
> >>
> >> Hey Feyman,
> >>
> >> thanks for the update. I assume we would rely entirely on the internal
> changes for `removeMemberFromGroup` by sending a DescribeGroup request
> first. With that in mind, I don't think we need to add member.id to
> MemberToRemove anymore, as it is only facing public where users will only
> configure group.instance.id?
> >> On Fri, Mar 27, 2020 at 5:04 PM feyman2009
> <fe...@aliyun.com.invalid> wrote:
> >> Bump, can anyone kindly take a look at the updated KIP-571? Thanks!
> >>
> >>
> >> ------------------------------------------------------------------
> >> 发件人：feyman2009 <fe...@aliyun.com.INVALID>
> >> 发送时间：2020年3月23日(星期一) 08:51
> >> 收件人：dev <de...@kafka.apache.org>
> >> 主 题：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in
> StreamsResetter
> >>
> >> Hi, team
> >> I have updated the KIP-571 according to our latest discussion
> results, would you mind to take a look? Thanks!
> >>
> >> Feyman
> >>
> >>
> >> ------------------------------------------------------------------
> >> 发件人：Boyang Chen <re...@gmail.com>
> >> 发送时间：2020年3月19日(星期四) 13:41
> >> 收件人：dev <de...@kafka.apache.org>; feyman2009 <fe...@aliyun.com>
> >> 主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove members in
> StreamsResetter
> >>
> >> Thanks for the insight Feyman. I personally feel adding another admin
> client command is redundant, so picking option 1). The memberInfos struct
> is internal and just used for result reference purposes. I think it could
> still work even we overload with `removeAll` option, if that makes sense.
> >>
> >> Boyang
> >> On Wed, Mar 18, 2020 at 8:51 PM feyman2009
> <fe...@aliyun.com.invalid> wrote:
> >> Hi, team
> >> Before going too far on the KIP update, I would like to hear your
> opinions about how we would change the interface of AdminClient, the two
> alternatives I could think of are:
> >> 1) Extend adminClient.removeMembersFromConsumerGroup to support
> remove all
> >> As Guochang suggested, we could add some flag param in
> RemoveMembersFromConsumerGroupOptions to indicated the "remove all" logic.
> >> 2) Add a new API like
> adminClient.removeAllMembersFromConsumerGroup(groupId, options)
> >>
> >> I think 1) will be more compact from the API perspective, but
> looking at the code, I found that the if we are going to remove all, then
> the RemoveMembersFromConsumerGroupResult#memberInfos/memberResult()/all()
> should be changed accordingly, and they seem not that meaningful under the
> "remove all" scenario.
> >>
> >> A minor thought about the
> adminClient.removeMembersFromConsumerGroup API is:
> >> Looking at some other deleteXX APIs, like deleteTopics,
> deleteRecords, the results contains only a Map<A, Future<B>>, I think it's
> enough to describe the related results, is it make sense that we may remove
> memberInfos in RemoveMembersFromConsumerGroupResult ? This KIP has no
> dependency on this if we choose alternative 2)
> >>
> >> Could you advise? Thanks!
> >>
> >> Feyman
> >>
> >>
> >> 送时间：2020年3月15日(星期日) 10:11
> >> 收件人：dev <de...@kafka.apache.org>
> >> 主 题：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in
> StreamsResetter
> >>
> >> Hi, all
> >> Thanks a lot for your feedback!
> >> According to the discussion, it seems we don't have some valid
> use cases for removing specific dynamic members, I think it makes sense to
> encapsulate the "get and delete" logic in adminClient. I will update the
> KIP shortly!
> >>
> >> Thanks!
> >>
> >> Feyman
> >>
> >>
> >> ------------------------------------------------------------------
> >> 发件人：Boyang Chen <re...@gmail.com>
> >> 发送时间：2020年3月14日(星期六) 00:39
> >> 收件人：dev <de...@kafka.apache.org>
> >> 主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove members
> in StreamsResetter
> >>
> >> Thanks Matthias and Guozhang for the feedback. I'm not worrying too
> much
> >> about the member.id exposure as we have done so in a couple of
> areas. As
> >> for the recommended admin client change, I think it makes sense in an
> >> encapsulation perspective. Maybe I'm still a bit hesitant as we are
> losing
> >> the flexibility of closing only a subset of `dynamic members`
> potentially,
> >> but we could always get back and address it if some user feels
> necessary to
> >> have it.
> >>
> >> My short answer would be, LGTM :)
> >>
> >> Boyang
> >>
> >> On Thu, Mar 12, 2020 at 5:26 PM Guozhang Wang <wa...@gmail.com>
> wrote:
> >>
> >> > Hi Matthias,
> >> >
> >> > About the AdminClient param API: that's a great point here. I think
> overall
> >> > if users want to just "remove all members" they should not need to
> first
> >> > get all the member.ids themselves, but instead internally the admin
> client
> >> > can first issue a describe-group request to get all the member.ids,
> and
> >> > then use them in the next issued leave-group request, all
> abstracted away
> >> > from the users. With that in mind, maybe in
> >> > RemoveMembersFromConsumerGroupOptions we can just introduce an
> overloaded
> >> > flag param besides "members" that indicate "remove all"?
> >> >
> >> > Guozhang
> >> >
> >> > On Thu, Mar 12, 2020 at 2:59 PM Matthias J. Sax <mj...@apache.org>
> wrote:
> >> >
> >> > > Feyman,
> >> > >
> >> > > some more comments/questions:
> >> > >
> >> > > The description of `LeaveGroupRequest` is clear but it's unclear
> how
> >> > > `MemberToRemove` should behave. Which parameter is required?
> Which is
> >> > > optional? What is the relationship between both.
> >> > >
> >> > > The `LeaveGroupRequest` description clearly states that
> specifying a
> >> > > `memberId` is optional if the `groupInstanceId` is provided. If
> >> > > `MemberToRemove` applies the same pattern, it must be explicitly
> defined
> >> > > in the KIP (and explained in the JavaDocs of `MemberToRemove`)
> because
> >> > > we cannot expect that an admin-client users knows that internally
> a
> >> > > `LeaveGroupRequest` is used nor what the semantics of a
> >> > > `LeaveGroupRequest` are.
> >> > >
> >> > >
> >> > > About Admin API:
> >> > >
> >> > > In general, I am also confused that we allow so specify a
> `memberId` at
> >> > > all, because the `memberId` is an internal id that is not really
> exposed
> >> > > to the user. Hence, from a AdminClient point of view, accepting a
> >> > > `memberId` as input seems questionable to me? Of course,
> `memberId` can
> >> > > be collected via `describeConsumerGroups()` but it will return the
> >> > > `memberId` of _all_ consumer in the group and thus how would a
> user know
> >> > > which member should be removed for a dynamic group (if an
> individual
> >> > > member should be removed)?
> >> > >
> >> > > Hence, how can any user get to know the `memberId` of an
> individual
> >> > > client in a programtic way?
> >> > >
> >> > > Also I am wondering in general, why the removal of single dynamic
> member
> >> > > is important? In general, I would expect a short
> `session.timeout` for
> >> > > dynamic groups and thus removing a specific member from the group
> seems
> >> > > not to be an important feature -- for static groups we expect a
> long
> >> > > `session.timeout` and a user can also identify individual clients
> via
> >> > > `groupInstandId`, hence the feature makes sense for this case and
> is
> >> > > straight forward to use.
> >> > >
> >> > >
> >> > > About StreamsResetter:
> >> > >
> >> > > For this case we just say "remove all members" and thus the
> >> > > `describeConsumerGroup` approach works. However, it seems to be a
> >> > > special case?
> >> > >
> >> > > Or, if we expected that the "remove all members" use case is the
> norm,
> >> > > why can't we make a change admin-client to directly accept a `
> group.id`?
> >> > > The admin-client can internal first do a `DescribeGroupRequest`
> and
> >> > > afterward corresponding `LeaveGroupRequest` -- i.e., instead of
> building
> >> > > this pattern in `StreamsResetter` we build it directly into
> >> > `AdminClient`.
> >> > >
> >> > > Last, for static group the main use case seems to be to remove an
> >> > > individual member from the group but this feature is not covered
> by the
> >> > > KIP: I think using `--force` to remove all members makes sense,
> but an
> >> > > important second feature to remove an individual static member
> would
> >> > > require it's own flag to specify a single `group.instance.id`.
> >> > >
> >> > >
> >> > > Thoughts?
> >> > >
> >> > >
> >> > > -Matthias
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > On 3/11/20 8:43 PM, feyman2009 wrote:
> >> > > > Hi, Sophie
> >> > > > For 1) Sorry, I found that my expression is kind of
> misleading,
> >> > > what I actually mean is: "if --force not specified, an exception
> saying
> >> > > there are still active members on broker side will be thrown and
> >> > > suggesting using StreamsResetter with --force", I just updated
> the KIP
> >> > > page.
> >> > > >
> >> > > > For 2)
> >> > > > I may also had some misleading expression previous, to
> clarify
> >> > :
> >> > > >
> >> > > > Also, it's more efficient to just send a single "clear the
> group"
> >> > > request vs sending a LeaveGroup
> >> > > > request for every single member. What do you think?
> >> > > > => the comparison is to send a single "clear the group" request
> vs
> >> > > sending a "get members" + a "remove members" request since the
> >> > > adminClient.removeMembersFromConsumerGroup support batch removal.
> We
> >> > > don't need to send lots of leaveGroup requests for every single
> member.
> >> > > >
> >> > > > I can understand your point, but I think we could reuse
> the
> >> > > current
> >> > > > adminClient.removeMembersFromConsumerGroup interface
> effectively with
> >> > > the KIP.
> >> > > > What do you think?
> >> > > >
> >> > > > Thanks!
> >> > > >
> >> > > > Feyman
> >> > > >
> >> > > >
> >> > > >
> ------------------------------------------------------------------
> >> > > > 发件人：Sophie Blee-Goldman <so...@confluent.io>
> >> > > > 发送时间：2020年3月10日(星期二) 03:02
> >> > > > 收件人：dev <de...@kafka.apache.org>; feyman2009 <
> feyman2009@aliyun.com>
> >> > > > 主 题：Re: 回复：回复：[Vote] KIP-571: Add option to force remove
> >> > > members in StreamsResetter
> >> > > >
> >> > > > Hey Feyman,
> >> > > >
> >> > > > 1) Regarding point 2 in your last email, if I understand
> correctly you
> >> > > propose to change
> >> > > > the current behavior of the reset tool when --force is not
> specified,
> >> > > and wait for (up to)
> >> > > > the session timeout for all members to be removed. I'm not sure
> we
> >> > > should change this,
> >> > > > especially now that we have a better way to handle the case
> when the
> >> > > group is not empty:
> >> > > > we should continue to throw an exception and fail fast, but can
> print
> >> > > a message suggesting
> >> > > > to use the new --force option to remove remaining group
> members. Why
> >> > > make users wait
> >> > > > for the session timeout when we've just added a new feature
> that means
> >> > > they don't have to?
> >> > > >
> >> > > > 2) Regarding Matthias' question:
> >> > > >
> >> > > >> I am really wondering, if for a static group, we should allow
> users
> >> > > toremove individual members? For a dynamic group this feature
> would not
> >> > > > make much sense IMHO, because the `memberId` is not know by the
> user.
> >> > > >
> >> > > > I think his point is similar to what I was trying to get at
> earlier,
> >> > > with the proposal to add a new
> >> > > > #removeAllMembers API rather than an API to remove individual
> members
> >> > > according to their
> >> > > > memberId. As he explained, removing based on memberId is likely
> not
> >> > > that useful in general.
> >> > > > Also, it's not actually what we want to do here; maybe we
> should avoid
> >> > > adding a new API
> >> > > > that we think may be useful in other contexts (remove individual
> >> > > member based on memberId),
> >> > > > and just add the API we actually need (remove all members from
> group)
> >> > > in this KIP? We can
> >> > > > always add the "remove individual member by memberId" API at a
> later
> >> > > point, if it turns out to
> >> > > > actually be requested for specific reasons?
> >> > > >
> >> > > > Also, it's more efficient to just send a single "clear the
> group"
> >> > > request vs sending a LeaveGroup
> >> > > > request for every single member. What do you think?
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Sat, Mar 7, 2020 at 1:41 AM feyman2009
> >> > > <fe...@aliyun.com.invalid> wrote:
> >> > > > Hi, Matthias
> >> > > > Thanks, I updated the KIP to mention the deprecated and
> newly
> >> > > added methods.
> >> > > >
> >> > > > 1. What happens is `groupInstanceId` is used for a dynamic
> group? What
> >> > > > happens if both parameters are specified? What happens if
> `memberId`
> >> > > > is specified for a static group?
> >> > > >
> >> > > > => my understanding is that the dynamic/static membership is
> member
> >> > > level other than group level, and I think above questions could be
> >> > > answered by the "Leave Group Logic Change" section in KIP-345:
> >> > >
> >> > >
> >> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> >> > > ,
> >> > > this KIP stays consistent with KIP-345.
> >> > > >
> >> > > > 2. About the `--force` option. Currently, StreamsResetter
> fails with
> >> > an
> >> > > > error if the consumer group is not empty. You state in your
> KIP that:
> >> > > >
> >> > > > > without --force, we need to wait for session timeout.
> >> > > >
> >> > > > Is this an intended behavior change if `--force` is not used
> or is the
> >> > > > KIP description incorrect?
> >> > > >
> >> > > > => This is the intended behavior. For this part, I think there
> are
> >> > > two ways to go:
> >> > > > 1) (the implicit way) Not introducing the new "--force"
> option, with
> >> > > this KIP, StreamsResetter will by default remove active
> members(with
> >> > > long session timeout configured) on broker side
> >> > > > 2) (the explicit way) Introduce the new "--force" option,
> users need
> >> > > to explicitly specify --force to remove the active members. If
> --force
> >> > > not specified, the StreamsResetter behaviour is as previous
> versions'.
> >> > > >
> >> > > > I think the two alternatives above are both feasible,
> personally I
> >> > > prefer way 2.
> >> > > >
> >> > > > 3. For my own understanding: with the `--force` option, we
> intend to
> >> > get
> >> > > > all `memberIds` and send a "remove member" request for each
> with
> >> > > > corresponding `memberId` to remove the member from the group
> >> > > > (independent is the group is static or dynamic)?
> >> > > >
> >> > > > => Yeah, minor thing to mention is we will send the "remove
> member"
> >> > > request for each member(could be dynamic member or static member)
> to
> >> > > remove them from group
> >> > > > for dynamic members, both "group.instance.id" and "member.id"
> will be
> >> > > specified
> >> > > > for dynamic members, only "member.id" will be specified
> >> > > >
> >> > > > 4. I am really wondering, if for a static group, we should
> allow users
> >> > > to
> >> > > > remove individual members? For a dynamic group this feature
> would not
> >> > > > make much sense IMHO, because the `memberId` is not know by
> the user.
> >> > > >
> >> > > > => KIP-345 introduced the batch removal feature for both static
> >> > > member and dynamic member, my understanding is that "allow users
> to
> >> > > > remove individual members" could be useful for rolling bounce
> and
> >> > > scale down accoding to KIP-345. KafkaAdminClient currently only
> support
> >> > > static member removal and this KIP-571 enables dynamic member
> removal
> >> > > for it, which is also consistent with the broker side logic.
> Users could
> >> > > get the member.id (and group.instance.id if for static member) by
> >> > > adminClient.describeConsumerGroups.
> >> > > >
> >> > > > Furthermore, I don't have an assumption that a consumer group
> should
> >> > > contain only static or dynamic members, and I think KIP-345 and
> this KIP
> >> > > don't need to be based on this assumption.
> >> > > > You could correct me if I have the wrong understanding :)
> >> > > >
> >> > > > Thanks!
> >> > > > Feyman
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> ------------------------------------------------------------------
> >> > > > 发件人：Matthias J. Sax <mj...@apache.org>
> >> > > > 发送时间：2020年3月6日(星期五) 02:20
> >> > > > 收件人：dev <de...@kafka.apache.org>
> >> > > > 主 题：Re: 回复：回复：[Vote] KIP-571: Add option to force remove
> >> > > members in StreamsResetter
> >> > > >
> >> > > > Feyman,
> >> > > >
> >> > > > thanks a lot for the KIP. Overall LGTM. I have a few more
> comment and
> >> > > > questions (sorry for the late reply):
> >> > > >
> >> > > >
> >> > > > The KIP mentions that some constructors will be deprecated.
> Those
> >> > should
> >> > > > be listed explicitly. For example:
> >> > > >
> >> > > >
> >> > > > public class MemberToRemove {
> >> > > >
> >> > > > // deprecated methods
> >> > > >
> >> > > > @Deprecated
> >> > > > public MemberToRemove(String groupInstanceId);
> >> > > >
> >> > > > // new methods
> >> > > >
> >> > > > public MemberToRemove()
> >> > > >
> >> > > > public MemberToRemove withGroupInstanceId(String
> groupInstanceId)
> >> > > >
> >> > > > public MemberToRemove withMemberId(String memberId)
> >> > > > }
> >> > > >
> >> > > > What happens is `groupInstanceId` is used for a dynamic group?
> What
> >> > > > happens if both parameters are specified? What happens if
> `memberId`
> >> > > > is specified for a static group?
> >> > > >
> >> > > >
> >> > > > About the `--force` option. Currently, StreamsResetter fails
> with an
> >> > > > error if the consumer group is not empty. You state in your KIP
> that:
> >> > > >
> >> > > >> without --force, we need to wait for session timeout.
> >> > > >
> >> > > > Is this an intended behavior change if `--force` is not used or
> is the
> >> > > > KIP description incorrect?
> >> > > >
> >> > > > For my own understanding: with the `--force` option, we intend
> to get
> >> > > > all `memberIds` and send a "remove member" request for each with
> >> > > > corresponding `memberId` to remove the member from the group
> >> > > > (independent is the group is static or dynamic)?
> >> > > >
> >> > > > I am really wondering, if for a static group, we should allow
> users to
> >> > > > remove individual members? For a dynamic group this feature
> would not
> >> > > > make much sense IMHO, because the `memberId` is not know by the
> user.
> >> > > >
> >> > > >
> >> > > >
> >> > > > -Matthias
> >> > > >
> >> > > >
> >> > > > On 3/5/20 7:15 AM, Bill Bejeck wrote:
> >> > > >> Thanks for the KIP. +1 (binding).
> >> > > >
> >> > > >> -Bill
> >> > > >
> >> > > >
> >> > > >
> >> > > >> On Wed, Mar 4, 2020 at 12:40 AM Guozhang Wang <
> wangguoz@gmail.com>
> >> > > >> wrote:
> >> > > >
> >> > > >>> Thanks, +1 from me (binding).
> >> > > >>>
> >> > > >>> On Tue, Mar 3, 2020 at 9:39 PM feyman2009 <
> feyman2009@aliyun.com>
> >> > > >>> wrote:
> >> > > >>>
> >> > > >>>> Hi, Guozhang Thanks a lot for the advice, that make sense! I
> >> > > >>>> have updated the KIP page with the operational steps of
> >> > > >>>> StreamsResetter.
> >> > > >>>>
> >> > > >>>> Thanks! Feyman
> >> > > >>>>
> >> > > >>>>
> ------------------------------------------------------------------
> >> > > >>>>
> >> > > >>>>
> >> > > > 发件人：Guozhang Wang <wa...@gmail.com>
> >> > > >>>> 发送时间：2020年3月3日(星期二) 14:22 收件人：dev <de...@kafka.apache.org>;
> >> > > >>>> feyman2009 <fe...@aliyun.com> 主 题：Re: 回复：回复：[Vote]
> >> > > >>>> KIP-571: Add option to force remove members in
> StreamsResetter
> >> > > >>>>
> >> > > >>>> Hello Feyman, thanks for the proposal!
> >> > > >>>>
> >> > > >>>> I read through the doc and overall it looks good to me.
> >> > > >>>>
> >> > > >>>> One minor thing I'd still like to point out is that, the
> >> > > >>>> "removeMembersFromConsumerGroup" only sends a leave-group
> >> > > >>>> request to the coordinator to let it remove the member,
> >> > > >>>> however, if the member is still there alive and running then
> it
> >> > > >>>> would soon be notified that it is no
> >> > > >>> longer
> >> > > >>>> a legal member of the group via heartbeats, and then
> >> > > >>>> automatically tries
> >> > > >>> to
> >> > > >>>> re-join the group. So on the operational side, it is still
> >> > > >>>> required that the following steps:
> >> > > >>>>
> >> > > >>>> 1) first stop the consumers (of streams instances), wait
> until
> >> > > >>>> the shutdown is complete. 2) then use admin client in case
> the
> >> > > >>>> stopped consumers are still registered at the broker side and
> >> > > >>>> we do not want to wait for session timeout.
> >> > > >>>>
> >> > > >>>> Even with this KIP, people should still not skip step 1)
> above,
> >> > > >>>> since otherwise the consumers would re-connect and re-join
> the
> >> > > >>>> group
> >> > > >>> immediately
> >> > > >>>> still.
> >> > > >>>>
> >> > > >>>> In your doc you've already mentioned "Furthermore, users
> should
> >> > > >>>> make sure all the stream applications are shutdown when
> running
> >> > > >>>> StreamsResetter
> >> > > >>> with
> >> > > >>>> --force, otherwise it might trigger unexpected rebalance. "
> >> > > >>>> What I'd want to clarify is that no matter if "--force"
> option
> >> > > >>>> is enabled, this is
> >> > > >>> always
> >> > > >>>> the case that users should shutdown the streams instances
> >> > > >>>> first, and then use the streams resetter :)
> >> > > >>>>
> >> > > >>>> As long as that is clarified in the proposal documentation,
> I'm
> >> > > >>>> +1 on
> >> > > >>> this
> >> > > >>>> KIP.
> >> > > >>>>
> >> > > >>>>
> >> > > >>>> Thanks again for the contribution, Guozhang
> >> > > >>>>
> >> > > >>>>
> >> > > >>>> On Mon, Mar 2, 2020 at 6:31 AM feyman2009
> >> > > >>>> <feyman2009@aliyun.com.invalid
> >> > > >>>>
> >> > > >>>> wrote: Hi, John Sorry, I have mistaken the KIP approval
> >> > > >>>> standard, anyway, I will
> >> > > >>> start
> >> > > >>>> the PR soon and waiting for more binding approvals.
> >> > > >>>>
> >> > > >>>> Thanks! Feyman
> >> > > >>>>
> >> > > >>>>
> >> > > >>>>
> ------------------------------------------------------------------
> >> > > >>>>
> >> > > >>>>
> >> > > > 发件人：John Roesler <vv...@apache.org>
> >> > > >>>> 发送时间：2020年3月2日(星期一) 22:00 收件人：dev
> >> > > > <de...@kafka.apache.org> 主
> >> > > >>>> 题：Re: 回复：回复：[Vote] KIP-571: Add option to force remove
> members
> >> > > >>>> in StreamsResetter
> >> > > >>>>
> >> > > >>>> Hi Feyman,
> >> > > >>>>
> >> > > >>>> Sorry, but we actually need 3 binding votes for the KIP to
> >> > > >>>> pass. Please feel free to keep bumping the thread until some
> >> > > >>>> more committers can take
> >> > > >>> a
> >> > > >>>> look.
> >> > > >>>>
> >> > > >>>> By the way, you can totally start a PR, but we can’t merge it
> >> > > >>>> until the KIP passes the vote.
> >> > > >>>>
> >> > > >>>> Thanks! John
> >> > > >>>>
> >> > > >>>> On Mon, Mar 2, 2020, at 00:24, feyman2009 wrote:
> >> > > >>>>> Hi,all Since currently we have 1 binding and two non-binding
> >> > > >>>>> +1, I will update the KIP-571 as adopted and initiate a PR
> >> > > >>>>> shortly
> >> > > >>>>>
> >> > > >>>>> Thanks! Feyman
> >> > > >>>>>
> >> > > >>>>>
> >> > > >>>>>
> ------------------------------------------------------------------
> >> > > >>>>>
> >> > > >>>>>
> >> > > > 发件人：Sophie Blee-Goldman <so...@confluent.io>
> >> > > >>>>> 发送时间：2020年2月28日(星期五) 10:17 收件人：dev
> >> > > > <de...@kafka.apache.org> 主
> >> > > >>>>> 题：Re: 回复：[Vote] KIP-571: Add option to force remove members
> >> > > >>>>> in
> >> > > >>>> StreamsResetter
> >> > > >>>>>
> >> > > >>>>> Thanks for the KIP, +1 (non-binding)
> >> > > >>>>>
> >> > > >>>>> On Thu, Feb 27, 2020 at 12:40 PM Boyang Chen <
> >> > > >>> reluctanthero104@gmail.com
> >> > > >>>>>
> >> > > >>>>> wrote:
> >> > > >>>>>
> >> > > >>>>>> Thanks Feyman, +1 (non-binding)
> >> > > >>>>>>
> >> > > >>>>>> On Thu, Feb 27, 2020 at 9:25 AM John Roesler
> >> > > >>>>>> <vv...@apache.org>
> >> > > >>>> wrote:
> >> > > >>>>>>
> >> > > >>>>>>> Thanks for the proposal!
> >> > > >>>>>>>
> >> > > >>>>>>> I'm +1 (binding) -John
> >> > > >>>>>>>
> >> > > >>>>>>> On Wed, Feb 26, 2020, at 19:41, feyman2009 wrote:
> >> > > >>>>>>>> Updated with the KIP link:
> >> > > >>>>>>>>
> >> > > >>>>>>>
> >> > > >>>>>>
> >> > > >>>>
> >> > > >>>
> >> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-571%3A+Add+opti
> >> > > > on+to+force+remove+members+in+StreamsResetter
> >> > > >>>>>>>>
> >> > > >>>>>>>>
> >> > > >>>>>>>>
> >> > > >>>
> >> > > >>>
> >> > > >
> ------------------------------------------------------------------
> >> > > >>>>>>>> 发件人：feyman2009 <fe...@aliyun.com.INVALID> 发送时
> >> > > >>>>>>>> 间：2020年2月27日(星期四) 09:38 收件人：dev <de...@kafka.apache.org>
> >> > > >>>>>>>> 主 题：[Vote] KIP-571: Add option to force remove members
> >> > > >>>>>>>> in
> >> > > >>>>>> StreamsResetter
> >> > > >>>>>>>>
> >> > > >>>>>>>>
> >> > > >>>>>>>> Hi, all I would like to start a vote on KIP-571: Add
> >> > > >>>>>>>> option to force
> >> > > >>>> remove
> >> > > >>>>>>>> members in StreamsResetter .
> >> > > >>>>>>>>
> >> > > >>>>>>>> Thanks! Feyman
> >> > > >>>>>>>>
> >> > > >>>>>>>>
> >> > > >>>>>>>
> >> > > >>>>>>
> >> > > >>>>>
> >> > > >>>>>
> >> > > >>>>
> >> > > >>>>
> >> > > >>>>
> >> > > >>>> -- -- Guozhang
> >> > > >>>>
> >> > > >>>>
> >> > > >>>>
> >> > > >>>
> >> > > >>> -- -- Guozhang
> >> > > >>>
> >> > > >
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> >
> >> > --
> >> > -- Guozhang
> >> >
> >>
> >>
> >>
> >>
> >>
> >
> >
>
>
回复：回复：回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in StreamsResetter
Posted by feyman2009 <fe...@aliyun.com.INVALID>.
Hi, team
I updated the KIP-571 since we took a slightly different implementation in the PR https://github.com/apache/kafka/pull/8589, basically:
In RemoveMembersFromConsumerGroupOptions, leveraging empty members rather than introducing a new field to imply the removeAll scenario.
Please let me know if you have any concerns, thanks a lot!
Feyman
------------------------------------------------------------------
发件人：feyman2009 <fe...@aliyun.com.INVALID>
发送时间：2020年4月13日(星期一) 08:47
收件人：dev <de...@kafka.apache.org>
主 题：回复：回复：回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in StreamsResetter
Thanks , John and Guochang!
------------------------------------------------------------------
发件人：Guozhang Wang <wa...@gmail.com>
发送时间：2020年4月11日(星期六) 03:07
收件人：dev <de...@kafka.apache.org>
主 题：Re: 回复：回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in StreamsResetter
Thanks Feyman,
I've looked at the update that you incorporated from Matthias and that LGTM
too. I'm still +1 :)
Guozhang
On Fri, Apr 10, 2020 at 11:18 AM John Roesler <jo...@vvcephei.org> wrote:
> Hey Feyman,
>
> Just to remove any ambiguity, I've been casually following the discussion,
> I've just looked at the KIP document again, and I'm still +1 (binding).
>
> Thanks,
> -John
>
> On Fri, Apr 10, 2020, at 01:44, feyman2009 wrote:
> > Hi, all
> > KIP-571 has already collected 4 bind +1 (John, Guochang, Bill,
> > Matthias) and 3 non-binding +1(Boyang, Sophie), I will mark it as
> > approved and create a PR shortly.
> > Thanks!
> >
> > Feyman
> > ------------------------------------------------------------------
> > 发件人：feyman2009 <fe...@aliyun.com.INVALID>
> > 发送时间：2020年4月8日(星期三) 14:21
> > 收件人：dev <de...@kafka.apache.org>; Boyang Chen <re...@gmail.com>
> > 主 题：回复：回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove
> > members in StreamsResetter
> >
> > Hi Boyang,
> > Thanks for reminding me of that!
> > I'm not sure about the convention, I thought it would need to
> > re-collect votes if the KIP has changed~
> > Let's leave the vote thread here for 2 days, if no objection, I
> > will take it as approved and update the PR accordingly.
> >
> > Thanks!
> > Feyman
> >
> >
> >
> > ------------------------------------------------------------------
> > 发件人：Boyang Chen <re...@gmail.com>
> > 发送时间：2020年4月8日(星期三) 12:42
> > 收件人：dev <de...@kafka.apache.org>; feyman2009 <fe...@aliyun.com>
> > 主 题：Re: 回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove
> > members in StreamsResetter
> >
> > You should already get enough votes if I'm counting correctly
> > (Guozhang, John, Matthias)
> > On Tue, Apr 7, 2020 at 6:59 PM feyman2009
> > <fe...@aliyun.com.invalid> wrote:
> > Hi, Boyang&Matthias
> > I think Matthias's proposal makes sense, but we can use the admin
> > tool for this scenario as Boyang mentioned or follow up later, so I
> > prefer to keep this KIP unchanged to minimize the scope.
> > Calling for vote ~
> >
> > Thanks!
> > Feyman
> >
> > ------------------------------------------------------------------
> > 发件人：Boyang Chen <re...@gmail.com>
> > 发送时间：2020年4月8日(星期三) 02:15
> > 收件人：dev <de...@kafka.apache.org>
> > 主 题：Re: 回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove
> > members in StreamsResetter
> >
> > Hey Feyman,
> >
> > I think Matthias' suggestion is optional, and we could just use admin
> tool
> > to remove single static members as well.
> >
> > Boyang
> >
> > On Tue, Apr 7, 2020 at 11:00 AM Matthias J. Sax <mj...@apache.org>
> wrote:
> >
> > > > Would you mind to elaborate why we still need that
> > >
> > > Sure.
> > >
> > > For static memership, the session timeout it usually set quite high.
> > > This make scaling in an application tricky: if you shut down one
> > > instance, no rebalance would happen until `session.timeout.ms` hits.
> > > This is specific to Kafka Streams, because when a Kafka Stream
> > client is
> > > closed, it does _not_ send a `LeaveGroupRequest`. Hence, the
> > > corresponding partitions would not be processed for a long time and
> > > thus, fall back.
> > >
> > > Given that each instance will have a unique `instance.id` provided by
> > > the user, we could allow users to remove the instance they want to
> > > decommission from the consumer group without the need to wait for
> > > `session.timeout.ms`.
> > >
> > > Hence, it's not an application reset scenario for which one wants to
> > > remove all members, but a scaling-in scenario. For dynamic
> > membership,
> > > this issue usually does not occur because the `session.timeout.ms` is
> > > set to a fairly low value and a rebalance would happen quickly after
> > an
> > > instance is decommissioned.
> > >
> > > Does this make sense?
> > >
> > > As said before, we may or may not include this in this KIP. It's up
> > to
> > > you if you want to address it or not.
> > >
> > >
> > > -Matthias
> > >
> > >
> > >
> > > On 4/7/20 7:12 AM, feyman2009 wrote:
> > > > Hi, Matthias
> > > > Thanks a lot!
> > > > So you do not plan so support removing a _single static_
> > member via
> > > `StreamsResetter`?
> > > > =>
> > > > Would you mind to elaborate why we still need that if we
> > are
> > > able to batch remove active members with adminClient?
> > > >
> > > > Thanks!
> > > >
> > > > Feyman
> > > > ------------------------------------------------------------------
> > > > 发件人：Matthias J. Sax <mj...@apache.org>
> > > > 发送时间：2020年4月7日(星期二) 08:25
> > > > 收件人：dev <de...@kafka.apache.org>
> > > > 主 题：Re: 回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove
> > members
> > > in StreamsResetter
> > > >
> > > > Overall LGTM.
> > > >
> > > > +1 (binding)
> > > >
> > > > So you do not plan so support removing a _single static_ member via
> > > > `StreamsResetter`? We can of course still add this as a follow up
> > but it
> > > > might be nice to just add it to this KIP right away. Up to you if
> > you
> > > > want to include it or not.
> > > >
> > > >
> > > > -Matthias
> > > >
> > > >
> > > >
> > > > On 3/30/20 8:16 AM, feyman2009 wrote:
> > > >> Hi, Boyang
> > > >> Thanks a lot, that make sense, we should not expose member.id
> > in
> > > the MemberToRemove anymore, I have fixed it in the KIP.
> > > >>
> > > >>
> > > >> Feyman
> > > >> ------------------------------------------------------------------
> > > >> 发件人：Boyang Chen <re...@gmail.com>
> > > >> 发送时间：2020年3月29日(星期日) 01:45
> > > >> 收件人：dev <de...@kafka.apache.org>; feyman2009 <fe...@aliyun.com>
> > > >> 主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove
> > members in
> > > StreamsResetter
> > > >>
> > > >> Hey Feyman,
> > > >>
> > > >> thanks for the update. I assume we would rely entirely on the
> > internal
> > > changes for `removeMemberFromGroup` by sending a DescribeGroup
> > request
> > > first. With that in mind, I don't think we need to add member.id to
> > > MemberToRemove anymore, as it is only facing public where users will
> > only
> > > configure group.instance.id?
> > > >> On Fri, Mar 27, 2020 at 5:04 PM feyman2009
> > > <fe...@aliyun.com.invalid> wrote:
> > > >> Bump, can anyone kindly take a look at the updated KIP-571?
> > Thanks!
> > > >>
> > > >>
> > > >>
> > ------------------------------------------------------------------
> > > >> 发件人：feyman2009 <fe...@aliyun.com.INVALID>
> > > >> 发送时间：2020年3月23日(星期一) 08:51
> > > >> 收件人：dev <de...@kafka.apache.org>
> > > >> 主 题：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove
> > members in
> > > StreamsResetter
> > > >>
> > > >> Hi, team
> > > >> I have updated the KIP-571 according to our latest discussion
> > > results, would you mind to take a look? Thanks!
> > > >>
> > > >> Feyman
> > > >>
> > > >>
> > > >>
> > ------------------------------------------------------------------
> > > >> 发件人：Boyang Chen <re...@gmail.com>
> > > >> 发送时间：2020年3月19日(星期四) 13:41
> > > >> 收件人：dev <de...@kafka.apache.org>; feyman2009
> > <fe...@aliyun.com>
> > > >> 主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove
> > members in
> > > StreamsResetter
> > > >>
> > > >> Thanks for the insight Feyman. I personally feel adding another
> > admin
> > > client command is redundant, so picking option 1). The memberInfos
> > struct
> > > is internal and just used for result reference purposes. I think it
> > could
> > > still work even we overload with `removeAll` option, if that makes
> > sense.
> > > >>
> > > >> Boyang
> > > >> On Wed, Mar 18, 2020 at 8:51 PM feyman2009
> > > <fe...@aliyun.com.invalid> wrote:
> > > >> Hi, team
> > > >> Before going too far on the KIP update, I would like to
> > hear your
> > > opinions about how we would change the interface of AdminClient, the
> > two
> > > alternatives I could think of are:
> > > >> 1) Extend adminClient.removeMembersFromConsumerGroup to
> > support
> > > remove all
> > > >> As Guochang suggested, we could add some flag param in
> > > RemoveMembersFromConsumerGroupOptions to indicated the "remove all"
> > logic.
> > > >> 2) Add a new API like
> > > adminClient.removeAllMembersFromConsumerGroup(groupId, options)
> > > >>
> > > >> I think 1) will be more compact from the API perspective,
> > but
> > > looking at the code, I found that the if we are going to remove all,
> > then
> > > the
> > RemoveMembersFromConsumerGroupResult#memberInfos/memberResult()/all()
> > > should be changed accordingly, and they seem not that meaningful
> > under the
> > > "remove all" scenario.
> > > >>
> > > >> A minor thought about the
> > > adminClient.removeMembersFromConsumerGroup API is:
> > > >> Looking at some other deleteXX APIs, like deleteTopics,
> > > deleteRecords, the results contains only a Map<A, Future<B>>, I
> > think it's
> > > enough to describe the related results, is it make sense that we may
> > remove
> > > memberInfos in RemoveMembersFromConsumerGroupResult ? This KIP has no
> > > dependency on this if we choose alternative 2)
> > > >>
> > > >> Could you advise? Thanks!
> > > >>
> > > >> Feyman
> > > >>
> > > >>
> > > >> 送时间：2020年3月15日(星期日) 10:11
> > > >> 收件人：dev <de...@kafka.apache.org>
> > > >> 主 题：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove
> > members in
> > > StreamsResetter
> > > >>
> > > >> Hi, all
> > > >> Thanks a lot for your feedback!
> > > >> According to the discussion, it seems we don't have some
> > valid
> > > use cases for removing specific dynamic members, I think it makes
> > sense to
> > > encapsulate the "get and delete" logic in adminClient. I will update
> > the
> > > KIP shortly!
> > > >>
> > > >> Thanks!
> > > >>
> > > >> Feyman
> > > >>
> > > >>
> > > >>
> > ------------------------------------------------------------------
> > > >> 发件人：Boyang Chen <re...@gmail.com>
> > > >> 发送时间：2020年3月14日(星期六) 00:39
> > > >> 收件人：dev <de...@kafka.apache.org>
> > > >> 主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove
> > members
> > > in StreamsResetter
> > > >>
> > > >> Thanks Matthias and Guozhang for the feedback. I'm not worrying
> > too
> > > much
> > > >> about the member.id exposure as we have done so in a couple of
> > > areas. As
> > > >> for the recommended admin client change, I think it makes sense
> > in an
> > > >> encapsulation perspective. Maybe I'm still a bit hesitant as we
> > are
> > > losing
> > > >> the flexibility of closing only a subset of `dynamic members`
> > > potentially,
> > > >> but we could always get back and address it if some user feels
> > > necessary to
> > > >> have it.
> > > >>
> > > >> My short answer would be, LGTM :)
> > > >>
> > > >> Boyang
> > > >>
> > > >> On Thu, Mar 12, 2020 at 5:26 PM Guozhang Wang
> > <wa...@gmail.com>
> > > wrote:
> > > >>
> > > >> > Hi Matthias,
> > > >> >
> > > >> > About the AdminClient param API: that's a great point here. I
> > think
> > > overall
> > > >> > if users want to just "remove all members" they should not
> > need to
> > > first
> > > >> > get all the member.ids themselves, but instead internally the
> > admin
> > > client
> > > >> > can first issue a describe-group request to get all the
> > member.ids,
> > > and
> > > >> > then use them in the next issued leave-group request, all
> > > abstracted away
> > > >> > from the users. With that in mind, maybe in
> > > >> > RemoveMembersFromConsumerGroupOptions we can just introduce an
> > > overloaded
> > > >> > flag param besides "members" that indicate "remove all"?
> > > >> >
> > > >> > Guozhang
> > > >> >
> > > >> > On Thu, Mar 12, 2020 at 2:59 PM Matthias J. Sax
> > <mj...@apache.org>
> > > wrote:
> > > >> >
> > > >> > > Feyman,
> > > >> > >
> > > >> > > some more comments/questions:
> > > >> > >
> > > >> > > The description of `LeaveGroupRequest` is clear but it's
> > unclear
> > > how
> > > >> > > `MemberToRemove` should behave. Which parameter is required?
> > > Which is
> > > >> > > optional? What is the relationship between both.
> > > >> > >
> > > >> > > The `LeaveGroupRequest` description clearly states that
> > > specifying a
> > > >> > > `memberId` is optional if the `groupInstanceId` is
> > provided. If
> > > >> > > `MemberToRemove` applies the same pattern, it must be
> > explicitly
> > > defined
> > > >> > > in the KIP (and explained in the JavaDocs of
> > `MemberToRemove`)
> > > because
> > > >> > > we cannot expect that an admin-client users knows that
> > internally
> > > a
> > > >> > > `LeaveGroupRequest` is used nor what the semantics of a
> > > >> > > `LeaveGroupRequest` are.
> > > >> > >
> > > >> > >
> > > >> > > About Admin API:
> > > >> > >
> > > >> > > In general, I am also confused that we allow so specify a
> > > `memberId` at
> > > >> > > all, because the `memberId` is an internal id that is not
> > really
> > > exposed
> > > >> > > to the user. Hence, from a AdminClient point of view,
> > accepting a
> > > >> > > `memberId` as input seems questionable to me? Of course,
> > > `memberId` can
> > > >> > > be collected via `describeConsumerGroups()` but it will
> > return the
> > > >> > > `memberId` of _all_ consumer in the group and thus how
> > would a
> > > user know
> > > >> > > which member should be removed for a dynamic group (if an
> > > individual
> > > >> > > member should be removed)?
> > > >> > >
> > > >> > > Hence, how can any user get to know the `memberId` of an
> > > individual
> > > >> > > client in a programtic way?
> > > >> > >
> > > >> > > Also I am wondering in general, why the removal of single
> > dynamic
> > > member
> > > >> > > is important? In general, I would expect a short
> > > `session.timeout` for
> > > >> > > dynamic groups and thus removing a specific member from the
> > group
> > > seems
> > > >> > > not to be an important feature -- for static groups we
> > expect a
> > > long
> > > >> > > `session.timeout` and a user can also identify individual
> > clients
> > > via
> > > >> > > `groupInstandId`, hence the feature makes sense for this
> > case and
> > > is
> > > >> > > straight forward to use.
> > > >> > >
> > > >> > >
> > > >> > > About StreamsResetter:
> > > >> > >
> > > >> > > For this case we just say "remove all members" and thus the
> > > >> > > `describeConsumerGroup` approach works. However, it seems
> > to be a
> > > >> > > special case?
> > > >> > >
> > > >> > > Or, if we expected that the "remove all members" use case
> > is the
> > > norm,
> > > >> > > why can't we make a change admin-client to directly accept
> > a `
> > > group.id`?
> > > >> > > The admin-client can internal first do a
> > `DescribeGroupRequest`
> > > and
> > > >> > > afterward corresponding `LeaveGroupRequest` -- i.e.,
> > instead of
> > > building
> > > >> > > this pattern in `StreamsResetter` we build it directly into
> > > >> > `AdminClient`.
> > > >> > >
> > > >> > > Last, for static group the main use case seems to be to
> > remove an
> > > >> > > individual member from the group but this feature is not
> > covered
> > > by the
> > > >> > > KIP: I think using `--force` to remove all members makes
> > sense,
> > > but an
> > > >> > > important second feature to remove an individual static
> > member
> > > would
> > > >> > > require it's own flag to specify a single
> > `group.instance.id`.
> > > >> > >
> > > >> > >
> > > >> > > Thoughts?
> > > >> > >
> > > >> > >
> > > >> > > -Matthias
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > On 3/11/20 8:43 PM, feyman2009 wrote:
> > > >> > > > Hi, Sophie
> > > >> > > > For 1) Sorry, I found that my expression is kind of
> > > misleading,
> > > >> > > what I actually mean is: "if --force not specified, an
> > exception
> > > saying
> > > >> > > there are still active members on broker side will be
> > thrown and
> > > >> > > suggesting using StreamsResetter with --force", I just
> > updated
> > > the KIP
> > > >> > > page.
> > > >> > > >
> > > >> > > > For 2)
> > > >> > > > I may also had some misleading expression
> > previous, to
> > > clarify
> > > >> > :
> > > >> > > >
> > > >> > > > Also, it's more efficient to just send a single "clear the
> > > group"
> > > >> > > request vs sending a LeaveGroup
> > > >> > > > request for every single member. What do you think?
> > > >> > > > => the comparison is to send a single "clear the group"
> > request
> > > vs
> > > >> > > sending a "get members" + a "remove members" request since
> > the
> > > >> > > adminClient.removeMembersFromConsumerGroup support batch
> > removal.
> > > We
> > > >> > > don't need to send lots of leaveGroup requests for every
> > single
> > > member.
> > > >> > > >
> > > >> > > > I can understand your point, but I think we could
> > reuse
> > > the
> > > >> > > current
> > > >> > > > adminClient.removeMembersFromConsumerGroup interface
> > > effectively with
> > > >> > > the KIP.
> > > >> > > > What do you think?
> > > >> > > >
> > > >> > > > Thanks!
> > > >> > > >
> > > >> > > > Feyman
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > ------------------------------------------------------------------
> > > >> > > > 发件人：Sophie Blee-Goldman <so...@confluent.io>
> > > >> > > > 发送时间：2020年3月10日(星期二) 03:02
> > > >> > > > 收件人：dev <de...@kafka.apache.org>; feyman2009 <
> > > feyman2009@aliyun.com>
> > > >> > > > 主 题：Re: 回复：回复：[Vote] KIP-571: Add option to force remove
> > > >> > > members in StreamsResetter
> > > >> > > >
> > > >> > > > Hey Feyman,
> > > >> > > >
> > > >> > > > 1) Regarding point 2 in your last email, if I understand
> > > correctly you
> > > >> > > propose to change
> > > >> > > > the current behavior of the reset tool when --force is not
> > > specified,
> > > >> > > and wait for (up to)
> > > >> > > > the session timeout for all members to be removed. I'm
> > not sure
> > > we
> > > >> > > should change this,
> > > >> > > > especially now that we have a better way to handle the
> > case
> > > when the
> > > >> > > group is not empty:
> > > >> > > > we should continue to throw an exception and fail fast,
> > but can
> > > print
> > > >> > > a message suggesting
> > > >> > > > to use the new --force option to remove remaining group
> > > members. Why
> > > >> > > make users wait
> > > >> > > > for the session timeout when we've just added a new
> > feature
> > > that means
> > > >> > > they don't have to?
> > > >> > > >
> > > >> > > > 2) Regarding Matthias' question:
> > > >> > > >
> > > >> > > >> I am really wondering, if for a static group, we should
> > allow
> > > users
> > > >> > > toremove individual members? For a dynamic group this
> > feature
> > > would not
> > > >> > > > make much sense IMHO, because the `memberId` is not know
> > by the
> > > user.
> > > >> > > >
> > > >> > > > I think his point is similar to what I was trying to get
> > at
> > > earlier,
> > > >> > > with the proposal to add a new
> > > >> > > > #removeAllMembers API rather than an API to remove
> > individual
> > > members
> > > >> > > according to their
> > > >> > > > memberId. As he explained, removing based on memberId is
> > likely
> > > not
> > > >> > > that useful in general.
> > > >> > > > Also, it's not actually what we want to do here; maybe we
> > > should avoid
> > > >> > > adding a new API
> > > >> > > > that we think may be useful in other contexts (remove
> > individual
> > > >> > > member based on memberId),
> > > >> > > > and just add the API we actually need (remove all members
> > from
> > > group)
> > > >> > > in this KIP? We can
> > > >> > > > always add the "remove individual member by memberId" API
> > at a
> > > later
> > > >> > > point, if it turns out to
> > > >> > > > actually be requested for specific reasons?
> > > >> > > >
> > > >> > > > Also, it's more efficient to just send a single "clear the
> > > group"
> > > >> > > request vs sending a LeaveGroup
> > > >> > > > request for every single member. What do you think?
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > On Sat, Mar 7, 2020 at 1:41 AM feyman2009
> > > >> > > <fe...@aliyun.com.invalid> wrote:
> > > >> > > > Hi, Matthias
> > > >> > > > Thanks, I updated the KIP to mention the deprecated
> > and
> > > newly
> > > >> > > added methods.
> > > >> > > >
> > > >> > > > 1. What happens is `groupInstanceId` is used for a
> > dynamic
> > > group? What
> > > >> > > > happens if both parameters are specified? What happens if
> > > `memberId`
> > > >> > > > is specified for a static group?
> > > >> > > >
> > > >> > > > => my understanding is that the dynamic/static
> > membership is
> > > member
> > > >> > > level other than group level, and I think above questions
> > could be
> > > >> > > answered by the "Leave Group Logic Change" section in
> > KIP-345:
> > > >> > >
> > > >> > >
> > > >> >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> > > >> > > ,
> > > >> > > this KIP stays consistent with KIP-345.
> > > >> > > >
> > > >> > > > 2. About the `--force` option. Currently, StreamsResetter
> > > fails with
> > > >> > an
> > > >> > > > error if the consumer group is not empty. You state in
> > your
> > > KIP that:
> > > >> > > >
> > > >> > > > > without --force, we need to wait for session timeout.
> > > >> > > >
> > > >> > > > Is this an intended behavior change if `--force` is not
> > used
> > > or is the
> > > >> > > > KIP description incorrect?
> > > >> > > >
> > > >> > > > => This is the intended behavior. For this part, I think
> > there
> > > are
> > > >> > > two ways to go:
> > > >> > > > 1) (the implicit way) Not introducing the new "--force"
> > > option, with
> > > >> > > this KIP, StreamsResetter will by default remove active
> > > members(with
> > > >> > > long session timeout configured) on broker side
> > > >> > > > 2) (the explicit way) Introduce the new "--force" option,
> > > users need
> > > >> > > to explicitly specify --force to remove the active members.
> > If
> > > --force
> > > >> > > not specified, the StreamsResetter behaviour is as previous
> > > versions'.
> > > >> > > >
> > > >> > > > I think the two alternatives above are both feasible,
> > > personally I
> > > >> > > prefer way 2.
> > > >> > > >
> > > >> > > > 3. For my own understanding: with the `--force` option,
> > we
> > > intend to
> > > >> > get
> > > >> > > > all `memberIds` and send a "remove member" request for
> > each
> > > with
> > > >> > > > corresponding `memberId` to remove the member from the
> > group
> > > >> > > > (independent is the group is static or dynamic)?
> > > >> > > >
> > > >> > > > => Yeah, minor thing to mention is we will send the
> > "remove
> > > member"
> > > >> > > request for each member(could be dynamic member or static
> > member)
> > > to
> > > >> > > remove them from group
> > > >> > > > for dynamic members, both "group.instance.id" and
> > "member.id"
> > > will be
> > > >> > > specified
> > > >> > > > for dynamic members, only "member.id" will be specified
> > > >> > > >
> > > >> > > > 4. I am really wondering, if for a static group, we
> > should
> > > allow users
> > > >> > > to
> > > >> > > > remove individual members? For a dynamic group this
> > feature
> > > would not
> > > >> > > > make much sense IMHO, because the `memberId` is not know
> > by
> > > the user.
> > > >> > > >
> > > >> > > > => KIP-345 introduced the batch removal feature for both
> > static
> > > >> > > member and dynamic member, my understanding is that "allow
> > users
> > > to
> > > >> > > > remove individual members" could be useful for rolling
> > bounce
> > > and
> > > >> > > scale down accoding to KIP-345. KafkaAdminClient currently
> > only
> > > support
> > > >> > > static member removal and this KIP-571 enables dynamic
> > member
> > > removal
> > > >> > > for it, which is also consistent with the broker side logic.
> > > Users could
> > > >> > > get the member.id (and group.instance.id if for static
> > member) by
> > > >> > > adminClient.describeConsumerGroups.
> > > >> > > >
> > > >> > > > Furthermore, I don't have an assumption that a consumer
> > group
> > > should
> > > >> > > contain only static or dynamic members, and I think KIP-345
> > and
> > > this KIP
> > > >> > > don't need to be based on this assumption.
> > > >> > > > You could correct me if I have the wrong understanding :)
> > > >> > > >
> > > >> > > > Thanks!
> > > >> > > > Feyman
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > ------------------------------------------------------------------
> > > >> > > > 发件人：Matthias J. Sax <mj...@apache.org>
> > > >> > > > 发送时间：2020年3月6日(星期五) 02:20
> > > >> > > > 收件人：dev <de...@kafka.apache.org>
> > > >> > > > 主 题：Re: 回复：回复：[Vote] KIP-571: Add option to force remove
> > > >> > > members in StreamsResetter
> > > >> > > >
> > > >> > > > Feyman,
> > > >> > > >
> > > >> > > > thanks a lot for the KIP. Overall LGTM. I have a few more
> > > comment and
> > > >> > > > questions (sorry for the late reply):
> > > >> > > >
> > > >> > > >
> > > >> > > > The KIP mentions that some constructors will be
> > deprecated.
> > > Those
> > > >> > should
> > > >> > > > be listed explicitly. For example:
> > > >> > > >
> > > >> > > >
> > > >> > > > public class MemberToRemove {
> > > >> > > >
> > > >> > > > // deprecated methods
> > > >> > > >
> > > >> > > > @Deprecated
> > > >> > > > public MemberToRemove(String groupInstanceId);
> > > >> > > >
> > > >> > > > // new methods
> > > >> > > >
> > > >> > > > public MemberToRemove()
> > > >> > > >
> > > >> > > > public MemberToRemove withGroupInstanceId(String
> > > groupInstanceId)
> > > >> > > >
> > > >> > > > public MemberToRemove withMemberId(String memberId)
> > > >> > > > }
> > > >> > > >
> > > >> > > > What happens is `groupInstanceId` is used for a dynamic
> > group?
> > > What
> > > >> > > > happens if both parameters are specified? What happens if
> > > `memberId`
> > > >> > > > is specified for a static group?
> > > >> > > >
> > > >> > > >
> > > >> > > > About the `--force` option. Currently, StreamsResetter
> > fails
> > > with an
> > > >> > > > error if the consumer group is not empty. You state in
> > your KIP
> > > that:
> > > >> > > >
> > > >> > > >> without --force, we need to wait for session timeout.
> > > >> > > >
> > > >> > > > Is this an intended behavior change if `--force` is not
> > used or
> > > is the
> > > >> > > > KIP description incorrect?
> > > >> > > >
> > > >> > > > For my own understanding: with the `--force` option, we
> > intend
> > > to get
> > > >> > > > all `memberIds` and send a "remove member" request for
> > each with
> > > >> > > > corresponding `memberId` to remove the member from the
> > group
> > > >> > > > (independent is the group is static or dynamic)?
> > > >> > > >
> > > >> > > > I am really wondering, if for a static group, we should
> > allow
> > > users to
> > > >> > > > remove individual members? For a dynamic group this
> > feature
> > > would not
> > > >> > > > make much sense IMHO, because the `memberId` is not know
> > by the
> > > user.
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > -Matthias
> > > >> > > >
> > > >> > > >
> > > >> > > > On 3/5/20 7:15 AM, Bill Bejeck wrote:
> > > >> > > >> Thanks for the KIP. +1 (binding).
> > > >> > > >
> > > >> > > >> -Bill
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >> On Wed, Mar 4, 2020 at 12:40 AM Guozhang Wang <
> > > wangguoz@gmail.com>
> > > >> > > >> wrote:
> > > >> > > >
> > > >> > > >>> Thanks, +1 from me (binding).
> > > >> > > >>>
> > > >> > > >>> On Tue, Mar 3, 2020 at 9:39 PM feyman2009 <
> > > feyman2009@aliyun.com>
> > > >> > > >>> wrote:
> > > >> > > >>>
> > > >> > > >>>> Hi, Guozhang Thanks a lot for the advice, that make
> > sense! I
> > > >> > > >>>> have updated the KIP page with the operational steps of
> > > >> > > >>>> StreamsResetter.
> > > >> > > >>>>
> > > >> > > >>>> Thanks! Feyman
> > > >> > > >>>>
> > > >> > > >>>>
> > > ------------------------------------------------------------------
> > > >> > > >>>>
> > > >> > > >>>>
> > > >> > > > 发件人：Guozhang Wang <wa...@gmail.com>
> > > >> > > >>>> 发送时间：2020年3月3日(星期二) 14:22 收件人：dev
> > <de...@kafka.apache.org>;
> > > >> > > >>>> feyman2009 <fe...@aliyun.com> 主 题：Re: 回复：回复：[Vote]
> > > >> > > >>>> KIP-571: Add option to force remove members in
> > > StreamsResetter
> > > >> > > >>>>
> > > >> > > >>>> Hello Feyman, thanks for the proposal!
> > > >> > > >>>>
> > > >> > > >>>> I read through the doc and overall it looks good to me.
> > > >> > > >>>>
> > > >> > > >>>> One minor thing I'd still like to point out is that,
> > the
> > > >> > > >>>> "removeMembersFromConsumerGroup" only sends a
> > leave-group
> > > >> > > >>>> request to the coordinator to let it remove the member,
> > > >> > > >>>> however, if the member is still there alive and
> > running then
> > > it
> > > >> > > >>>> would soon be notified that it is no
> > > >> > > >>> longer
> > > >> > > >>>> a legal member of the group via heartbeats, and then
> > > >> > > >>>> automatically tries
> > > >> > > >>> to
> > > >> > > >>>> re-join the group. So on the operational side, it is
> > still
> > > >> > > >>>> required that the following steps:
> > > >> > > >>>>
> > > >> > > >>>> 1) first stop the consumers (of streams instances),
> > wait
> > > until
> > > >> > > >>>> the shutdown is complete. 2) then use admin client in
> > case
> > > the
> > > >> > > >>>> stopped consumers are still registered at the broker
> > side and
> > > >> > > >>>> we do not want to wait for session timeout.
> > > >> > > >>>>
> > > >> > > >>>> Even with this KIP, people should still not skip step
> > 1)
> > > above,
> > > >> > > >>>> since otherwise the consumers would re-connect and
> > re-join
> > > the
> > > >> > > >>>> group
> > > >> > > >>> immediately
> > > >> > > >>>> still.
> > > >> > > >>>>
> > > >> > > >>>> In your doc you've already mentioned "Furthermore,
> > users
> > > should
> > > >> > > >>>> make sure all the stream applications are shutdown when
> > > running
> > > >> > > >>>> StreamsResetter
> > > >> > > >>> with
> > > >> > > >>>> --force, otherwise it might trigger unexpected
> > rebalance. "
> > > >> > > >>>> What I'd want to clarify is that no matter if "--force"
> > > option
> > > >> > > >>>> is enabled, this is
> > > >> > > >>> always
> > > >> > > >>>> the case that users should shutdown the streams
> > instances
> > > >> > > >>>> first, and then use the streams resetter :)
> > > >> > > >>>>
> > > >> > > >>>> As long as that is clarified in the proposal
> > documentation,
> > > I'm
> > > >> > > >>>> +1 on
> > > >> > > >>> this
> > > >> > > >>>> KIP.
> > > >> > > >>>>
> > > >> > > >>>>
> > > >> > > >>>> Thanks again for the contribution, Guozhang
> > > >> > > >>>>
> > > >> > > >>>>
> > > >> > > >>>> On Mon, Mar 2, 2020 at 6:31 AM feyman2009
> > > >> > > >>>> <feyman2009@aliyun.com.invalid
> > > >> > > >>>>
> > > >> > > >>>> wrote: Hi, John Sorry, I have mistaken the KIP approval
> > > >> > > >>>> standard, anyway, I will
> > > >> > > >>> start
> > > >> > > >>>> the PR soon and waiting for more binding approvals.
> > > >> > > >>>>
> > > >> > > >>>> Thanks! Feyman
> > > >> > > >>>>
> > > >> > > >>>>
> > > >> > > >>>>
> > > ------------------------------------------------------------------
> > > >> > > >>>>
> > > >> > > >>>>
> > > >> > > > 发件人：John Roesler <vv...@apache.org>
> > > >> > > >>>> 发送时间：2020年3月2日(星期一) 22:00 收件人：dev
> > > >> > > > <de...@kafka.apache.org> 主
> > > >> > > >>>> 题：Re: 回复：回复：[Vote] KIP-571: Add option to force remove
> > > members
> > > >> > > >>>> in StreamsResetter
> > > >> > > >>>>
> > > >> > > >>>> Hi Feyman,
> > > >> > > >>>>
> > > >> > > >>>> Sorry, but we actually need 3 binding votes for the
> > KIP to
> > > >> > > >>>> pass. Please feel free to keep bumping the thread
> > until some
> > > >> > > >>>> more committers can take
> > > >> > > >>> a
> > > >> > > >>>> look.
> > > >> > > >>>>
> > > >> > > >>>> By the way, you can totally start a PR, but we can’t
> > merge it
> > > >> > > >>>> until the KIP passes the vote.
> > > >> > > >>>>
> > > >> > > >>>> Thanks! John
> > > >> > > >>>>
> > > >> > > >>>> On Mon, Mar 2, 2020, at 00:24, feyman2009 wrote:
> > > >> > > >>>>> Hi,all Since currently we have 1 binding and two
> > non-binding
> > > >> > > >>>>> +1, I will update the KIP-571 as adopted and initiate
> > a PR
> > > >> > > >>>>> shortly
> > > >> > > >>>>>
> > > >> > > >>>>> Thanks! Feyman
> > > >> > > >>>>>
> > > >> > > >>>>>
> > > >> > > >>>>>
> > > ------------------------------------------------------------------
> > > >> > > >>>>>
> > > >> > > >>>>>
> > > >> > > > 发件人：Sophie Blee-Goldman <so...@confluent.io>
> > > >> > > >>>>> 发送时间：2020年2月28日(星期五) 10:17 收件人：dev
> > > >> > > > <de...@kafka.apache.org> 主
> > > >> > > >>>>> 题：Re: 回复：[Vote] KIP-571: Add option to force remove
> > members
> > > >> > > >>>>> in
> > > >> > > >>>> StreamsResetter
> > > >> > > >>>>>
> > > >> > > >>>>> Thanks for the KIP, +1 (non-binding)
> > > >> > > >>>>>
> > > >> > > >>>>> On Thu, Feb 27, 2020 at 12:40 PM Boyang Chen <
> > > >> > > >>> reluctanthero104@gmail.com
> > > >> > > >>>>>
> > > >> > > >>>>> wrote:
> > > >> > > >>>>>
> > > >> > > >>>>>> Thanks Feyman, +1 (non-binding)
> > > >> > > >>>>>>
> > > >> > > >>>>>> On Thu, Feb 27, 2020 at 9:25 AM John Roesler
> > > >> > > >>>>>> <vv...@apache.org>
> > > >> > > >>>> wrote:
> > > >> > > >>>>>>
> > > >> > > >>>>>>> Thanks for the proposal!
> > > >> > > >>>>>>>
> > > >> > > >>>>>>> I'm +1 (binding) -John
> > > >> > > >>>>>>>
> > > >> > > >>>>>>> On Wed, Feb 26, 2020, at 19:41, feyman2009 wrote:
> > > >> > > >>>>>>>> Updated with the KIP link:
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>
> > > >> > > >>>>>>
> > > >> > > >>>>
> > > >> > > >>>
> > > >> >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-571%3A+Add+opti
> > > >> > > > on+to+force+remove+members+in+StreamsResetter
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>>
> > > >> > > >>>
> > > >> > > >>>
> > > >> > > >
> > > ------------------------------------------------------------------
> > > >> > > >>>>>>>> 发件人：feyman2009 <fe...@aliyun.com.INVALID> 发送时
> > > >> > > >>>>>>>> 间：2020年2月27日(星期四) 09:38 收件人：dev
> > <de...@kafka.apache.org>
> > > >> > > >>>>>>>> 主 题：[Vote] KIP-571: Add option to force remove
> > members
> > > >> > > >>>>>>>> in
> > > >> > > >>>>>> StreamsResetter
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>> Hi, all I would like to start a vote on KIP-571:
> > Add
> > > >> > > >>>>>>>> option to force
> > > >> > > >>>> remove
> > > >> > > >>>>>>>> members in StreamsResetter .
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>> Thanks! Feyman
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>
> > > >> > > >>>>>>
> > > >> > > >>>>>
> > > >> > > >>>>>
> > > >> > > >>>>
> > > >> > > >>>>
> > > >> > > >>>>
> > > >> > > >>>> -- -- Guozhang
> > > >> > > >>>>
> > > >> > > >>>>
> > > >> > > >>>>
> > > >> > > >>>
> > > >> > > >>> -- -- Guozhang
> > > >> > > >>>
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >> > --
> > > >> > -- Guozhang
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >
> > > >
> > >
> > >
> >
> >
> >
> >
>
>
--
-- Guozhang
回复：回复：回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in StreamsResetter
Posted by feyman2009 <fe...@aliyun.com.INVALID>.
Thanks , John and Guochang!
------------------------------------------------------------------
发件人：Guozhang Wang <wa...@gmail.com>
发送时间：2020年4月11日(星期六) 03:07
收件人：dev <de...@kafka.apache.org>
主 题：Re: 回复：回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in StreamsResetter
Thanks Feyman,
I've looked at the update that you incorporated from Matthias and that LGTM
too. I'm still +1 :)
Guozhang
On Fri, Apr 10, 2020 at 11:18 AM John Roesler <jo...@vvcephei.org> wrote:
> Hey Feyman,
>
> Just to remove any ambiguity, I've been casually following the discussion,
> I've just looked at the KIP document again, and I'm still +1 (binding).
>
> Thanks,
> -John
>
> On Fri, Apr 10, 2020, at 01:44, feyman2009 wrote:
> > Hi, all
> > KIP-571 has already collected 4 bind +1 (John, Guochang, Bill,
> > Matthias) and 3 non-binding +1(Boyang, Sophie), I will mark it as
> > approved and create a PR shortly.
> > Thanks!
> >
> > Feyman
> > ------------------------------------------------------------------
> > 发件人：feyman2009 <fe...@aliyun.com.INVALID>
> > 发送时间：2020年4月8日(星期三) 14:21
> > 收件人：dev <de...@kafka.apache.org>; Boyang Chen <re...@gmail.com>
> > 主 题：回复：回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove
> > members in StreamsResetter
> >
> > Hi Boyang,
> > Thanks for reminding me of that!
> > I'm not sure about the convention, I thought it would need to
> > re-collect votes if the KIP has changed~
> > Let's leave the vote thread here for 2 days, if no objection, I
> > will take it as approved and update the PR accordingly.
> >
> > Thanks!
> > Feyman
> >
> >
> >
> > ------------------------------------------------------------------
> > 发件人：Boyang Chen <re...@gmail.com>
> > 发送时间：2020年4月8日(星期三) 12:42
> > 收件人：dev <de...@kafka.apache.org>; feyman2009 <fe...@aliyun.com>
> > 主 题：Re: 回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove
> > members in StreamsResetter
> >
> > You should already get enough votes if I'm counting correctly
> > (Guozhang, John, Matthias)
> > On Tue, Apr 7, 2020 at 6:59 PM feyman2009
> > <fe...@aliyun.com.invalid> wrote:
> > Hi, Boyang&Matthias
> > I think Matthias's proposal makes sense, but we can use the admin
> > tool for this scenario as Boyang mentioned or follow up later, so I
> > prefer to keep this KIP unchanged to minimize the scope.
> > Calling for vote ~
> >
> > Thanks!
> > Feyman
> >
> > ------------------------------------------------------------------
> > 发件人：Boyang Chen <re...@gmail.com>
> > 发送时间：2020年4月8日(星期三) 02:15
> > 收件人：dev <de...@kafka.apache.org>
> > 主 题：Re: 回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove
> > members in StreamsResetter
> >
> > Hey Feyman,
> >
> > I think Matthias' suggestion is optional, and we could just use admin
> tool
> > to remove single static members as well.
> >
> > Boyang
> >
> > On Tue, Apr 7, 2020 at 11:00 AM Matthias J. Sax <mj...@apache.org>
> wrote:
> >
> > > > Would you mind to elaborate why we still need that
> > >
> > > Sure.
> > >
> > > For static memership, the session timeout it usually set quite high.
> > > This make scaling in an application tricky: if you shut down one
> > > instance, no rebalance would happen until `session.timeout.ms` hits.
> > > This is specific to Kafka Streams, because when a Kafka Stream
> > client is
> > > closed, it does _not_ send a `LeaveGroupRequest`. Hence, the
> > > corresponding partitions would not be processed for a long time and
> > > thus, fall back.
> > >
> > > Given that each instance will have a unique `instance.id` provided by
> > > the user, we could allow users to remove the instance they want to
> > > decommission from the consumer group without the need to wait for
> > > `session.timeout.ms`.
> > >
> > > Hence, it's not an application reset scenario for which one wants to
> > > remove all members, but a scaling-in scenario. For dynamic
> > membership,
> > > this issue usually does not occur because the `session.timeout.ms` is
> > > set to a fairly low value and a rebalance would happen quickly after
> > an
> > > instance is decommissioned.
> > >
> > > Does this make sense?
> > >
> > > As said before, we may or may not include this in this KIP. It's up
> > to
> > > you if you want to address it or not.
> > >
> > >
> > > -Matthias
> > >
> > >
> > >
> > > On 4/7/20 7:12 AM, feyman2009 wrote:
> > > > Hi, Matthias
> > > > Thanks a lot!
> > > > So you do not plan so support removing a _single static_
> > member via
> > > `StreamsResetter`?
> > > > =>
> > > > Would you mind to elaborate why we still need that if we
> > are
> > > able to batch remove active members with adminClient?
> > > >
> > > > Thanks!
> > > >
> > > > Feyman
> > > > ------------------------------------------------------------------
> > > > 发件人：Matthias J. Sax <mj...@apache.org>
> > > > 发送时间：2020年4月7日(星期二) 08:25
> > > > 收件人：dev <de...@kafka.apache.org>
> > > > 主 题：Re: 回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove
> > members
> > > in StreamsResetter
> > > >
> > > > Overall LGTM.
> > > >
> > > > +1 (binding)
> > > >
> > > > So you do not plan so support removing a _single static_ member via
> > > > `StreamsResetter`? We can of course still add this as a follow up
> > but it
> > > > might be nice to just add it to this KIP right away. Up to you if
> > you
> > > > want to include it or not.
> > > >
> > > >
> > > > -Matthias
> > > >
> > > >
> > > >
> > > > On 3/30/20 8:16 AM, feyman2009 wrote:
> > > >> Hi, Boyang
> > > >> Thanks a lot, that make sense, we should not expose member.id
> > in
> > > the MemberToRemove anymore, I have fixed it in the KIP.
> > > >>
> > > >>
> > > >> Feyman
> > > >> ------------------------------------------------------------------
> > > >> 发件人：Boyang Chen <re...@gmail.com>
> > > >> 发送时间：2020年3月29日(星期日) 01:45
> > > >> 收件人：dev <de...@kafka.apache.org>; feyman2009 <fe...@aliyun.com>
> > > >> 主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove
> > members in
> > > StreamsResetter
> > > >>
> > > >> Hey Feyman,
> > > >>
> > > >> thanks for the update. I assume we would rely entirely on the
> > internal
> > > changes for `removeMemberFromGroup` by sending a DescribeGroup
> > request
> > > first. With that in mind, I don't think we need to add member.id to
> > > MemberToRemove anymore, as it is only facing public where users will
> > only
> > > configure group.instance.id?
> > > >> On Fri, Mar 27, 2020 at 5:04 PM feyman2009
> > > <fe...@aliyun.com.invalid> wrote:
> > > >> Bump, can anyone kindly take a look at the updated KIP-571?
> > Thanks!
> > > >>
> > > >>
> > > >>
> > ------------------------------------------------------------------
> > > >> 发件人：feyman2009 <fe...@aliyun.com.INVALID>
> > > >> 发送时间：2020年3月23日(星期一) 08:51
> > > >> 收件人：dev <de...@kafka.apache.org>
> > > >> 主 题：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove
> > members in
> > > StreamsResetter
> > > >>
> > > >> Hi, team
> > > >> I have updated the KIP-571 according to our latest discussion
> > > results, would you mind to take a look? Thanks!
> > > >>
> > > >> Feyman
> > > >>
> > > >>
> > > >>
> > ------------------------------------------------------------------
> > > >> 发件人：Boyang Chen <re...@gmail.com>
> > > >> 发送时间：2020年3月19日(星期四) 13:41
> > > >> 收件人：dev <de...@kafka.apache.org>; feyman2009
> > <fe...@aliyun.com>
> > > >> 主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove
> > members in
> > > StreamsResetter
> > > >>
> > > >> Thanks for the insight Feyman. I personally feel adding another
> > admin
> > > client command is redundant, so picking option 1). The memberInfos
> > struct
> > > is internal and just used for result reference purposes. I think it
> > could
> > > still work even we overload with `removeAll` option, if that makes
> > sense.
> > > >>
> > > >> Boyang
> > > >> On Wed, Mar 18, 2020 at 8:51 PM feyman2009
> > > <fe...@aliyun.com.invalid> wrote:
> > > >> Hi, team
> > > >> Before going too far on the KIP update, I would like to
> > hear your
> > > opinions about how we would change the interface of AdminClient, the
> > two
> > > alternatives I could think of are:
> > > >> 1) Extend adminClient.removeMembersFromConsumerGroup to
> > support
> > > remove all
> > > >> As Guochang suggested, we could add some flag param in
> > > RemoveMembersFromConsumerGroupOptions to indicated the "remove all"
> > logic.
> > > >> 2) Add a new API like
> > > adminClient.removeAllMembersFromConsumerGroup(groupId, options)
> > > >>
> > > >> I think 1) will be more compact from the API perspective,
> > but
> > > looking at the code, I found that the if we are going to remove all,
> > then
> > > the
> > RemoveMembersFromConsumerGroupResult#memberInfos/memberResult()/all()
> > > should be changed accordingly, and they seem not that meaningful
> > under the
> > > "remove all" scenario.
> > > >>
> > > >> A minor thought about the
> > > adminClient.removeMembersFromConsumerGroup API is:
> > > >> Looking at some other deleteXX APIs, like deleteTopics,
> > > deleteRecords, the results contains only a Map<A, Future<B>>, I
> > think it's
> > > enough to describe the related results, is it make sense that we may
> > remove
> > > memberInfos in RemoveMembersFromConsumerGroupResult ? This KIP has no
> > > dependency on this if we choose alternative 2)
> > > >>
> > > >> Could you advise? Thanks!
> > > >>
> > > >> Feyman
> > > >>
> > > >>
> > > >> 送时间：2020年3月15日(星期日) 10:11
> > > >> 收件人：dev <de...@kafka.apache.org>
> > > >> 主 题：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove
> > members in
> > > StreamsResetter
> > > >>
> > > >> Hi, all
> > > >> Thanks a lot for your feedback!
> > > >> According to the discussion, it seems we don't have some
> > valid
> > > use cases for removing specific dynamic members, I think it makes
> > sense to
> > > encapsulate the "get and delete" logic in adminClient. I will update
> > the
> > > KIP shortly!
> > > >>
> > > >> Thanks!
> > > >>
> > > >> Feyman
> > > >>
> > > >>
> > > >>
> > ------------------------------------------------------------------
> > > >> 发件人：Boyang Chen <re...@gmail.com>
> > > >> 发送时间：2020年3月14日(星期六) 00:39
> > > >> 收件人：dev <de...@kafka.apache.org>
> > > >> 主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove
> > members
> > > in StreamsResetter
> > > >>
> > > >> Thanks Matthias and Guozhang for the feedback. I'm not worrying
> > too
> > > much
> > > >> about the member.id exposure as we have done so in a couple of
> > > areas. As
> > > >> for the recommended admin client change, I think it makes sense
> > in an
> > > >> encapsulation perspective. Maybe I'm still a bit hesitant as we
> > are
> > > losing
> > > >> the flexibility of closing only a subset of `dynamic members`
> > > potentially,
> > > >> but we could always get back and address it if some user feels
> > > necessary to
> > > >> have it.
> > > >>
> > > >> My short answer would be, LGTM :)
> > > >>
> > > >> Boyang
> > > >>
> > > >> On Thu, Mar 12, 2020 at 5:26 PM Guozhang Wang
> > <wa...@gmail.com>
> > > wrote:
> > > >>
> > > >> > Hi Matthias,
> > > >> >
> > > >> > About the AdminClient param API: that's a great point here. I
> > think
> > > overall
> > > >> > if users want to just "remove all members" they should not
> > need to
> > > first
> > > >> > get all the member.ids themselves, but instead internally the
> > admin
> > > client
> > > >> > can first issue a describe-group request to get all the
> > member.ids,
> > > and
> > > >> > then use them in the next issued leave-group request, all
> > > abstracted away
> > > >> > from the users. With that in mind, maybe in
> > > >> > RemoveMembersFromConsumerGroupOptions we can just introduce an
> > > overloaded
> > > >> > flag param besides "members" that indicate "remove all"?
> > > >> >
> > > >> > Guozhang
> > > >> >
> > > >> > On Thu, Mar 12, 2020 at 2:59 PM Matthias J. Sax
> > <mj...@apache.org>
> > > wrote:
> > > >> >
> > > >> > > Feyman,
> > > >> > >
> > > >> > > some more comments/questions:
> > > >> > >
> > > >> > > The description of `LeaveGroupRequest` is clear but it's
> > unclear
> > > how
> > > >> > > `MemberToRemove` should behave. Which parameter is required?
> > > Which is
> > > >> > > optional? What is the relationship between both.
> > > >> > >
> > > >> > > The `LeaveGroupRequest` description clearly states that
> > > specifying a
> > > >> > > `memberId` is optional if the `groupInstanceId` is
> > provided. If
> > > >> > > `MemberToRemove` applies the same pattern, it must be
> > explicitly
> > > defined
> > > >> > > in the KIP (and explained in the JavaDocs of
> > `MemberToRemove`)
> > > because
> > > >> > > we cannot expect that an admin-client users knows that
> > internally
> > > a
> > > >> > > `LeaveGroupRequest` is used nor what the semantics of a
> > > >> > > `LeaveGroupRequest` are.
> > > >> > >
> > > >> > >
> > > >> > > About Admin API:
> > > >> > >
> > > >> > > In general, I am also confused that we allow so specify a
> > > `memberId` at
> > > >> > > all, because the `memberId` is an internal id that is not
> > really
> > > exposed
> > > >> > > to the user. Hence, from a AdminClient point of view,
> > accepting a
> > > >> > > `memberId` as input seems questionable to me? Of course,
> > > `memberId` can
> > > >> > > be collected via `describeConsumerGroups()` but it will
> > return the
> > > >> > > `memberId` of _all_ consumer in the group and thus how
> > would a
> > > user know
> > > >> > > which member should be removed for a dynamic group (if an
> > > individual
> > > >> > > member should be removed)?
> > > >> > >
> > > >> > > Hence, how can any user get to know the `memberId` of an
> > > individual
> > > >> > > client in a programtic way?
> > > >> > >
> > > >> > > Also I am wondering in general, why the removal of single
> > dynamic
> > > member
> > > >> > > is important? In general, I would expect a short
> > > `session.timeout` for
> > > >> > > dynamic groups and thus removing a specific member from the
> > group
> > > seems
> > > >> > > not to be an important feature -- for static groups we
> > expect a
> > > long
> > > >> > > `session.timeout` and a user can also identify individual
> > clients
> > > via
> > > >> > > `groupInstandId`, hence the feature makes sense for this
> > case and
> > > is
> > > >> > > straight forward to use.
> > > >> > >
> > > >> > >
> > > >> > > About StreamsResetter:
> > > >> > >
> > > >> > > For this case we just say "remove all members" and thus the
> > > >> > > `describeConsumerGroup` approach works. However, it seems
> > to be a
> > > >> > > special case?
> > > >> > >
> > > >> > > Or, if we expected that the "remove all members" use case
> > is the
> > > norm,
> > > >> > > why can't we make a change admin-client to directly accept
> > a `
> > > group.id`?
> > > >> > > The admin-client can internal first do a
> > `DescribeGroupRequest`
> > > and
> > > >> > > afterward corresponding `LeaveGroupRequest` -- i.e.,
> > instead of
> > > building
> > > >> > > this pattern in `StreamsResetter` we build it directly into
> > > >> > `AdminClient`.
> > > >> > >
> > > >> > > Last, for static group the main use case seems to be to
> > remove an
> > > >> > > individual member from the group but this feature is not
> > covered
> > > by the
> > > >> > > KIP: I think using `--force` to remove all members makes
> > sense,
> > > but an
> > > >> > > important second feature to remove an individual static
> > member
> > > would
> > > >> > > require it's own flag to specify a single
> > `group.instance.id`.
> > > >> > >
> > > >> > >
> > > >> > > Thoughts?
> > > >> > >
> > > >> > >
> > > >> > > -Matthias
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > On 3/11/20 8:43 PM, feyman2009 wrote:
> > > >> > > > Hi, Sophie
> > > >> > > > For 1) Sorry, I found that my expression is kind of
> > > misleading,
> > > >> > > what I actually mean is: "if --force not specified, an
> > exception
> > > saying
> > > >> > > there are still active members on broker side will be
> > thrown and
> > > >> > > suggesting using StreamsResetter with --force", I just
> > updated
> > > the KIP
> > > >> > > page.
> > > >> > > >
> > > >> > > > For 2)
> > > >> > > > I may also had some misleading expression
> > previous, to
> > > clarify
> > > >> > :
> > > >> > > >
> > > >> > > > Also, it's more efficient to just send a single "clear the
> > > group"
> > > >> > > request vs sending a LeaveGroup
> > > >> > > > request for every single member. What do you think?
> > > >> > > > => the comparison is to send a single "clear the group"
> > request
> > > vs
> > > >> > > sending a "get members" + a "remove members" request since
> > the
> > > >> > > adminClient.removeMembersFromConsumerGroup support batch
> > removal.
> > > We
> > > >> > > don't need to send lots of leaveGroup requests for every
> > single
> > > member.
> > > >> > > >
> > > >> > > > I can understand your point, but I think we could
> > reuse
> > > the
> > > >> > > current
> > > >> > > > adminClient.removeMembersFromConsumerGroup interface
> > > effectively with
> > > >> > > the KIP.
> > > >> > > > What do you think?
> > > >> > > >
> > > >> > > > Thanks!
> > > >> > > >
> > > >> > > > Feyman
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > ------------------------------------------------------------------
> > > >> > > > 发件人：Sophie Blee-Goldman <so...@confluent.io>
> > > >> > > > 发送时间：2020年3月10日(星期二) 03:02
> > > >> > > > 收件人：dev <de...@kafka.apache.org>; feyman2009 <
> > > feyman2009@aliyun.com>
> > > >> > > > 主 题：Re: 回复：回复：[Vote] KIP-571: Add option to force remove
> > > >> > > members in StreamsResetter
> > > >> > > >
> > > >> > > > Hey Feyman,
> > > >> > > >
> > > >> > > > 1) Regarding point 2 in your last email, if I understand
> > > correctly you
> > > >> > > propose to change
> > > >> > > > the current behavior of the reset tool when --force is not
> > > specified,
> > > >> > > and wait for (up to)
> > > >> > > > the session timeout for all members to be removed. I'm
> > not sure
> > > we
> > > >> > > should change this,
> > > >> > > > especially now that we have a better way to handle the
> > case
> > > when the
> > > >> > > group is not empty:
> > > >> > > > we should continue to throw an exception and fail fast,
> > but can
> > > print
> > > >> > > a message suggesting
> > > >> > > > to use the new --force option to remove remaining group
> > > members. Why
> > > >> > > make users wait
> > > >> > > > for the session timeout when we've just added a new
> > feature
> > > that means
> > > >> > > they don't have to?
> > > >> > > >
> > > >> > > > 2) Regarding Matthias' question:
> > > >> > > >
> > > >> > > >> I am really wondering, if for a static group, we should
> > allow
> > > users
> > > >> > > toremove individual members? For a dynamic group this
> > feature
> > > would not
> > > >> > > > make much sense IMHO, because the `memberId` is not know
> > by the
> > > user.
> > > >> > > >
> > > >> > > > I think his point is similar to what I was trying to get
> > at
> > > earlier,
> > > >> > > with the proposal to add a new
> > > >> > > > #removeAllMembers API rather than an API to remove
> > individual
> > > members
> > > >> > > according to their
> > > >> > > > memberId. As he explained, removing based on memberId is
> > likely
> > > not
> > > >> > > that useful in general.
> > > >> > > > Also, it's not actually what we want to do here; maybe we
> > > should avoid
> > > >> > > adding a new API
> > > >> > > > that we think may be useful in other contexts (remove
> > individual
> > > >> > > member based on memberId),
> > > >> > > > and just add the API we actually need (remove all members
> > from
> > > group)
> > > >> > > in this KIP? We can
> > > >> > > > always add the "remove individual member by memberId" API
> > at a
> > > later
> > > >> > > point, if it turns out to
> > > >> > > > actually be requested for specific reasons?
> > > >> > > >
> > > >> > > > Also, it's more efficient to just send a single "clear the
> > > group"
> > > >> > > request vs sending a LeaveGroup
> > > >> > > > request for every single member. What do you think?
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > On Sat, Mar 7, 2020 at 1:41 AM feyman2009
> > > >> > > <fe...@aliyun.com.invalid> wrote:
> > > >> > > > Hi, Matthias
> > > >> > > > Thanks, I updated the KIP to mention the deprecated
> > and
> > > newly
> > > >> > > added methods.
> > > >> > > >
> > > >> > > > 1. What happens is `groupInstanceId` is used for a
> > dynamic
> > > group? What
> > > >> > > > happens if both parameters are specified? What happens if
> > > `memberId`
> > > >> > > > is specified for a static group?
> > > >> > > >
> > > >> > > > => my understanding is that the dynamic/static
> > membership is
> > > member
> > > >> > > level other than group level, and I think above questions
> > could be
> > > >> > > answered by the "Leave Group Logic Change" section in
> > KIP-345:
> > > >> > >
> > > >> > >
> > > >> >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> > > >> > > ,
> > > >> > > this KIP stays consistent with KIP-345.
> > > >> > > >
> > > >> > > > 2. About the `--force` option. Currently, StreamsResetter
> > > fails with
> > > >> > an
> > > >> > > > error if the consumer group is not empty. You state in
> > your
> > > KIP that:
> > > >> > > >
> > > >> > > > > without --force, we need to wait for session timeout.
> > > >> > > >
> > > >> > > > Is this an intended behavior change if `--force` is not
> > used
> > > or is the
> > > >> > > > KIP description incorrect?
> > > >> > > >
> > > >> > > > => This is the intended behavior. For this part, I think
> > there
> > > are
> > > >> > > two ways to go:
> > > >> > > > 1) (the implicit way) Not introducing the new "--force"
> > > option, with
> > > >> > > this KIP, StreamsResetter will by default remove active
> > > members(with
> > > >> > > long session timeout configured) on broker side
> > > >> > > > 2) (the explicit way) Introduce the new "--force" option,
> > > users need
> > > >> > > to explicitly specify --force to remove the active members.
> > If
> > > --force
> > > >> > > not specified, the StreamsResetter behaviour is as previous
> > > versions'.
> > > >> > > >
> > > >> > > > I think the two alternatives above are both feasible,
> > > personally I
> > > >> > > prefer way 2.
> > > >> > > >
> > > >> > > > 3. For my own understanding: with the `--force` option,
> > we
> > > intend to
> > > >> > get
> > > >> > > > all `memberIds` and send a "remove member" request for
> > each
> > > with
> > > >> > > > corresponding `memberId` to remove the member from the
> > group
> > > >> > > > (independent is the group is static or dynamic)?
> > > >> > > >
> > > >> > > > => Yeah, minor thing to mention is we will send the
> > "remove
> > > member"
> > > >> > > request for each member(could be dynamic member or static
> > member)
> > > to
> > > >> > > remove them from group
> > > >> > > > for dynamic members, both "group.instance.id" and
> > "member.id"
> > > will be
> > > >> > > specified
> > > >> > > > for dynamic members, only "member.id" will be specified
> > > >> > > >
> > > >> > > > 4. I am really wondering, if for a static group, we
> > should
> > > allow users
> > > >> > > to
> > > >> > > > remove individual members? For a dynamic group this
> > feature
> > > would not
> > > >> > > > make much sense IMHO, because the `memberId` is not know
> > by
> > > the user.
> > > >> > > >
> > > >> > > > => KIP-345 introduced the batch removal feature for both
> > static
> > > >> > > member and dynamic member, my understanding is that "allow
> > users
> > > to
> > > >> > > > remove individual members" could be useful for rolling
> > bounce
> > > and
> > > >> > > scale down accoding to KIP-345. KafkaAdminClient currently
> > only
> > > support
> > > >> > > static member removal and this KIP-571 enables dynamic
> > member
> > > removal
> > > >> > > for it, which is also consistent with the broker side logic.
> > > Users could
> > > >> > > get the member.id (and group.instance.id if for static
> > member) by
> > > >> > > adminClient.describeConsumerGroups.
> > > >> > > >
> > > >> > > > Furthermore, I don't have an assumption that a consumer
> > group
> > > should
> > > >> > > contain only static or dynamic members, and I think KIP-345
> > and
> > > this KIP
> > > >> > > don't need to be based on this assumption.
> > > >> > > > You could correct me if I have the wrong understanding :)
> > > >> > > >
> > > >> > > > Thanks!
> > > >> > > > Feyman
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > ------------------------------------------------------------------
> > > >> > > > 发件人：Matthias J. Sax <mj...@apache.org>
> > > >> > > > 发送时间：2020年3月6日(星期五) 02:20
> > > >> > > > 收件人：dev <de...@kafka.apache.org>
> > > >> > > > 主 题：Re: 回复：回复：[Vote] KIP-571: Add option to force remove
> > > >> > > members in StreamsResetter
> > > >> > > >
> > > >> > > > Feyman,
> > > >> > > >
> > > >> > > > thanks a lot for the KIP. Overall LGTM. I have a few more
> > > comment and
> > > >> > > > questions (sorry for the late reply):
> > > >> > > >
> > > >> > > >
> > > >> > > > The KIP mentions that some constructors will be
> > deprecated.
> > > Those
> > > >> > should
> > > >> > > > be listed explicitly. For example:
> > > >> > > >
> > > >> > > >
> > > >> > > > public class MemberToRemove {
> > > >> > > >
> > > >> > > > // deprecated methods
> > > >> > > >
> > > >> > > > @Deprecated
> > > >> > > > public MemberToRemove(String groupInstanceId);
> > > >> > > >
> > > >> > > > // new methods
> > > >> > > >
> > > >> > > > public MemberToRemove()
> > > >> > > >
> > > >> > > > public MemberToRemove withGroupInstanceId(String
> > > groupInstanceId)
> > > >> > > >
> > > >> > > > public MemberToRemove withMemberId(String memberId)
> > > >> > > > }
> > > >> > > >
> > > >> > > > What happens is `groupInstanceId` is used for a dynamic
> > group?
> > > What
> > > >> > > > happens if both parameters are specified? What happens if
> > > `memberId`
> > > >> > > > is specified for a static group?
> > > >> > > >
> > > >> > > >
> > > >> > > > About the `--force` option. Currently, StreamsResetter
> > fails
> > > with an
> > > >> > > > error if the consumer group is not empty. You state in
> > your KIP
> > > that:
> > > >> > > >
> > > >> > > >> without --force, we need to wait for session timeout.
> > > >> > > >
> > > >> > > > Is this an intended behavior change if `--force` is not
> > used or
> > > is the
> > > >> > > > KIP description incorrect?
> > > >> > > >
> > > >> > > > For my own understanding: with the `--force` option, we
> > intend
> > > to get
> > > >> > > > all `memberIds` and send a "remove member" request for
> > each with
> > > >> > > > corresponding `memberId` to remove the member from the
> > group
> > > >> > > > (independent is the group is static or dynamic)?
> > > >> > > >
> > > >> > > > I am really wondering, if for a static group, we should
> > allow
> > > users to
> > > >> > > > remove individual members? For a dynamic group this
> > feature
> > > would not
> > > >> > > > make much sense IMHO, because the `memberId` is not know
> > by the
> > > user.
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > -Matthias
> > > >> > > >
> > > >> > > >
> > > >> > > > On 3/5/20 7:15 AM, Bill Bejeck wrote:
> > > >> > > >> Thanks for the KIP. +1 (binding).
> > > >> > > >
> > > >> > > >> -Bill
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >> On Wed, Mar 4, 2020 at 12:40 AM Guozhang Wang <
> > > wangguoz@gmail.com>
> > > >> > > >> wrote:
> > > >> > > >
> > > >> > > >>> Thanks, +1 from me (binding).
> > > >> > > >>>
> > > >> > > >>> On Tue, Mar 3, 2020 at 9:39 PM feyman2009 <
> > > feyman2009@aliyun.com>
> > > >> > > >>> wrote:
> > > >> > > >>>
> > > >> > > >>>> Hi, Guozhang Thanks a lot for the advice, that make
> > sense! I
> > > >> > > >>>> have updated the KIP page with the operational steps of
> > > >> > > >>>> StreamsResetter.
> > > >> > > >>>>
> > > >> > > >>>> Thanks! Feyman
> > > >> > > >>>>
> > > >> > > >>>>
> > > ------------------------------------------------------------------
> > > >> > > >>>>
> > > >> > > >>>>
> > > >> > > > 发件人：Guozhang Wang <wa...@gmail.com>
> > > >> > > >>>> 发送时间：2020年3月3日(星期二) 14:22 收件人：dev
> > <de...@kafka.apache.org>;
> > > >> > > >>>> feyman2009 <fe...@aliyun.com> 主 题：Re: 回复：回复：[Vote]
> > > >> > > >>>> KIP-571: Add option to force remove members in
> > > StreamsResetter
> > > >> > > >>>>
> > > >> > > >>>> Hello Feyman, thanks for the proposal!
> > > >> > > >>>>
> > > >> > > >>>> I read through the doc and overall it looks good to me.
> > > >> > > >>>>
> > > >> > > >>>> One minor thing I'd still like to point out is that,
> > the
> > > >> > > >>>> "removeMembersFromConsumerGroup" only sends a
> > leave-group
> > > >> > > >>>> request to the coordinator to let it remove the member,
> > > >> > > >>>> however, if the member is still there alive and
> > running then
> > > it
> > > >> > > >>>> would soon be notified that it is no
> > > >> > > >>> longer
> > > >> > > >>>> a legal member of the group via heartbeats, and then
> > > >> > > >>>> automatically tries
> > > >> > > >>> to
> > > >> > > >>>> re-join the group. So on the operational side, it is
> > still
> > > >> > > >>>> required that the following steps:
> > > >> > > >>>>
> > > >> > > >>>> 1) first stop the consumers (of streams instances),
> > wait
> > > until
> > > >> > > >>>> the shutdown is complete. 2) then use admin client in
> > case
> > > the
> > > >> > > >>>> stopped consumers are still registered at the broker
> > side and
> > > >> > > >>>> we do not want to wait for session timeout.
> > > >> > > >>>>
> > > >> > > >>>> Even with this KIP, people should still not skip step
> > 1)
> > > above,
> > > >> > > >>>> since otherwise the consumers would re-connect and
> > re-join
> > > the
> > > >> > > >>>> group
> > > >> > > >>> immediately
> > > >> > > >>>> still.
> > > >> > > >>>>
> > > >> > > >>>> In your doc you've already mentioned "Furthermore,
> > users
> > > should
> > > >> > > >>>> make sure all the stream applications are shutdown when
> > > running
> > > >> > > >>>> StreamsResetter
> > > >> > > >>> with
> > > >> > > >>>> --force, otherwise it might trigger unexpected
> > rebalance. "
> > > >> > > >>>> What I'd want to clarify is that no matter if "--force"
> > > option
> > > >> > > >>>> is enabled, this is
> > > >> > > >>> always
> > > >> > > >>>> the case that users should shutdown the streams
> > instances
> > > >> > > >>>> first, and then use the streams resetter :)
> > > >> > > >>>>
> > > >> > > >>>> As long as that is clarified in the proposal
> > documentation,
> > > I'm
> > > >> > > >>>> +1 on
> > > >> > > >>> this
> > > >> > > >>>> KIP.
> > > >> > > >>>>
> > > >> > > >>>>
> > > >> > > >>>> Thanks again for the contribution, Guozhang
> > > >> > > >>>>
> > > >> > > >>>>
> > > >> > > >>>> On Mon, Mar 2, 2020 at 6:31 AM feyman2009
> > > >> > > >>>> <feyman2009@aliyun.com.invalid
> > > >> > > >>>>
> > > >> > > >>>> wrote: Hi, John Sorry, I have mistaken the KIP approval
> > > >> > > >>>> standard, anyway, I will
> > > >> > > >>> start
> > > >> > > >>>> the PR soon and waiting for more binding approvals.
> > > >> > > >>>>
> > > >> > > >>>> Thanks! Feyman
> > > >> > > >>>>
> > > >> > > >>>>
> > > >> > > >>>>
> > > ------------------------------------------------------------------
> > > >> > > >>>>
> > > >> > > >>>>
> > > >> > > > 发件人：John Roesler <vv...@apache.org>
> > > >> > > >>>> 发送时间：2020年3月2日(星期一) 22:00 收件人：dev
> > > >> > > > <de...@kafka.apache.org> 主
> > > >> > > >>>> 题：Re: 回复：回复：[Vote] KIP-571: Add option to force remove
> > > members
> > > >> > > >>>> in StreamsResetter
> > > >> > > >>>>
> > > >> > > >>>> Hi Feyman,
> > > >> > > >>>>
> > > >> > > >>>> Sorry, but we actually need 3 binding votes for the
> > KIP to
> > > >> > > >>>> pass. Please feel free to keep bumping the thread
> > until some
> > > >> > > >>>> more committers can take
> > > >> > > >>> a
> > > >> > > >>>> look.
> > > >> > > >>>>
> > > >> > > >>>> By the way, you can totally start a PR, but we can’t
> > merge it
> > > >> > > >>>> until the KIP passes the vote.
> > > >> > > >>>>
> > > >> > > >>>> Thanks! John
> > > >> > > >>>>
> > > >> > > >>>> On Mon, Mar 2, 2020, at 00:24, feyman2009 wrote:
> > > >> > > >>>>> Hi,all Since currently we have 1 binding and two
> > non-binding
> > > >> > > >>>>> +1, I will update the KIP-571 as adopted and initiate
> > a PR
> > > >> > > >>>>> shortly
> > > >> > > >>>>>
> > > >> > > >>>>> Thanks! Feyman
> > > >> > > >>>>>
> > > >> > > >>>>>
> > > >> > > >>>>>
> > > ------------------------------------------------------------------
> > > >> > > >>>>>
> > > >> > > >>>>>
> > > >> > > > 发件人：Sophie Blee-Goldman <so...@confluent.io>
> > > >> > > >>>>> 发送时间：2020年2月28日(星期五) 10:17 收件人：dev
> > > >> > > > <de...@kafka.apache.org> 主
> > > >> > > >>>>> 题：Re: 回复：[Vote] KIP-571: Add option to force remove
> > members
> > > >> > > >>>>> in
> > > >> > > >>>> StreamsResetter
> > > >> > > >>>>>
> > > >> > > >>>>> Thanks for the KIP, +1 (non-binding)
> > > >> > > >>>>>
> > > >> > > >>>>> On Thu, Feb 27, 2020 at 12:40 PM Boyang Chen <
> > > >> > > >>> reluctanthero104@gmail.com
> > > >> > > >>>>>
> > > >> > > >>>>> wrote:
> > > >> > > >>>>>
> > > >> > > >>>>>> Thanks Feyman, +1 (non-binding)
> > > >> > > >>>>>>
> > > >> > > >>>>>> On Thu, Feb 27, 2020 at 9:25 AM John Roesler
> > > >> > > >>>>>> <vv...@apache.org>
> > > >> > > >>>> wrote:
> > > >> > > >>>>>>
> > > >> > > >>>>>>> Thanks for the proposal!
> > > >> > > >>>>>>>
> > > >> > > >>>>>>> I'm +1 (binding) -John
> > > >> > > >>>>>>>
> > > >> > > >>>>>>> On Wed, Feb 26, 2020, at 19:41, feyman2009 wrote:
> > > >> > > >>>>>>>> Updated with the KIP link:
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>
> > > >> > > >>>>>>
> > > >> > > >>>>
> > > >> > > >>>
> > > >> >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-571%3A+Add+opti
> > > >> > > > on+to+force+remove+members+in+StreamsResetter
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>>
> > > >> > > >>>
> > > >> > > >>>
> > > >> > > >
> > > ------------------------------------------------------------------
> > > >> > > >>>>>>>> 发件人：feyman2009 <fe...@aliyun.com.INVALID> 发送时
> > > >> > > >>>>>>>> 间：2020年2月27日(星期四) 09:38 收件人：dev
> > <de...@kafka.apache.org>
> > > >> > > >>>>>>>> 主 题：[Vote] KIP-571: Add option to force remove
> > members
> > > >> > > >>>>>>>> in
> > > >> > > >>>>>> StreamsResetter
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>> Hi, all I would like to start a vote on KIP-571:
> > Add
> > > >> > > >>>>>>>> option to force
> > > >> > > >>>> remove
> > > >> > > >>>>>>>> members in StreamsResetter .
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>> Thanks! Feyman
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>
> > > >> > > >>>>>>
> > > >> > > >>>>>
> > > >> > > >>>>>
> > > >> > > >>>>
> > > >> > > >>>>
> > > >> > > >>>>
> > > >> > > >>>> -- -- Guozhang
> > > >> > > >>>>
> > > >> > > >>>>
> > > >> > > >>>>
> > > >> > > >>>
> > > >> > > >>> -- -- Guozhang
> > > >> > > >>>
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >> > --
> > > >> > -- Guozhang
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >
> > > >
> > >
> > >
> >
> >
> >
> >
>
>
--
-- Guozhang
Re: 回复：回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in StreamsResetter
Posted by Guozhang Wang <wa...@gmail.com>.
Thanks Feyman,
I've looked at the update that you incorporated from Matthias and that LGTM
too. I'm still +1 :)
Guozhang
On Fri, Apr 10, 2020 at 11:18 AM John Roesler <jo...@vvcephei.org> wrote:
> Hey Feyman,
>
> Just to remove any ambiguity, I've been casually following the discussion,
> I've just looked at the KIP document again, and I'm still +1 (binding).
>
> Thanks,
> -John
>
> On Fri, Apr 10, 2020, at 01:44, feyman2009 wrote:
> > Hi, all
> > KIP-571 has already collected 4 bind +1 (John, Guochang, Bill,
> > Matthias) and 3 non-binding +1(Boyang, Sophie), I will mark it as
> > approved and create a PR shortly.
> > Thanks!
> >
> > Feyman
> > ------------------------------------------------------------------
> > 发件人：feyman2009 <fe...@aliyun.com.INVALID>
> > 发送时间：2020年4月8日(星期三) 14:21
> > 收件人：dev <de...@kafka.apache.org>; Boyang Chen <re...@gmail.com>
> > 主 题：回复：回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove
> > members in StreamsResetter
> >
> > Hi Boyang,
> > Thanks for reminding me of that!
> > I'm not sure about the convention, I thought it would need to
> > re-collect votes if the KIP has changed~
> > Let's leave the vote thread here for 2 days, if no objection, I
> > will take it as approved and update the PR accordingly.
> >
> > Thanks!
> > Feyman
> >
> >
> >
> > ------------------------------------------------------------------
> > 发件人：Boyang Chen <re...@gmail.com>
> > 发送时间：2020年4月8日(星期三) 12:42
> > 收件人：dev <de...@kafka.apache.org>; feyman2009 <fe...@aliyun.com>
> > 主 题：Re: 回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove
> > members in StreamsResetter
> >
> > You should already get enough votes if I'm counting correctly
> > (Guozhang, John, Matthias)
> > On Tue, Apr 7, 2020 at 6:59 PM feyman2009
> > <fe...@aliyun.com.invalid> wrote:
> > Hi, Boyang&Matthias
> > I think Matthias's proposal makes sense, but we can use the admin
> > tool for this scenario as Boyang mentioned or follow up later, so I
> > prefer to keep this KIP unchanged to minimize the scope.
> > Calling for vote ~
> >
> > Thanks!
> > Feyman
> >
> > ------------------------------------------------------------------
> > 发件人：Boyang Chen <re...@gmail.com>
> > 发送时间：2020年4月8日(星期三) 02:15
> > 收件人：dev <de...@kafka.apache.org>
> > 主 题：Re: 回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove
> > members in StreamsResetter
> >
> > Hey Feyman,
> >
> > I think Matthias' suggestion is optional, and we could just use admin
> tool
> > to remove single static members as well.
> >
> > Boyang
> >
> > On Tue, Apr 7, 2020 at 11:00 AM Matthias J. Sax <mj...@apache.org>
> wrote:
> >
> > > > Would you mind to elaborate why we still need that
> > >
> > > Sure.
> > >
> > > For static memership, the session timeout it usually set quite high.
> > > This make scaling in an application tricky: if you shut down one
> > > instance, no rebalance would happen until `session.timeout.ms` hits.
> > > This is specific to Kafka Streams, because when a Kafka Stream
> > client is
> > > closed, it does _not_ send a `LeaveGroupRequest`. Hence, the
> > > corresponding partitions would not be processed for a long time and
> > > thus, fall back.
> > >
> > > Given that each instance will have a unique `instance.id` provided by
> > > the user, we could allow users to remove the instance they want to
> > > decommission from the consumer group without the need to wait for
> > > `session.timeout.ms`.
> > >
> > > Hence, it's not an application reset scenario for which one wants to
> > > remove all members, but a scaling-in scenario. For dynamic
> > membership,
> > > this issue usually does not occur because the `session.timeout.ms` is
> > > set to a fairly low value and a rebalance would happen quickly after
> > an
> > > instance is decommissioned.
> > >
> > > Does this make sense?
> > >
> > > As said before, we may or may not include this in this KIP. It's up
> > to
> > > you if you want to address it or not.
> > >
> > >
> > > -Matthias
> > >
> > >
> > >
> > > On 4/7/20 7:12 AM, feyman2009 wrote:
> > > > Hi, Matthias
> > > > Thanks a lot!
> > > > So you do not plan so support removing a _single static_
> > member via
> > > `StreamsResetter`?
> > > > =>
> > > > Would you mind to elaborate why we still need that if we
> > are
> > > able to batch remove active members with adminClient?
> > > >
> > > > Thanks!
> > > >
> > > > Feyman
> > > > ------------------------------------------------------------------
> > > > 发件人：Matthias J. Sax <mj...@apache.org>
> > > > 发送时间：2020年4月7日(星期二) 08:25
> > > > 收件人：dev <de...@kafka.apache.org>
> > > > 主 题：Re: 回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove
> > members
> > > in StreamsResetter
> > > >
> > > > Overall LGTM.
> > > >
> > > > +1 (binding)
> > > >
> > > > So you do not plan so support removing a _single static_ member via
> > > > `StreamsResetter`? We can of course still add this as a follow up
> > but it
> > > > might be nice to just add it to this KIP right away. Up to you if
> > you
> > > > want to include it or not.
> > > >
> > > >
> > > > -Matthias
> > > >
> > > >
> > > >
> > > > On 3/30/20 8:16 AM, feyman2009 wrote:
> > > >> Hi, Boyang
> > > >> Thanks a lot, that make sense, we should not expose member.id
> > in
> > > the MemberToRemove anymore, I have fixed it in the KIP.
> > > >>
> > > >>
> > > >> Feyman
> > > >> ------------------------------------------------------------------
> > > >> 发件人：Boyang Chen <re...@gmail.com>
> > > >> 发送时间：2020年3月29日(星期日) 01:45
> > > >> 收件人：dev <de...@kafka.apache.org>; feyman2009 <fe...@aliyun.com>
> > > >> 主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove
> > members in
> > > StreamsResetter
> > > >>
> > > >> Hey Feyman,
> > > >>
> > > >> thanks for the update. I assume we would rely entirely on the
> > internal
> > > changes for `removeMemberFromGroup` by sending a DescribeGroup
> > request
> > > first. With that in mind, I don't think we need to add member.id to
> > > MemberToRemove anymore, as it is only facing public where users will
> > only
> > > configure group.instance.id?
> > > >> On Fri, Mar 27, 2020 at 5:04 PM feyman2009
> > > <fe...@aliyun.com.invalid> wrote:
> > > >> Bump, can anyone kindly take a look at the updated KIP-571?
> > Thanks!
> > > >>
> > > >>
> > > >>
> > ------------------------------------------------------------------
> > > >> 发件人：feyman2009 <fe...@aliyun.com.INVALID>
> > > >> 发送时间：2020年3月23日(星期一) 08:51
> > > >> 收件人：dev <de...@kafka.apache.org>
> > > >> 主 题：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove
> > members in
> > > StreamsResetter
> > > >>
> > > >> Hi, team
> > > >> I have updated the KIP-571 according to our latest discussion
> > > results, would you mind to take a look? Thanks!
> > > >>
> > > >> Feyman
> > > >>
> > > >>
> > > >>
> > ------------------------------------------------------------------
> > > >> 发件人：Boyang Chen <re...@gmail.com>
> > > >> 发送时间：2020年3月19日(星期四) 13:41
> > > >> 收件人：dev <de...@kafka.apache.org>; feyman2009
> > <fe...@aliyun.com>
> > > >> 主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove
> > members in
> > > StreamsResetter
> > > >>
> > > >> Thanks for the insight Feyman. I personally feel adding another
> > admin
> > > client command is redundant, so picking option 1). The memberInfos
> > struct
> > > is internal and just used for result reference purposes. I think it
> > could
> > > still work even we overload with `removeAll` option, if that makes
> > sense.
> > > >>
> > > >> Boyang
> > > >> On Wed, Mar 18, 2020 at 8:51 PM feyman2009
> > > <fe...@aliyun.com.invalid> wrote:
> > > >> Hi, team
> > > >> Before going too far on the KIP update, I would like to
> > hear your
> > > opinions about how we would change the interface of AdminClient, the
> > two
> > > alternatives I could think of are:
> > > >> 1) Extend adminClient.removeMembersFromConsumerGroup to
> > support
> > > remove all
> > > >> As Guochang suggested, we could add some flag param in
> > > RemoveMembersFromConsumerGroupOptions to indicated the "remove all"
> > logic.
> > > >> 2) Add a new API like
> > > adminClient.removeAllMembersFromConsumerGroup(groupId, options)
> > > >>
> > > >> I think 1) will be more compact from the API perspective,
> > but
> > > looking at the code, I found that the if we are going to remove all,
> > then
> > > the
> > RemoveMembersFromConsumerGroupResult#memberInfos/memberResult()/all()
> > > should be changed accordingly, and they seem not that meaningful
> > under the
> > > "remove all" scenario.
> > > >>
> > > >> A minor thought about the
> > > adminClient.removeMembersFromConsumerGroup API is:
> > > >> Looking at some other deleteXX APIs, like deleteTopics,
> > > deleteRecords, the results contains only a Map<A, Future<B>>, I
> > think it's
> > > enough to describe the related results, is it make sense that we may
> > remove
> > > memberInfos in RemoveMembersFromConsumerGroupResult ? This KIP has no
> > > dependency on this if we choose alternative 2)
> > > >>
> > > >> Could you advise? Thanks!
> > > >>
> > > >> Feyman
> > > >>
> > > >>
> > > >> 送时间：2020年3月15日(星期日) 10:11
> > > >> 收件人：dev <de...@kafka.apache.org>
> > > >> 主 题：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove
> > members in
> > > StreamsResetter
> > > >>
> > > >> Hi, all
> > > >> Thanks a lot for your feedback!
> > > >> According to the discussion, it seems we don't have some
> > valid
> > > use cases for removing specific dynamic members, I think it makes
> > sense to
> > > encapsulate the "get and delete" logic in adminClient. I will update
> > the
> > > KIP shortly!
> > > >>
> > > >> Thanks!
> > > >>
> > > >> Feyman
> > > >>
> > > >>
> > > >>
> > ------------------------------------------------------------------
> > > >> 发件人：Boyang Chen <re...@gmail.com>
> > > >> 发送时间：2020年3月14日(星期六) 00:39
> > > >> 收件人：dev <de...@kafka.apache.org>
> > > >> 主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove
> > members
> > > in StreamsResetter
> > > >>
> > > >> Thanks Matthias and Guozhang for the feedback. I'm not worrying
> > too
> > > much
> > > >> about the member.id exposure as we have done so in a couple of
> > > areas. As
> > > >> for the recommended admin client change, I think it makes sense
> > in an
> > > >> encapsulation perspective. Maybe I'm still a bit hesitant as we
> > are
> > > losing
> > > >> the flexibility of closing only a subset of `dynamic members`
> > > potentially,
> > > >> but we could always get back and address it if some user feels
> > > necessary to
> > > >> have it.
> > > >>
> > > >> My short answer would be, LGTM :)
> > > >>
> > > >> Boyang
> > > >>
> > > >> On Thu, Mar 12, 2020 at 5:26 PM Guozhang Wang
> > <wa...@gmail.com>
> > > wrote:
> > > >>
> > > >> > Hi Matthias,
> > > >> >
> > > >> > About the AdminClient param API: that's a great point here. I
> > think
> > > overall
> > > >> > if users want to just "remove all members" they should not
> > need to
> > > first
> > > >> > get all the member.ids themselves, but instead internally the
> > admin
> > > client
> > > >> > can first issue a describe-group request to get all the
> > member.ids,
> > > and
> > > >> > then use them in the next issued leave-group request, all
> > > abstracted away
> > > >> > from the users. With that in mind, maybe in
> > > >> > RemoveMembersFromConsumerGroupOptions we can just introduce an
> > > overloaded
> > > >> > flag param besides "members" that indicate "remove all"?
> > > >> >
> > > >> > Guozhang
> > > >> >
> > > >> > On Thu, Mar 12, 2020 at 2:59 PM Matthias J. Sax
> > <mj...@apache.org>
> > > wrote:
> > > >> >
> > > >> > > Feyman,
> > > >> > >
> > > >> > > some more comments/questions:
> > > >> > >
> > > >> > > The description of `LeaveGroupRequest` is clear but it's
> > unclear
> > > how
> > > >> > > `MemberToRemove` should behave. Which parameter is required?
> > > Which is
> > > >> > > optional? What is the relationship between both.
> > > >> > >
> > > >> > > The `LeaveGroupRequest` description clearly states that
> > > specifying a
> > > >> > > `memberId` is optional if the `groupInstanceId` is
> > provided. If
> > > >> > > `MemberToRemove` applies the same pattern, it must be
> > explicitly
> > > defined
> > > >> > > in the KIP (and explained in the JavaDocs of
> > `MemberToRemove`)
> > > because
> > > >> > > we cannot expect that an admin-client users knows that
> > internally
> > > a
> > > >> > > `LeaveGroupRequest` is used nor what the semantics of a
> > > >> > > `LeaveGroupRequest` are.
> > > >> > >
> > > >> > >
> > > >> > > About Admin API:
> > > >> > >
> > > >> > > In general, I am also confused that we allow so specify a
> > > `memberId` at
> > > >> > > all, because the `memberId` is an internal id that is not
> > really
> > > exposed
> > > >> > > to the user. Hence, from a AdminClient point of view,
> > accepting a
> > > >> > > `memberId` as input seems questionable to me? Of course,
> > > `memberId` can
> > > >> > > be collected via `describeConsumerGroups()` but it will
> > return the
> > > >> > > `memberId` of _all_ consumer in the group and thus how
> > would a
> > > user know
> > > >> > > which member should be removed for a dynamic group (if an
> > > individual
> > > >> > > member should be removed)?
> > > >> > >
> > > >> > > Hence, how can any user get to know the `memberId` of an
> > > individual
> > > >> > > client in a programtic way?
> > > >> > >
> > > >> > > Also I am wondering in general, why the removal of single
> > dynamic
> > > member
> > > >> > > is important? In general, I would expect a short
> > > `session.timeout` for
> > > >> > > dynamic groups and thus removing a specific member from the
> > group
> > > seems
> > > >> > > not to be an important feature -- for static groups we
> > expect a
> > > long
> > > >> > > `session.timeout` and a user can also identify individual
> > clients
> > > via
> > > >> > > `groupInstandId`, hence the feature makes sense for this
> > case and
> > > is
> > > >> > > straight forward to use.
> > > >> > >
> > > >> > >
> > > >> > > About StreamsResetter:
> > > >> > >
> > > >> > > For this case we just say "remove all members" and thus the
> > > >> > > `describeConsumerGroup` approach works. However, it seems
> > to be a
> > > >> > > special case?
> > > >> > >
> > > >> > > Or, if we expected that the "remove all members" use case
> > is the
> > > norm,
> > > >> > > why can't we make a change admin-client to directly accept
> > a `
> > > group.id`?
> > > >> > > The admin-client can internal first do a
> > `DescribeGroupRequest`
> > > and
> > > >> > > afterward corresponding `LeaveGroupRequest` -- i.e.,
> > instead of
> > > building
> > > >> > > this pattern in `StreamsResetter` we build it directly into
> > > >> > `AdminClient`.
> > > >> > >
> > > >> > > Last, for static group the main use case seems to be to
> > remove an
> > > >> > > individual member from the group but this feature is not
> > covered
> > > by the
> > > >> > > KIP: I think using `--force` to remove all members makes
> > sense,
> > > but an
> > > >> > > important second feature to remove an individual static
> > member
> > > would
> > > >> > > require it's own flag to specify a single
> > `group.instance.id`.
> > > >> > >
> > > >> > >
> > > >> > > Thoughts?
> > > >> > >
> > > >> > >
> > > >> > > -Matthias
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > On 3/11/20 8:43 PM, feyman2009 wrote:
> > > >> > > > Hi, Sophie
> > > >> > > > For 1) Sorry, I found that my expression is kind of
> > > misleading,
> > > >> > > what I actually mean is: "if --force not specified, an
> > exception
> > > saying
> > > >> > > there are still active members on broker side will be
> > thrown and
> > > >> > > suggesting using StreamsResetter with --force", I just
> > updated
> > > the KIP
> > > >> > > page.
> > > >> > > >
> > > >> > > > For 2)
> > > >> > > > I may also had some misleading expression
> > previous, to
> > > clarify
> > > >> > :
> > > >> > > >
> > > >> > > > Also, it's more efficient to just send a single "clear the
> > > group"
> > > >> > > request vs sending a LeaveGroup
> > > >> > > > request for every single member. What do you think?
> > > >> > > > => the comparison is to send a single "clear the group"
> > request
> > > vs
> > > >> > > sending a "get members" + a "remove members" request since
> > the
> > > >> > > adminClient.removeMembersFromConsumerGroup support batch
> > removal.
> > > We
> > > >> > > don't need to send lots of leaveGroup requests for every
> > single
> > > member.
> > > >> > > >
> > > >> > > > I can understand your point, but I think we could
> > reuse
> > > the
> > > >> > > current
> > > >> > > > adminClient.removeMembersFromConsumerGroup interface
> > > effectively with
> > > >> > > the KIP.
> > > >> > > > What do you think?
> > > >> > > >
> > > >> > > > Thanks!
> > > >> > > >
> > > >> > > > Feyman
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > ------------------------------------------------------------------
> > > >> > > > 发件人：Sophie Blee-Goldman <so...@confluent.io>
> > > >> > > > 发送时间：2020年3月10日(星期二) 03:02
> > > >> > > > 收件人：dev <de...@kafka.apache.org>; feyman2009 <
> > > feyman2009@aliyun.com>
> > > >> > > > 主 题：Re: 回复：回复：[Vote] KIP-571: Add option to force remove
> > > >> > > members in StreamsResetter
> > > >> > > >
> > > >> > > > Hey Feyman,
> > > >> > > >
> > > >> > > > 1) Regarding point 2 in your last email, if I understand
> > > correctly you
> > > >> > > propose to change
> > > >> > > > the current behavior of the reset tool when --force is not
> > > specified,
> > > >> > > and wait for (up to)
> > > >> > > > the session timeout for all members to be removed. I'm
> > not sure
> > > we
> > > >> > > should change this,
> > > >> > > > especially now that we have a better way to handle the
> > case
> > > when the
> > > >> > > group is not empty:
> > > >> > > > we should continue to throw an exception and fail fast,
> > but can
> > > print
> > > >> > > a message suggesting
> > > >> > > > to use the new --force option to remove remaining group
> > > members. Why
> > > >> > > make users wait
> > > >> > > > for the session timeout when we've just added a new
> > feature
> > > that means
> > > >> > > they don't have to?
> > > >> > > >
> > > >> > > > 2) Regarding Matthias' question:
> > > >> > > >
> > > >> > > >> I am really wondering, if for a static group, we should
> > allow
> > > users
> > > >> > > toremove individual members? For a dynamic group this
> > feature
> > > would not
> > > >> > > > make much sense IMHO, because the `memberId` is not know
> > by the
> > > user.
> > > >> > > >
> > > >> > > > I think his point is similar to what I was trying to get
> > at
> > > earlier,
> > > >> > > with the proposal to add a new
> > > >> > > > #removeAllMembers API rather than an API to remove
> > individual
> > > members
> > > >> > > according to their
> > > >> > > > memberId. As he explained, removing based on memberId is
> > likely
> > > not
> > > >> > > that useful in general.
> > > >> > > > Also, it's not actually what we want to do here; maybe we
> > > should avoid
> > > >> > > adding a new API
> > > >> > > > that we think may be useful in other contexts (remove
> > individual
> > > >> > > member based on memberId),
> > > >> > > > and just add the API we actually need (remove all members
> > from
> > > group)
> > > >> > > in this KIP? We can
> > > >> > > > always add the "remove individual member by memberId" API
> > at a
> > > later
> > > >> > > point, if it turns out to
> > > >> > > > actually be requested for specific reasons?
> > > >> > > >
> > > >> > > > Also, it's more efficient to just send a single "clear the
> > > group"
> > > >> > > request vs sending a LeaveGroup
> > > >> > > > request for every single member. What do you think?
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > On Sat, Mar 7, 2020 at 1:41 AM feyman2009
> > > >> > > <fe...@aliyun.com.invalid> wrote:
> > > >> > > > Hi, Matthias
> > > >> > > > Thanks, I updated the KIP to mention the deprecated
> > and
> > > newly
> > > >> > > added methods.
> > > >> > > >
> > > >> > > > 1. What happens is `groupInstanceId` is used for a
> > dynamic
> > > group? What
> > > >> > > > happens if both parameters are specified? What happens if
> > > `memberId`
> > > >> > > > is specified for a static group?
> > > >> > > >
> > > >> > > > => my understanding is that the dynamic/static
> > membership is
> > > member
> > > >> > > level other than group level, and I think above questions
> > could be
> > > >> > > answered by the "Leave Group Logic Change" section in
> > KIP-345:
> > > >> > >
> > > >> > >
> > > >> >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> > > >> > > ,
> > > >> > > this KIP stays consistent with KIP-345.
> > > >> > > >
> > > >> > > > 2. About the `--force` option. Currently, StreamsResetter
> > > fails with
> > > >> > an
> > > >> > > > error if the consumer group is not empty. You state in
> > your
> > > KIP that:
> > > >> > > >
> > > >> > > > > without --force, we need to wait for session timeout.
> > > >> > > >
> > > >> > > > Is this an intended behavior change if `--force` is not
> > used
> > > or is the
> > > >> > > > KIP description incorrect?
> > > >> > > >
> > > >> > > > => This is the intended behavior. For this part, I think
> > there
> > > are
> > > >> > > two ways to go:
> > > >> > > > 1) (the implicit way) Not introducing the new "--force"
> > > option, with
> > > >> > > this KIP, StreamsResetter will by default remove active
> > > members(with
> > > >> > > long session timeout configured) on broker side
> > > >> > > > 2) (the explicit way) Introduce the new "--force" option,
> > > users need
> > > >> > > to explicitly specify --force to remove the active members.
> > If
> > > --force
> > > >> > > not specified, the StreamsResetter behaviour is as previous
> > > versions'.
> > > >> > > >
> > > >> > > > I think the two alternatives above are both feasible,
> > > personally I
> > > >> > > prefer way 2.
> > > >> > > >
> > > >> > > > 3. For my own understanding: with the `--force` option,
> > we
> > > intend to
> > > >> > get
> > > >> > > > all `memberIds` and send a "remove member" request for
> > each
> > > with
> > > >> > > > corresponding `memberId` to remove the member from the
> > group
> > > >> > > > (independent is the group is static or dynamic)?
> > > >> > > >
> > > >> > > > => Yeah, minor thing to mention is we will send the
> > "remove
> > > member"
> > > >> > > request for each member(could be dynamic member or static
> > member)
> > > to
> > > >> > > remove them from group
> > > >> > > > for dynamic members, both "group.instance.id" and
> > "member.id"
> > > will be
> > > >> > > specified
> > > >> > > > for dynamic members, only "member.id" will be specified
> > > >> > > >
> > > >> > > > 4. I am really wondering, if for a static group, we
> > should
> > > allow users
> > > >> > > to
> > > >> > > > remove individual members? For a dynamic group this
> > feature
> > > would not
> > > >> > > > make much sense IMHO, because the `memberId` is not know
> > by
> > > the user.
> > > >> > > >
> > > >> > > > => KIP-345 introduced the batch removal feature for both
> > static
> > > >> > > member and dynamic member, my understanding is that "allow
> > users
> > > to
> > > >> > > > remove individual members" could be useful for rolling
> > bounce
> > > and
> > > >> > > scale down accoding to KIP-345. KafkaAdminClient currently
> > only
> > > support
> > > >> > > static member removal and this KIP-571 enables dynamic
> > member
> > > removal
> > > >> > > for it, which is also consistent with the broker side logic.
> > > Users could
> > > >> > > get the member.id (and group.instance.id if for static
> > member) by
> > > >> > > adminClient.describeConsumerGroups.
> > > >> > > >
> > > >> > > > Furthermore, I don't have an assumption that a consumer
> > group
> > > should
> > > >> > > contain only static or dynamic members, and I think KIP-345
> > and
> > > this KIP
> > > >> > > don't need to be based on this assumption.
> > > >> > > > You could correct me if I have the wrong understanding :)
> > > >> > > >
> > > >> > > > Thanks!
> > > >> > > > Feyman
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > ------------------------------------------------------------------
> > > >> > > > 发件人：Matthias J. Sax <mj...@apache.org>
> > > >> > > > 发送时间：2020年3月6日(星期五) 02:20
> > > >> > > > 收件人：dev <de...@kafka.apache.org>
> > > >> > > > 主 题：Re: 回复：回复：[Vote] KIP-571: Add option to force remove
> > > >> > > members in StreamsResetter
> > > >> > > >
> > > >> > > > Feyman,
> > > >> > > >
> > > >> > > > thanks a lot for the KIP. Overall LGTM. I have a few more
> > > comment and
> > > >> > > > questions (sorry for the late reply):
> > > >> > > >
> > > >> > > >
> > > >> > > > The KIP mentions that some constructors will be
> > deprecated.
> > > Those
> > > >> > should
> > > >> > > > be listed explicitly. For example:
> > > >> > > >
> > > >> > > >
> > > >> > > > public class MemberToRemove {
> > > >> > > >
> > > >> > > > // deprecated methods
> > > >> > > >
> > > >> > > > @Deprecated
> > > >> > > > public MemberToRemove(String groupInstanceId);
> > > >> > > >
> > > >> > > > // new methods
> > > >> > > >
> > > >> > > > public MemberToRemove()
> > > >> > > >
> > > >> > > > public MemberToRemove withGroupInstanceId(String
> > > groupInstanceId)
> > > >> > > >
> > > >> > > > public MemberToRemove withMemberId(String memberId)
> > > >> > > > }
> > > >> > > >
> > > >> > > > What happens is `groupInstanceId` is used for a dynamic
> > group?
> > > What
> > > >> > > > happens if both parameters are specified? What happens if
> > > `memberId`
> > > >> > > > is specified for a static group?
> > > >> > > >
> > > >> > > >
> > > >> > > > About the `--force` option. Currently, StreamsResetter
> > fails
> > > with an
> > > >> > > > error if the consumer group is not empty. You state in
> > your KIP
> > > that:
> > > >> > > >
> > > >> > > >> without --force, we need to wait for session timeout.
> > > >> > > >
> > > >> > > > Is this an intended behavior change if `--force` is not
> > used or
> > > is the
> > > >> > > > KIP description incorrect?
> > > >> > > >
> > > >> > > > For my own understanding: with the `--force` option, we
> > intend
> > > to get
> > > >> > > > all `memberIds` and send a "remove member" request for
> > each with
> > > >> > > > corresponding `memberId` to remove the member from the
> > group
> > > >> > > > (independent is the group is static or dynamic)?
> > > >> > > >
> > > >> > > > I am really wondering, if for a static group, we should
> > allow
> > > users to
> > > >> > > > remove individual members? For a dynamic group this
> > feature
> > > would not
> > > >> > > > make much sense IMHO, because the `memberId` is not know
> > by the
> > > user.
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > -Matthias
> > > >> > > >
> > > >> > > >
> > > >> > > > On 3/5/20 7:15 AM, Bill Bejeck wrote:
> > > >> > > >> Thanks for the KIP. +1 (binding).
> > > >> > > >
> > > >> > > >> -Bill
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >> On Wed, Mar 4, 2020 at 12:40 AM Guozhang Wang <
> > > wangguoz@gmail.com>
> > > >> > > >> wrote:
> > > >> > > >
> > > >> > > >>> Thanks, +1 from me (binding).
> > > >> > > >>>
> > > >> > > >>> On Tue, Mar 3, 2020 at 9:39 PM feyman2009 <
> > > feyman2009@aliyun.com>
> > > >> > > >>> wrote:
> > > >> > > >>>
> > > >> > > >>>> Hi, Guozhang Thanks a lot for the advice, that make
> > sense! I
> > > >> > > >>>> have updated the KIP page with the operational steps of
> > > >> > > >>>> StreamsResetter.
> > > >> > > >>>>
> > > >> > > >>>> Thanks! Feyman
> > > >> > > >>>>
> > > >> > > >>>>
> > > ------------------------------------------------------------------
> > > >> > > >>>>
> > > >> > > >>>>
> > > >> > > > 发件人：Guozhang Wang <wa...@gmail.com>
> > > >> > > >>>> 发送时间：2020年3月3日(星期二) 14:22 收件人：dev
> > <de...@kafka.apache.org>;
> > > >> > > >>>> feyman2009 <fe...@aliyun.com> 主 题：Re: 回复：回复：[Vote]
> > > >> > > >>>> KIP-571: Add option to force remove members in
> > > StreamsResetter
> > > >> > > >>>>
> > > >> > > >>>> Hello Feyman, thanks for the proposal!
> > > >> > > >>>>
> > > >> > > >>>> I read through the doc and overall it looks good to me.
> > > >> > > >>>>
> > > >> > > >>>> One minor thing I'd still like to point out is that,
> > the
> > > >> > > >>>> "removeMembersFromConsumerGroup" only sends a
> > leave-group
> > > >> > > >>>> request to the coordinator to let it remove the member,
> > > >> > > >>>> however, if the member is still there alive and
> > running then
> > > it
> > > >> > > >>>> would soon be notified that it is no
> > > >> > > >>> longer
> > > >> > > >>>> a legal member of the group via heartbeats, and then
> > > >> > > >>>> automatically tries
> > > >> > > >>> to
> > > >> > > >>>> re-join the group. So on the operational side, it is
> > still
> > > >> > > >>>> required that the following steps:
> > > >> > > >>>>
> > > >> > > >>>> 1) first stop the consumers (of streams instances),
> > wait
> > > until
> > > >> > > >>>> the shutdown is complete. 2) then use admin client in
> > case
> > > the
> > > >> > > >>>> stopped consumers are still registered at the broker
> > side and
> > > >> > > >>>> we do not want to wait for session timeout.
> > > >> > > >>>>
> > > >> > > >>>> Even with this KIP, people should still not skip step
> > 1)
> > > above,
> > > >> > > >>>> since otherwise the consumers would re-connect and
> > re-join
> > > the
> > > >> > > >>>> group
> > > >> > > >>> immediately
> > > >> > > >>>> still.
> > > >> > > >>>>
> > > >> > > >>>> In your doc you've already mentioned "Furthermore,
> > users
> > > should
> > > >> > > >>>> make sure all the stream applications are shutdown when
> > > running
> > > >> > > >>>> StreamsResetter
> > > >> > > >>> with
> > > >> > > >>>> --force, otherwise it might trigger unexpected
> > rebalance. "
> > > >> > > >>>> What I'd want to clarify is that no matter if "--force"
> > > option
> > > >> > > >>>> is enabled, this is
> > > >> > > >>> always
> > > >> > > >>>> the case that users should shutdown the streams
> > instances
> > > >> > > >>>> first, and then use the streams resetter :)
> > > >> > > >>>>
> > > >> > > >>>> As long as that is clarified in the proposal
> > documentation,
> > > I'm
> > > >> > > >>>> +1 on
> > > >> > > >>> this
> > > >> > > >>>> KIP.
> > > >> > > >>>>
> > > >> > > >>>>
> > > >> > > >>>> Thanks again for the contribution, Guozhang
> > > >> > > >>>>
> > > >> > > >>>>
> > > >> > > >>>> On Mon, Mar 2, 2020 at 6:31 AM feyman2009
> > > >> > > >>>> <feyman2009@aliyun.com.invalid
> > > >> > > >>>>
> > > >> > > >>>> wrote: Hi, John Sorry, I have mistaken the KIP approval
> > > >> > > >>>> standard, anyway, I will
> > > >> > > >>> start
> > > >> > > >>>> the PR soon and waiting for more binding approvals.
> > > >> > > >>>>
> > > >> > > >>>> Thanks! Feyman
> > > >> > > >>>>
> > > >> > > >>>>
> > > >> > > >>>>
> > > ------------------------------------------------------------------
> > > >> > > >>>>
> > > >> > > >>>>
> > > >> > > > 发件人：John Roesler <vv...@apache.org>
> > > >> > > >>>> 发送时间：2020年3月2日(星期一) 22:00 收件人：dev
> > > >> > > > <de...@kafka.apache.org> 主
> > > >> > > >>>> 题：Re: 回复：回复：[Vote] KIP-571: Add option to force remove
> > > members
> > > >> > > >>>> in StreamsResetter
> > > >> > > >>>>
> > > >> > > >>>> Hi Feyman,
> > > >> > > >>>>
> > > >> > > >>>> Sorry, but we actually need 3 binding votes for the
> > KIP to
> > > >> > > >>>> pass. Please feel free to keep bumping the thread
> > until some
> > > >> > > >>>> more committers can take
> > > >> > > >>> a
> > > >> > > >>>> look.
> > > >> > > >>>>
> > > >> > > >>>> By the way, you can totally start a PR, but we can’t
> > merge it
> > > >> > > >>>> until the KIP passes the vote.
> > > >> > > >>>>
> > > >> > > >>>> Thanks! John
> > > >> > > >>>>
> > > >> > > >>>> On Mon, Mar 2, 2020, at 00:24, feyman2009 wrote:
> > > >> > > >>>>> Hi,all Since currently we have 1 binding and two
> > non-binding
> > > >> > > >>>>> +1, I will update the KIP-571 as adopted and initiate
> > a PR
> > > >> > > >>>>> shortly
> > > >> > > >>>>>
> > > >> > > >>>>> Thanks! Feyman
> > > >> > > >>>>>
> > > >> > > >>>>>
> > > >> > > >>>>>
> > > ------------------------------------------------------------------
> > > >> > > >>>>>
> > > >> > > >>>>>
> > > >> > > > 发件人：Sophie Blee-Goldman <so...@confluent.io>
> > > >> > > >>>>> 发送时间：2020年2月28日(星期五) 10:17 收件人：dev
> > > >> > > > <de...@kafka.apache.org> 主
> > > >> > > >>>>> 题：Re: 回复：[Vote] KIP-571: Add option to force remove
> > members
> > > >> > > >>>>> in
> > > >> > > >>>> StreamsResetter
> > > >> > > >>>>>
> > > >> > > >>>>> Thanks for the KIP, +1 (non-binding)
> > > >> > > >>>>>
> > > >> > > >>>>> On Thu, Feb 27, 2020 at 12:40 PM Boyang Chen <
> > > >> > > >>> reluctanthero104@gmail.com
> > > >> > > >>>>>
> > > >> > > >>>>> wrote:
> > > >> > > >>>>>
> > > >> > > >>>>>> Thanks Feyman, +1 (non-binding)
> > > >> > > >>>>>>
> > > >> > > >>>>>> On Thu, Feb 27, 2020 at 9:25 AM John Roesler
> > > >> > > >>>>>> <vv...@apache.org>
> > > >> > > >>>> wrote:
> > > >> > > >>>>>>
> > > >> > > >>>>>>> Thanks for the proposal!
> > > >> > > >>>>>>>
> > > >> > > >>>>>>> I'm +1 (binding) -John
> > > >> > > >>>>>>>
> > > >> > > >>>>>>> On Wed, Feb 26, 2020, at 19:41, feyman2009 wrote:
> > > >> > > >>>>>>>> Updated with the KIP link:
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>
> > > >> > > >>>>>>
> > > >> > > >>>>
> > > >> > > >>>
> > > >> >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-571%3A+Add+opti
> > > >> > > > on+to+force+remove+members+in+StreamsResetter
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>>
> > > >> > > >>>
> > > >> > > >>>
> > > >> > > >
> > > ------------------------------------------------------------------
> > > >> > > >>>>>>>> 发件人：feyman2009 <fe...@aliyun.com.INVALID> 发送时
> > > >> > > >>>>>>>> 间：2020年2月27日(星期四) 09:38 收件人：dev
> > <de...@kafka.apache.org>
> > > >> > > >>>>>>>> 主 题：[Vote] KIP-571: Add option to force remove
> > members
> > > >> > > >>>>>>>> in
> > > >> > > >>>>>> StreamsResetter
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>> Hi, all I would like to start a vote on KIP-571:
> > Add
> > > >> > > >>>>>>>> option to force
> > > >> > > >>>> remove
> > > >> > > >>>>>>>> members in StreamsResetter .
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>> Thanks! Feyman
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>
> > > >> > > >>>>>>
> > > >> > > >>>>>
> > > >> > > >>>>>
> > > >> > > >>>>
> > > >> > > >>>>
> > > >> > > >>>>
> > > >> > > >>>> -- -- Guozhang
> > > >> > > >>>>
> > > >> > > >>>>
> > > >> > > >>>>
> > > >> > > >>>
> > > >> > > >>> -- -- Guozhang
> > > >> > > >>>
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >> > --
> > > >> > -- Guozhang
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >
> > > >
> > >
> > >
> >
> >
> >
> >
>
>
--
-- Guozhang
Re: 回复：回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in StreamsResetter
Posted by John Roesler <jo...@vvcephei.org>.
Hey Feyman,
Just to remove any ambiguity, I've been casually following the discussion, I've just looked at the KIP document again, and I'm still +1 (binding).
Thanks,
-John
On Fri, Apr 10, 2020, at 01:44, feyman2009 wrote:
> Hi, all
> KIP-571 has already collected 4 bind +1 (John, Guochang, Bill,
> Matthias) and 3 non-binding +1(Boyang, Sophie), I will mark it as
> approved and create a PR shortly.
> Thanks!
>
> Feyman
> ------------------------------------------------------------------
> 发件人：feyman2009 <fe...@aliyun.com.INVALID>
> 发送时间：2020年4月8日(星期三) 14:21
> 收件人：dev <de...@kafka.apache.org>; Boyang Chen <re...@gmail.com>
> 主 题：回复：回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove
> members in StreamsResetter
>
> Hi Boyang,
> Thanks for reminding me of that!
> I'm not sure about the convention, I thought it would need to
> re-collect votes if the KIP has changed~
> Let's leave the vote thread here for 2 days, if no objection, I
> will take it as approved and update the PR accordingly.
>
> Thanks!
> Feyman
>
>
>
> ------------------------------------------------------------------
> 发件人：Boyang Chen <re...@gmail.com>
> 发送时间：2020年4月8日(星期三) 12:42
> 收件人：dev <de...@kafka.apache.org>; feyman2009 <fe...@aliyun.com>
> 主 题：Re: 回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove
> members in StreamsResetter
>
> You should already get enough votes if I'm counting correctly
> (Guozhang, John, Matthias)
> On Tue, Apr 7, 2020 at 6:59 PM feyman2009
> <fe...@aliyun.com.invalid> wrote:
> Hi, Boyang&Matthias
> I think Matthias's proposal makes sense, but we can use the admin
> tool for this scenario as Boyang mentioned or follow up later, so I
> prefer to keep this KIP unchanged to minimize the scope.
> Calling for vote ~
>
> Thanks!
> Feyman
>
> ------------------------------------------------------------------
> 发件人：Boyang Chen <re...@gmail.com>
> 发送时间：2020年4月8日(星期三) 02:15
> 收件人：dev <de...@kafka.apache.org>
> 主 题：Re: 回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove
> members in StreamsResetter
>
> Hey Feyman,
>
> I think Matthias' suggestion is optional, and we could just use admin tool
> to remove single static members as well.
>
> Boyang
>
> On Tue, Apr 7, 2020 at 11:00 AM Matthias J. Sax <mj...@apache.org> wrote:
>
> > > Would you mind to elaborate why we still need that
> >
> > Sure.
> >
> > For static memership, the session timeout it usually set quite high.
> > This make scaling in an application tricky: if you shut down one
> > instance, no rebalance would happen until `session.timeout.ms` hits.
> > This is specific to Kafka Streams, because when a Kafka Stream
> client is
> > closed, it does _not_ send a `LeaveGroupRequest`. Hence, the
> > corresponding partitions would not be processed for a long time and
> > thus, fall back.
> >
> > Given that each instance will have a unique `instance.id` provided by
> > the user, we could allow users to remove the instance they want to
> > decommission from the consumer group without the need to wait for
> > `session.timeout.ms`.
> >
> > Hence, it's not an application reset scenario for which one wants to
> > remove all members, but a scaling-in scenario. For dynamic
> membership,
> > this issue usually does not occur because the `session.timeout.ms` is
> > set to a fairly low value and a rebalance would happen quickly after
> an
> > instance is decommissioned.
> >
> > Does this make sense?
> >
> > As said before, we may or may not include this in this KIP. It's up
> to
> > you if you want to address it or not.
> >
> >
> > -Matthias
> >
> >
> >
> > On 4/7/20 7:12 AM, feyman2009 wrote:
> > > Hi, Matthias
> > > Thanks a lot!
> > > So you do not plan so support removing a _single static_
> member via
> > `StreamsResetter`?
> > > =>
> > > Would you mind to elaborate why we still need that if we
> are
> > able to batch remove active members with adminClient?
> > >
> > > Thanks!
> > >
> > > Feyman
> > > ------------------------------------------------------------------
> > > 发件人：Matthias J. Sax <mj...@apache.org>
> > > 发送时间：2020年4月7日(星期二) 08:25
> > > 收件人：dev <de...@kafka.apache.org>
> > > 主 题：Re: 回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove
> members
> > in StreamsResetter
> > >
> > > Overall LGTM.
> > >
> > > +1 (binding)
> > >
> > > So you do not plan so support removing a _single static_ member via
> > > `StreamsResetter`? We can of course still add this as a follow up
> but it
> > > might be nice to just add it to this KIP right away. Up to you if
> you
> > > want to include it or not.
> > >
> > >
> > > -Matthias
> > >
> > >
> > >
> > > On 3/30/20 8:16 AM, feyman2009 wrote:
> > >> Hi, Boyang
> > >> Thanks a lot, that make sense, we should not expose member.id
> in
> > the MemberToRemove anymore, I have fixed it in the KIP.
> > >>
> > >>
> > >> Feyman
> > >> ------------------------------------------------------------------
> > >> 发件人：Boyang Chen <re...@gmail.com>
> > >> 发送时间：2020年3月29日(星期日) 01:45
> > >> 收件人：dev <de...@kafka.apache.org>; feyman2009 <fe...@aliyun.com>
> > >> 主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove
> members in
> > StreamsResetter
> > >>
> > >> Hey Feyman,
> > >>
> > >> thanks for the update. I assume we would rely entirely on the
> internal
> > changes for `removeMemberFromGroup` by sending a DescribeGroup
> request
> > first. With that in mind, I don't think we need to add member.id to
> > MemberToRemove anymore, as it is only facing public where users will
> only
> > configure group.instance.id?
> > >> On Fri, Mar 27, 2020 at 5:04 PM feyman2009
> > <fe...@aliyun.com.invalid> wrote:
> > >> Bump, can anyone kindly take a look at the updated KIP-571?
> Thanks!
> > >>
> > >>
> > >>
> ------------------------------------------------------------------
> > >> 发件人：feyman2009 <fe...@aliyun.com.INVALID>
> > >> 发送时间：2020年3月23日(星期一) 08:51
> > >> 收件人：dev <de...@kafka.apache.org>
> > >> 主 题：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove
> members in
> > StreamsResetter
> > >>
> > >> Hi, team
> > >> I have updated the KIP-571 according to our latest discussion
> > results, would you mind to take a look? Thanks!
> > >>
> > >> Feyman
> > >>
> > >>
> > >>
> ------------------------------------------------------------------
> > >> 发件人：Boyang Chen <re...@gmail.com>
> > >> 发送时间：2020年3月19日(星期四) 13:41
> > >> 收件人：dev <de...@kafka.apache.org>; feyman2009
> <fe...@aliyun.com>
> > >> 主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove
> members in
> > StreamsResetter
> > >>
> > >> Thanks for the insight Feyman. I personally feel adding another
> admin
> > client command is redundant, so picking option 1). The memberInfos
> struct
> > is internal and just used for result reference purposes. I think it
> could
> > still work even we overload with `removeAll` option, if that makes
> sense.
> > >>
> > >> Boyang
> > >> On Wed, Mar 18, 2020 at 8:51 PM feyman2009
> > <fe...@aliyun.com.invalid> wrote:
> > >> Hi, team
> > >> Before going too far on the KIP update, I would like to
> hear your
> > opinions about how we would change the interface of AdminClient, the
> two
> > alternatives I could think of are:
> > >> 1) Extend adminClient.removeMembersFromConsumerGroup to
> support
> > remove all
> > >> As Guochang suggested, we could add some flag param in
> > RemoveMembersFromConsumerGroupOptions to indicated the "remove all"
> logic.
> > >> 2) Add a new API like
> > adminClient.removeAllMembersFromConsumerGroup(groupId, options)
> > >>
> > >> I think 1) will be more compact from the API perspective,
> but
> > looking at the code, I found that the if we are going to remove all,
> then
> > the
> RemoveMembersFromConsumerGroupResult#memberInfos/memberResult()/all()
> > should be changed accordingly, and they seem not that meaningful
> under the
> > "remove all" scenario.
> > >>
> > >> A minor thought about the
> > adminClient.removeMembersFromConsumerGroup API is:
> > >> Looking at some other deleteXX APIs, like deleteTopics,
> > deleteRecords, the results contains only a Map<A, Future<B>>, I
> think it's
> > enough to describe the related results, is it make sense that we may
> remove
> > memberInfos in RemoveMembersFromConsumerGroupResult ? This KIP has no
> > dependency on this if we choose alternative 2)
> > >>
> > >> Could you advise? Thanks!
> > >>
> > >> Feyman
> > >>
> > >>
> > >> 送时间：2020年3月15日(星期日) 10:11
> > >> 收件人：dev <de...@kafka.apache.org>
> > >> 主 题：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove
> members in
> > StreamsResetter
> > >>
> > >> Hi, all
> > >> Thanks a lot for your feedback!
> > >> According to the discussion, it seems we don't have some
> valid
> > use cases for removing specific dynamic members, I think it makes
> sense to
> > encapsulate the "get and delete" logic in adminClient. I will update
> the
> > KIP shortly!
> > >>
> > >> Thanks!
> > >>
> > >> Feyman
> > >>
> > >>
> > >>
> ------------------------------------------------------------------
> > >> 发件人：Boyang Chen <re...@gmail.com>
> > >> 发送时间：2020年3月14日(星期六) 00:39
> > >> 收件人：dev <de...@kafka.apache.org>
> > >> 主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove
> members
> > in StreamsResetter
> > >>
> > >> Thanks Matthias and Guozhang for the feedback. I'm not worrying
> too
> > much
> > >> about the member.id exposure as we have done so in a couple of
> > areas. As
> > >> for the recommended admin client change, I think it makes sense
> in an
> > >> encapsulation perspective. Maybe I'm still a bit hesitant as we
> are
> > losing
> > >> the flexibility of closing only a subset of `dynamic members`
> > potentially,
> > >> but we could always get back and address it if some user feels
> > necessary to
> > >> have it.
> > >>
> > >> My short answer would be, LGTM :)
> > >>
> > >> Boyang
> > >>
> > >> On Thu, Mar 12, 2020 at 5:26 PM Guozhang Wang
> <wa...@gmail.com>
> > wrote:
> > >>
> > >> > Hi Matthias,
> > >> >
> > >> > About the AdminClient param API: that's a great point here. I
> think
> > overall
> > >> > if users want to just "remove all members" they should not
> need to
> > first
> > >> > get all the member.ids themselves, but instead internally the
> admin
> > client
> > >> > can first issue a describe-group request to get all the
> member.ids,
> > and
> > >> > then use them in the next issued leave-group request, all
> > abstracted away
> > >> > from the users. With that in mind, maybe in
> > >> > RemoveMembersFromConsumerGroupOptions we can just introduce an
> > overloaded
> > >> > flag param besides "members" that indicate "remove all"?
> > >> >
> > >> > Guozhang
> > >> >
> > >> > On Thu, Mar 12, 2020 at 2:59 PM Matthias J. Sax
> <mj...@apache.org>
> > wrote:
> > >> >
> > >> > > Feyman,
> > >> > >
> > >> > > some more comments/questions:
> > >> > >
> > >> > > The description of `LeaveGroupRequest` is clear but it's
> unclear
> > how
> > >> > > `MemberToRemove` should behave. Which parameter is required?
> > Which is
> > >> > > optional? What is the relationship between both.
> > >> > >
> > >> > > The `LeaveGroupRequest` description clearly states that
> > specifying a
> > >> > > `memberId` is optional if the `groupInstanceId` is
> provided. If
> > >> > > `MemberToRemove` applies the same pattern, it must be
> explicitly
> > defined
> > >> > > in the KIP (and explained in the JavaDocs of
> `MemberToRemove`)
> > because
> > >> > > we cannot expect that an admin-client users knows that
> internally
> > a
> > >> > > `LeaveGroupRequest` is used nor what the semantics of a
> > >> > > `LeaveGroupRequest` are.
> > >> > >
> > >> > >
> > >> > > About Admin API:
> > >> > >
> > >> > > In general, I am also confused that we allow so specify a
> > `memberId` at
> > >> > > all, because the `memberId` is an internal id that is not
> really
> > exposed
> > >> > > to the user. Hence, from a AdminClient point of view,
> accepting a
> > >> > > `memberId` as input seems questionable to me? Of course,
> > `memberId` can
> > >> > > be collected via `describeConsumerGroups()` but it will
> return the
> > >> > > `memberId` of _all_ consumer in the group and thus how
> would a
> > user know
> > >> > > which member should be removed for a dynamic group (if an
> > individual
> > >> > > member should be removed)?
> > >> > >
> > >> > > Hence, how can any user get to know the `memberId` of an
> > individual
> > >> > > client in a programtic way?
> > >> > >
> > >> > > Also I am wondering in general, why the removal of single
> dynamic
> > member
> > >> > > is important? In general, I would expect a short
> > `session.timeout` for
> > >> > > dynamic groups and thus removing a specific member from the
> group
> > seems
> > >> > > not to be an important feature -- for static groups we
> expect a
> > long
> > >> > > `session.timeout` and a user can also identify individual
> clients
> > via
> > >> > > `groupInstandId`, hence the feature makes sense for this
> case and
> > is
> > >> > > straight forward to use.
> > >> > >
> > >> > >
> > >> > > About StreamsResetter:
> > >> > >
> > >> > > For this case we just say "remove all members" and thus the
> > >> > > `describeConsumerGroup` approach works. However, it seems
> to be a
> > >> > > special case?
> > >> > >
> > >> > > Or, if we expected that the "remove all members" use case
> is the
> > norm,
> > >> > > why can't we make a change admin-client to directly accept
> a `
> > group.id`?
> > >> > > The admin-client can internal first do a
> `DescribeGroupRequest`
> > and
> > >> > > afterward corresponding `LeaveGroupRequest` -- i.e.,
> instead of
> > building
> > >> > > this pattern in `StreamsResetter` we build it directly into
> > >> > `AdminClient`.
> > >> > >
> > >> > > Last, for static group the main use case seems to be to
> remove an
> > >> > > individual member from the group but this feature is not
> covered
> > by the
> > >> > > KIP: I think using `--force` to remove all members makes
> sense,
> > but an
> > >> > > important second feature to remove an individual static
> member
> > would
> > >> > > require it's own flag to specify a single
> `group.instance.id`.
> > >> > >
> > >> > >
> > >> > > Thoughts?
> > >> > >
> > >> > >
> > >> > > -Matthias
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > On 3/11/20 8:43 PM, feyman2009 wrote:
> > >> > > > Hi, Sophie
> > >> > > > For 1) Sorry, I found that my expression is kind of
> > misleading,
> > >> > > what I actually mean is: "if --force not specified, an
> exception
> > saying
> > >> > > there are still active members on broker side will be
> thrown and
> > >> > > suggesting using StreamsResetter with --force", I just
> updated
> > the KIP
> > >> > > page.
> > >> > > >
> > >> > > > For 2)
> > >> > > > I may also had some misleading expression
> previous, to
> > clarify
> > >> > :
> > >> > > >
> > >> > > > Also, it's more efficient to just send a single "clear the
> > group"
> > >> > > request vs sending a LeaveGroup
> > >> > > > request for every single member. What do you think?
> > >> > > > => the comparison is to send a single "clear the group"
> request
> > vs
> > >> > > sending a "get members" + a "remove members" request since
> the
> > >> > > adminClient.removeMembersFromConsumerGroup support batch
> removal.
> > We
> > >> > > don't need to send lots of leaveGroup requests for every
> single
> > member.
> > >> > > >
> > >> > > > I can understand your point, but I think we could
> reuse
> > the
> > >> > > current
> > >> > > > adminClient.removeMembersFromConsumerGroup interface
> > effectively with
> > >> > > the KIP.
> > >> > > > What do you think?
> > >> > > >
> > >> > > > Thanks!
> > >> > > >
> > >> > > > Feyman
> > >> > > >
> > >> > > >
> > >> > > >
> > ------------------------------------------------------------------
> > >> > > > 发件人：Sophie Blee-Goldman <so...@confluent.io>
> > >> > > > 发送时间：2020年3月10日(星期二) 03:02
> > >> > > > 收件人：dev <de...@kafka.apache.org>; feyman2009 <
> > feyman2009@aliyun.com>
> > >> > > > 主 题：Re: 回复：回复：[Vote] KIP-571: Add option to force remove
> > >> > > members in StreamsResetter
> > >> > > >
> > >> > > > Hey Feyman,
> > >> > > >
> > >> > > > 1) Regarding point 2 in your last email, if I understand
> > correctly you
> > >> > > propose to change
> > >> > > > the current behavior of the reset tool when --force is not
> > specified,
> > >> > > and wait for (up to)
> > >> > > > the session timeout for all members to be removed. I'm
> not sure
> > we
> > >> > > should change this,
> > >> > > > especially now that we have a better way to handle the
> case
> > when the
> > >> > > group is not empty:
> > >> > > > we should continue to throw an exception and fail fast,
> but can
> > print
> > >> > > a message suggesting
> > >> > > > to use the new --force option to remove remaining group
> > members. Why
> > >> > > make users wait
> > >> > > > for the session timeout when we've just added a new
> feature
> > that means
> > >> > > they don't have to?
> > >> > > >
> > >> > > > 2) Regarding Matthias' question:
> > >> > > >
> > >> > > >> I am really wondering, if for a static group, we should
> allow
> > users
> > >> > > toremove individual members? For a dynamic group this
> feature
> > would not
> > >> > > > make much sense IMHO, because the `memberId` is not know
> by the
> > user.
> > >> > > >
> > >> > > > I think his point is similar to what I was trying to get
> at
> > earlier,
> > >> > > with the proposal to add a new
> > >> > > > #removeAllMembers API rather than an API to remove
> individual
> > members
> > >> > > according to their
> > >> > > > memberId. As he explained, removing based on memberId is
> likely
> > not
> > >> > > that useful in general.
> > >> > > > Also, it's not actually what we want to do here; maybe we
> > should avoid
> > >> > > adding a new API
> > >> > > > that we think may be useful in other contexts (remove
> individual
> > >> > > member based on memberId),
> > >> > > > and just add the API we actually need (remove all members
> from
> > group)
> > >> > > in this KIP? We can
> > >> > > > always add the "remove individual member by memberId" API
> at a
> > later
> > >> > > point, if it turns out to
> > >> > > > actually be requested for specific reasons?
> > >> > > >
> > >> > > > Also, it's more efficient to just send a single "clear the
> > group"
> > >> > > request vs sending a LeaveGroup
> > >> > > > request for every single member. What do you think?
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On Sat, Mar 7, 2020 at 1:41 AM feyman2009
> > >> > > <fe...@aliyun.com.invalid> wrote:
> > >> > > > Hi, Matthias
> > >> > > > Thanks, I updated the KIP to mention the deprecated
> and
> > newly
> > >> > > added methods.
> > >> > > >
> > >> > > > 1. What happens is `groupInstanceId` is used for a
> dynamic
> > group? What
> > >> > > > happens if both parameters are specified? What happens if
> > `memberId`
> > >> > > > is specified for a static group?
> > >> > > >
> > >> > > > => my understanding is that the dynamic/static
> membership is
> > member
> > >> > > level other than group level, and I think above questions
> could be
> > >> > > answered by the "Leave Group Logic Change" section in
> KIP-345:
> > >> > >
> > >> > >
> > >> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> > >> > > ,
> > >> > > this KIP stays consistent with KIP-345.
> > >> > > >
> > >> > > > 2. About the `--force` option. Currently, StreamsResetter
> > fails with
> > >> > an
> > >> > > > error if the consumer group is not empty. You state in
> your
> > KIP that:
> > >> > > >
> > >> > > > > without --force, we need to wait for session timeout.
> > >> > > >
> > >> > > > Is this an intended behavior change if `--force` is not
> used
> > or is the
> > >> > > > KIP description incorrect?
> > >> > > >
> > >> > > > => This is the intended behavior. For this part, I think
> there
> > are
> > >> > > two ways to go:
> > >> > > > 1) (the implicit way) Not introducing the new "--force"
> > option, with
> > >> > > this KIP, StreamsResetter will by default remove active
> > members(with
> > >> > > long session timeout configured) on broker side
> > >> > > > 2) (the explicit way) Introduce the new "--force" option,
> > users need
> > >> > > to explicitly specify --force to remove the active members.
> If
> > --force
> > >> > > not specified, the StreamsResetter behaviour is as previous
> > versions'.
> > >> > > >
> > >> > > > I think the two alternatives above are both feasible,
> > personally I
> > >> > > prefer way 2.
> > >> > > >
> > >> > > > 3. For my own understanding: with the `--force` option,
> we
> > intend to
> > >> > get
> > >> > > > all `memberIds` and send a "remove member" request for
> each
> > with
> > >> > > > corresponding `memberId` to remove the member from the
> group
> > >> > > > (independent is the group is static or dynamic)?
> > >> > > >
> > >> > > > => Yeah, minor thing to mention is we will send the
> "remove
> > member"
> > >> > > request for each member(could be dynamic member or static
> member)
> > to
> > >> > > remove them from group
> > >> > > > for dynamic members, both "group.instance.id" and
> "member.id"
> > will be
> > >> > > specified
> > >> > > > for dynamic members, only "member.id" will be specified
> > >> > > >
> > >> > > > 4. I am really wondering, if for a static group, we
> should
> > allow users
> > >> > > to
> > >> > > > remove individual members? For a dynamic group this
> feature
> > would not
> > >> > > > make much sense IMHO, because the `memberId` is not know
> by
> > the user.
> > >> > > >
> > >> > > > => KIP-345 introduced the batch removal feature for both
> static
> > >> > > member and dynamic member, my understanding is that "allow
> users
> > to
> > >> > > > remove individual members" could be useful for rolling
> bounce
> > and
> > >> > > scale down accoding to KIP-345. KafkaAdminClient currently
> only
> > support
> > >> > > static member removal and this KIP-571 enables dynamic
> member
> > removal
> > >> > > for it, which is also consistent with the broker side logic.
> > Users could
> > >> > > get the member.id (and group.instance.id if for static
> member) by
> > >> > > adminClient.describeConsumerGroups.
> > >> > > >
> > >> > > > Furthermore, I don't have an assumption that a consumer
> group
> > should
> > >> > > contain only static or dynamic members, and I think KIP-345
> and
> > this KIP
> > >> > > don't need to be based on this assumption.
> > >> > > > You could correct me if I have the wrong understanding :)
> > >> > > >
> > >> > > > Thanks!
> > >> > > > Feyman
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > ------------------------------------------------------------------
> > >> > > > 发件人：Matthias J. Sax <mj...@apache.org>
> > >> > > > 发送时间：2020年3月6日(星期五) 02:20
> > >> > > > 收件人：dev <de...@kafka.apache.org>
> > >> > > > 主 题：Re: 回复：回复：[Vote] KIP-571: Add option to force remove
> > >> > > members in StreamsResetter
> > >> > > >
> > >> > > > Feyman,
> > >> > > >
> > >> > > > thanks a lot for the KIP. Overall LGTM. I have a few more
> > comment and
> > >> > > > questions (sorry for the late reply):
> > >> > > >
> > >> > > >
> > >> > > > The KIP mentions that some constructors will be
> deprecated.
> > Those
> > >> > should
> > >> > > > be listed explicitly. For example:
> > >> > > >
> > >> > > >
> > >> > > > public class MemberToRemove {
> > >> > > >
> > >> > > > // deprecated methods
> > >> > > >
> > >> > > > @Deprecated
> > >> > > > public MemberToRemove(String groupInstanceId);
> > >> > > >
> > >> > > > // new methods
> > >> > > >
> > >> > > > public MemberToRemove()
> > >> > > >
> > >> > > > public MemberToRemove withGroupInstanceId(String
> > groupInstanceId)
> > >> > > >
> > >> > > > public MemberToRemove withMemberId(String memberId)
> > >> > > > }
> > >> > > >
> > >> > > > What happens is `groupInstanceId` is used for a dynamic
> group?
> > What
> > >> > > > happens if both parameters are specified? What happens if
> > `memberId`
> > >> > > > is specified for a static group?
> > >> > > >
> > >> > > >
> > >> > > > About the `--force` option. Currently, StreamsResetter
> fails
> > with an
> > >> > > > error if the consumer group is not empty. You state in
> your KIP
> > that:
> > >> > > >
> > >> > > >> without --force, we need to wait for session timeout.
> > >> > > >
> > >> > > > Is this an intended behavior change if `--force` is not
> used or
> > is the
> > >> > > > KIP description incorrect?
> > >> > > >
> > >> > > > For my own understanding: with the `--force` option, we
> intend
> > to get
> > >> > > > all `memberIds` and send a "remove member" request for
> each with
> > >> > > > corresponding `memberId` to remove the member from the
> group
> > >> > > > (independent is the group is static or dynamic)?
> > >> > > >
> > >> > > > I am really wondering, if for a static group, we should
> allow
> > users to
> > >> > > > remove individual members? For a dynamic group this
> feature
> > would not
> > >> > > > make much sense IMHO, because the `memberId` is not know
> by the
> > user.
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > -Matthias
> > >> > > >
> > >> > > >
> > >> > > > On 3/5/20 7:15 AM, Bill Bejeck wrote:
> > >> > > >> Thanks for the KIP. +1 (binding).
> > >> > > >
> > >> > > >> -Bill
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >> On Wed, Mar 4, 2020 at 12:40 AM Guozhang Wang <
> > wangguoz@gmail.com>
> > >> > > >> wrote:
> > >> > > >
> > >> > > >>> Thanks, +1 from me (binding).
> > >> > > >>>
> > >> > > >>> On Tue, Mar 3, 2020 at 9:39 PM feyman2009 <
> > feyman2009@aliyun.com>
> > >> > > >>> wrote:
> > >> > > >>>
> > >> > > >>>> Hi, Guozhang Thanks a lot for the advice, that make
> sense! I
> > >> > > >>>> have updated the KIP page with the operational steps of
> > >> > > >>>> StreamsResetter.
> > >> > > >>>>
> > >> > > >>>> Thanks! Feyman
> > >> > > >>>>
> > >> > > >>>>
> > ------------------------------------------------------------------
> > >> > > >>>>
> > >> > > >>>>
> > >> > > > 发件人：Guozhang Wang <wa...@gmail.com>
> > >> > > >>>> 发送时间：2020年3月3日(星期二) 14:22 收件人：dev
> <de...@kafka.apache.org>;
> > >> > > >>>> feyman2009 <fe...@aliyun.com> 主 题：Re: 回复：回复：[Vote]
> > >> > > >>>> KIP-571: Add option to force remove members in
> > StreamsResetter
> > >> > > >>>>
> > >> > > >>>> Hello Feyman, thanks for the proposal!
> > >> > > >>>>
> > >> > > >>>> I read through the doc and overall it looks good to me.
> > >> > > >>>>
> > >> > > >>>> One minor thing I'd still like to point out is that,
> the
> > >> > > >>>> "removeMembersFromConsumerGroup" only sends a
> leave-group
> > >> > > >>>> request to the coordinator to let it remove the member,
> > >> > > >>>> however, if the member is still there alive and
> running then
> > it
> > >> > > >>>> would soon be notified that it is no
> > >> > > >>> longer
> > >> > > >>>> a legal member of the group via heartbeats, and then
> > >> > > >>>> automatically tries
> > >> > > >>> to
> > >> > > >>>> re-join the group. So on the operational side, it is
> still
> > >> > > >>>> required that the following steps:
> > >> > > >>>>
> > >> > > >>>> 1) first stop the consumers (of streams instances),
> wait
> > until
> > >> > > >>>> the shutdown is complete. 2) then use admin client in
> case
> > the
> > >> > > >>>> stopped consumers are still registered at the broker
> side and
> > >> > > >>>> we do not want to wait for session timeout.
> > >> > > >>>>
> > >> > > >>>> Even with this KIP, people should still not skip step
> 1)
> > above,
> > >> > > >>>> since otherwise the consumers would re-connect and
> re-join
> > the
> > >> > > >>>> group
> > >> > > >>> immediately
> > >> > > >>>> still.
> > >> > > >>>>
> > >> > > >>>> In your doc you've already mentioned "Furthermore,
> users
> > should
> > >> > > >>>> make sure all the stream applications are shutdown when
> > running
> > >> > > >>>> StreamsResetter
> > >> > > >>> with
> > >> > > >>>> --force, otherwise it might trigger unexpected
> rebalance. "
> > >> > > >>>> What I'd want to clarify is that no matter if "--force"
> > option
> > >> > > >>>> is enabled, this is
> > >> > > >>> always
> > >> > > >>>> the case that users should shutdown the streams
> instances
> > >> > > >>>> first, and then use the streams resetter :)
> > >> > > >>>>
> > >> > > >>>> As long as that is clarified in the proposal
> documentation,
> > I'm
> > >> > > >>>> +1 on
> > >> > > >>> this
> > >> > > >>>> KIP.
> > >> > > >>>>
> > >> > > >>>>
> > >> > > >>>> Thanks again for the contribution, Guozhang
> > >> > > >>>>
> > >> > > >>>>
> > >> > > >>>> On Mon, Mar 2, 2020 at 6:31 AM feyman2009
> > >> > > >>>> <feyman2009@aliyun.com.invalid
> > >> > > >>>>
> > >> > > >>>> wrote: Hi, John Sorry, I have mistaken the KIP approval
> > >> > > >>>> standard, anyway, I will
> > >> > > >>> start
> > >> > > >>>> the PR soon and waiting for more binding approvals.
> > >> > > >>>>
> > >> > > >>>> Thanks! Feyman
> > >> > > >>>>
> > >> > > >>>>
> > >> > > >>>>
> > ------------------------------------------------------------------
> > >> > > >>>>
> > >> > > >>>>
> > >> > > > 发件人：John Roesler <vv...@apache.org>
> > >> > > >>>> 发送时间：2020年3月2日(星期一) 22:00 收件人：dev
> > >> > > > <de...@kafka.apache.org> 主
> > >> > > >>>> 题：Re: 回复：回复：[Vote] KIP-571: Add option to force remove
> > members
> > >> > > >>>> in StreamsResetter
> > >> > > >>>>
> > >> > > >>>> Hi Feyman,
> > >> > > >>>>
> > >> > > >>>> Sorry, but we actually need 3 binding votes for the
> KIP to
> > >> > > >>>> pass. Please feel free to keep bumping the thread
> until some
> > >> > > >>>> more committers can take
> > >> > > >>> a
> > >> > > >>>> look.
> > >> > > >>>>
> > >> > > >>>> By the way, you can totally start a PR, but we can’t
> merge it
> > >> > > >>>> until the KIP passes the vote.
> > >> > > >>>>
> > >> > > >>>> Thanks! John
> > >> > > >>>>
> > >> > > >>>> On Mon, Mar 2, 2020, at 00:24, feyman2009 wrote:
> > >> > > >>>>> Hi,all Since currently we have 1 binding and two
> non-binding
> > >> > > >>>>> +1, I will update the KIP-571 as adopted and initiate
> a PR
> > >> > > >>>>> shortly
> > >> > > >>>>>
> > >> > > >>>>> Thanks! Feyman
> > >> > > >>>>>
> > >> > > >>>>>
> > >> > > >>>>>
> > ------------------------------------------------------------------
> > >> > > >>>>>
> > >> > > >>>>>
> > >> > > > 发件人：Sophie Blee-Goldman <so...@confluent.io>
> > >> > > >>>>> 发送时间：2020年2月28日(星期五) 10:17 收件人：dev
> > >> > > > <de...@kafka.apache.org> 主
> > >> > > >>>>> 题：Re: 回复：[Vote] KIP-571: Add option to force remove
> members
> > >> > > >>>>> in
> > >> > > >>>> StreamsResetter
> > >> > > >>>>>
> > >> > > >>>>> Thanks for the KIP, +1 (non-binding)
> > >> > > >>>>>
> > >> > > >>>>> On Thu, Feb 27, 2020 at 12:40 PM Boyang Chen <
> > >> > > >>> reluctanthero104@gmail.com
> > >> > > >>>>>
> > >> > > >>>>> wrote:
> > >> > > >>>>>
> > >> > > >>>>>> Thanks Feyman, +1 (non-binding)
> > >> > > >>>>>>
> > >> > > >>>>>> On Thu, Feb 27, 2020 at 9:25 AM John Roesler
> > >> > > >>>>>> <vv...@apache.org>
> > >> > > >>>> wrote:
> > >> > > >>>>>>
> > >> > > >>>>>>> Thanks for the proposal!
> > >> > > >>>>>>>
> > >> > > >>>>>>> I'm +1 (binding) -John
> > >> > > >>>>>>>
> > >> > > >>>>>>> On Wed, Feb 26, 2020, at 19:41, feyman2009 wrote:
> > >> > > >>>>>>>> Updated with the KIP link:
> > >> > > >>>>>>>>
> > >> > > >>>>>>>
> > >> > > >>>>>>
> > >> > > >>>>
> > >> > > >>>
> > >> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-571%3A+Add+opti
> > >> > > > on+to+force+remove+members+in+StreamsResetter
> > >> > > >>>>>>>>
> > >> > > >>>>>>>>
> > >> > > >>>>>>>>
> > >> > > >>>
> > >> > > >>>
> > >> > > >
> > ------------------------------------------------------------------
> > >> > > >>>>>>>> 发件人：feyman2009 <fe...@aliyun.com.INVALID> 发送时
> > >> > > >>>>>>>> 间：2020年2月27日(星期四) 09:38 收件人：dev
> <de...@kafka.apache.org>
> > >> > > >>>>>>>> 主 题：[Vote] KIP-571: Add option to force remove
> members
> > >> > > >>>>>>>> in
> > >> > > >>>>>> StreamsResetter
> > >> > > >>>>>>>>
> > >> > > >>>>>>>>
> > >> > > >>>>>>>> Hi, all I would like to start a vote on KIP-571:
> Add
> > >> > > >>>>>>>> option to force
> > >> > > >>>> remove
> > >> > > >>>>>>>> members in StreamsResetter .
> > >> > > >>>>>>>>
> > >> > > >>>>>>>> Thanks! Feyman
> > >> > > >>>>>>>>
> > >> > > >>>>>>>>
> > >> > > >>>>>>>
> > >> > > >>>>>>
> > >> > > >>>>>
> > >> > > >>>>>
> > >> > > >>>>
> > >> > > >>>>
> > >> > > >>>>
> > >> > > >>>> -- -- Guozhang
> > >> > > >>>>
> > >> > > >>>>
> > >> > > >>>>
> > >> > > >>>
> > >> > > >>> -- -- Guozhang
> > >> > > >>>
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > >
> > >> > >
> > >> >
> > >> > --
> > >> > -- Guozhang
> > >> >
> > >>
> > >>
> > >>
> > >>
> > >>
> > >
> > >
> >
> >
>
>
>
>
回复：回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in StreamsResetter
Posted by feyman2009 <fe...@aliyun.com.INVALID>.
Hi, all
KIP-571 has already collected 4 bind +1 (John, Guochang, Bill, Matthias) and 3 non-binding +1(Boyang, Sophie), I will mark it as approved and create a PR shortly.
Thanks!
Feyman
------------------------------------------------------------------
发件人：feyman2009 <fe...@aliyun.com.INVALID>
发送时间：2020年4月8日(星期三) 14:21
收件人：dev <de...@kafka.apache.org>; Boyang Chen <re...@gmail.com>
主 题：回复：回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in StreamsResetter
Hi Boyang,
Thanks for reminding me of that!
I'm not sure about the convention, I thought it would need to re-collect votes if the KIP has changed~
Let's leave the vote thread here for 2 days, if no objection, I will take it as approved and update the PR accordingly.
Thanks!
Feyman
------------------------------------------------------------------
发件人：Boyang Chen <re...@gmail.com>
发送时间：2020年4月8日(星期三) 12:42
收件人：dev <de...@kafka.apache.org>; feyman2009 <fe...@aliyun.com>
主 题：Re: 回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in StreamsResetter
You should already get enough votes if I'm counting correctly (Guozhang, John, Matthias)
On Tue, Apr 7, 2020 at 6:59 PM feyman2009 <fe...@aliyun.com.invalid> wrote:
Hi, Boyang&Matthias
I think Matthias's proposal makes sense, but we can use the admin tool for this scenario as Boyang mentioned or follow up later, so I prefer to keep this KIP unchanged to minimize the scope.
Calling for vote ~
Thanks!
Feyman
------------------------------------------------------------------
发件人：Boyang Chen <re...@gmail.com>
发送时间：2020年4月8日(星期三) 02:15
收件人：dev <de...@kafka.apache.org>
主 题：Re: 回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in StreamsResetter
Hey Feyman,
I think Matthias' suggestion is optional, and we could just use admin tool
to remove single static members as well.
Boyang
On Tue, Apr 7, 2020 at 11:00 AM Matthias J. Sax <mj...@apache.org> wrote:
> > Would you mind to elaborate why we still need that
>
> Sure.
>
> For static memership, the session timeout it usually set quite high.
> This make scaling in an application tricky: if you shut down one
> instance, no rebalance would happen until `session.timeout.ms` hits.
> This is specific to Kafka Streams, because when a Kafka Stream client is
> closed, it does _not_ send a `LeaveGroupRequest`. Hence, the
> corresponding partitions would not be processed for a long time and
> thus, fall back.
>
> Given that each instance will have a unique `instance.id` provided by
> the user, we could allow users to remove the instance they want to
> decommission from the consumer group without the need to wait for
> `session.timeout.ms`.
>
> Hence, it's not an application reset scenario for which one wants to
> remove all members, but a scaling-in scenario. For dynamic membership,
> this issue usually does not occur because the `session.timeout.ms` is
> set to a fairly low value and a rebalance would happen quickly after an
> instance is decommissioned.
>
> Does this make sense?
>
> As said before, we may or may not include this in this KIP. It's up to
> you if you want to address it or not.
>
>
> -Matthias
>
>
>
> On 4/7/20 7:12 AM, feyman2009 wrote:
> > Hi, Matthias
> > Thanks a lot!
> > So you do not plan so support removing a _single static_ member via
> `StreamsResetter`?
> > =>
> > Would you mind to elaborate why we still need that if we are
> able to batch remove active members with adminClient?
> >
> > Thanks!
> >
> > Feyman
> > ------------------------------------------------------------------
> > 发件人：Matthias J. Sax <mj...@apache.org>
> > 发送时间：2020年4月7日(星期二) 08:25
> > 收件人：dev <de...@kafka.apache.org>
> > 主 题：Re: 回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members
> in StreamsResetter
> >
> > Overall LGTM.
> >
> > +1 (binding)
> >
> > So you do not plan so support removing a _single static_ member via
> > `StreamsResetter`? We can of course still add this as a follow up but it
> > might be nice to just add it to this KIP right away. Up to you if you
> > want to include it or not.
> >
> >
> > -Matthias
> >
> >
> >
> > On 3/30/20 8:16 AM, feyman2009 wrote:
> >> Hi, Boyang
> >> Thanks a lot, that make sense, we should not expose member.id in
> the MemberToRemove anymore, I have fixed it in the KIP.
> >>
> >>
> >> Feyman
> >> ------------------------------------------------------------------
> >> 发件人：Boyang Chen <re...@gmail.com>
> >> 发送时间：2020年3月29日(星期日) 01:45
> >> 收件人：dev <de...@kafka.apache.org>; feyman2009 <fe...@aliyun.com>
> >> 主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove members in
> StreamsResetter
> >>
> >> Hey Feyman,
> >>
> >> thanks for the update. I assume we would rely entirely on the internal
> changes for `removeMemberFromGroup` by sending a DescribeGroup request
> first. With that in mind, I don't think we need to add member.id to
> MemberToRemove anymore, as it is only facing public where users will only
> configure group.instance.id?
> >> On Fri, Mar 27, 2020 at 5:04 PM feyman2009
> <fe...@aliyun.com.invalid> wrote:
> >> Bump, can anyone kindly take a look at the updated KIP-571? Thanks!
> >>
> >>
> >> ------------------------------------------------------------------
> >> 发件人：feyman2009 <fe...@aliyun.com.INVALID>
> >> 发送时间：2020年3月23日(星期一) 08:51
> >> 收件人：dev <de...@kafka.apache.org>
> >> 主 题：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in
> StreamsResetter
> >>
> >> Hi, team
> >> I have updated the KIP-571 according to our latest discussion
> results, would you mind to take a look? Thanks!
> >>
> >> Feyman
> >>
> >>
> >> ------------------------------------------------------------------
> >> 发件人：Boyang Chen <re...@gmail.com>
> >> 发送时间：2020年3月19日(星期四) 13:41
> >> 收件人：dev <de...@kafka.apache.org>; feyman2009 <fe...@aliyun.com>
> >> 主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove members in
> StreamsResetter
> >>
> >> Thanks for the insight Feyman. I personally feel adding another admin
> client command is redundant, so picking option 1). The memberInfos struct
> is internal and just used for result reference purposes. I think it could
> still work even we overload with `removeAll` option, if that makes sense.
> >>
> >> Boyang
> >> On Wed, Mar 18, 2020 at 8:51 PM feyman2009
> <fe...@aliyun.com.invalid> wrote:
> >> Hi, team
> >> Before going too far on the KIP update, I would like to hear your
> opinions about how we would change the interface of AdminClient, the two
> alternatives I could think of are:
> >> 1) Extend adminClient.removeMembersFromConsumerGroup to support
> remove all
> >> As Guochang suggested, we could add some flag param in
> RemoveMembersFromConsumerGroupOptions to indicated the "remove all" logic.
> >> 2) Add a new API like
> adminClient.removeAllMembersFromConsumerGroup(groupId, options)
> >>
> >> I think 1) will be more compact from the API perspective, but
> looking at the code, I found that the if we are going to remove all, then
> the RemoveMembersFromConsumerGroupResult#memberInfos/memberResult()/all()
> should be changed accordingly, and they seem not that meaningful under the
> "remove all" scenario.
> >>
> >> A minor thought about the
> adminClient.removeMembersFromConsumerGroup API is:
> >> Looking at some other deleteXX APIs, like deleteTopics,
> deleteRecords, the results contains only a Map<A, Future<B>>, I think it's
> enough to describe the related results, is it make sense that we may remove
> memberInfos in RemoveMembersFromConsumerGroupResult ? This KIP has no
> dependency on this if we choose alternative 2)
> >>
> >> Could you advise? Thanks!
> >>
> >> Feyman
> >>
> >>
> >> 送时间：2020年3月15日(星期日) 10:11
> >> 收件人：dev <de...@kafka.apache.org>
> >> 主 题：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in
> StreamsResetter
> >>
> >> Hi, all
> >> Thanks a lot for your feedback!
> >> According to the discussion, it seems we don't have some valid
> use cases for removing specific dynamic members, I think it makes sense to
> encapsulate the "get and delete" logic in adminClient. I will update the
> KIP shortly!
> >>
> >> Thanks!
> >>
> >> Feyman
> >>
> >>
> >> ------------------------------------------------------------------
> >> 发件人：Boyang Chen <re...@gmail.com>
> >> 发送时间：2020年3月14日(星期六) 00:39
> >> 收件人：dev <de...@kafka.apache.org>
> >> 主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove members
> in StreamsResetter
> >>
> >> Thanks Matthias and Guozhang for the feedback. I'm not worrying too
> much
> >> about the member.id exposure as we have done so in a couple of
> areas. As
> >> for the recommended admin client change, I think it makes sense in an
> >> encapsulation perspective. Maybe I'm still a bit hesitant as we are
> losing
> >> the flexibility of closing only a subset of `dynamic members`
> potentially,
> >> but we could always get back and address it if some user feels
> necessary to
> >> have it.
> >>
> >> My short answer would be, LGTM :)
> >>
> >> Boyang
> >>
> >> On Thu, Mar 12, 2020 at 5:26 PM Guozhang Wang <wa...@gmail.com>
> wrote:
> >>
> >> > Hi Matthias,
> >> >
> >> > About the AdminClient param API: that's a great point here. I think
> overall
> >> > if users want to just "remove all members" they should not need to
> first
> >> > get all the member.ids themselves, but instead internally the admin
> client
> >> > can first issue a describe-group request to get all the member.ids,
> and
> >> > then use them in the next issued leave-group request, all
> abstracted away
> >> > from the users. With that in mind, maybe in
> >> > RemoveMembersFromConsumerGroupOptions we can just introduce an
> overloaded
> >> > flag param besides "members" that indicate "remove all"?
> >> >
> >> > Guozhang
> >> >
> >> > On Thu, Mar 12, 2020 at 2:59 PM Matthias J. Sax <mj...@apache.org>
> wrote:
> >> >
> >> > > Feyman,
> >> > >
> >> > > some more comments/questions:
> >> > >
> >> > > The description of `LeaveGroupRequest` is clear but it's unclear
> how
> >> > > `MemberToRemove` should behave. Which parameter is required?
> Which is
> >> > > optional? What is the relationship between both.
> >> > >
> >> > > The `LeaveGroupRequest` description clearly states that
> specifying a
> >> > > `memberId` is optional if the `groupInstanceId` is provided. If
> >> > > `MemberToRemove` applies the same pattern, it must be explicitly
> defined
> >> > > in the KIP (and explained in the JavaDocs of `MemberToRemove`)
> because
> >> > > we cannot expect that an admin-client users knows that internally
> a
> >> > > `LeaveGroupRequest` is used nor what the semantics of a
> >> > > `LeaveGroupRequest` are.
> >> > >
> >> > >
> >> > > About Admin API:
> >> > >
> >> > > In general, I am also confused that we allow so specify a
> `memberId` at
> >> > > all, because the `memberId` is an internal id that is not really
> exposed
> >> > > to the user. Hence, from a AdminClient point of view, accepting a
> >> > > `memberId` as input seems questionable to me? Of course,
> `memberId` can
> >> > > be collected via `describeConsumerGroups()` but it will return the
> >> > > `memberId` of _all_ consumer in the group and thus how would a
> user know
> >> > > which member should be removed for a dynamic group (if an
> individual
> >> > > member should be removed)?
> >> > >
> >> > > Hence, how can any user get to know the `memberId` of an
> individual
> >> > > client in a programtic way?
> >> > >
> >> > > Also I am wondering in general, why the removal of single dynamic
> member
> >> > > is important? In general, I would expect a short
> `session.timeout` for
> >> > > dynamic groups and thus removing a specific member from the group
> seems
> >> > > not to be an important feature -- for static groups we expect a
> long
> >> > > `session.timeout` and a user can also identify individual clients
> via
> >> > > `groupInstandId`, hence the feature makes sense for this case and
> is
> >> > > straight forward to use.
> >> > >
> >> > >
> >> > > About StreamsResetter:
> >> > >
> >> > > For this case we just say "remove all members" and thus the
> >> > > `describeConsumerGroup` approach works. However, it seems to be a
> >> > > special case?
> >> > >
> >> > > Or, if we expected that the "remove all members" use case is the
> norm,
> >> > > why can't we make a change admin-client to directly accept a `
> group.id`?
> >> > > The admin-client can internal first do a `DescribeGroupRequest`
> and
> >> > > afterward corresponding `LeaveGroupRequest` -- i.e., instead of
> building
> >> > > this pattern in `StreamsResetter` we build it directly into
> >> > `AdminClient`.
> >> > >
> >> > > Last, for static group the main use case seems to be to remove an
> >> > > individual member from the group but this feature is not covered
> by the
> >> > > KIP: I think using `--force` to remove all members makes sense,
> but an
> >> > > important second feature to remove an individual static member
> would
> >> > > require it's own flag to specify a single `group.instance.id`.
> >> > >
> >> > >
> >> > > Thoughts?
> >> > >
> >> > >
> >> > > -Matthias
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > On 3/11/20 8:43 PM, feyman2009 wrote:
> >> > > > Hi, Sophie
> >> > > > For 1) Sorry, I found that my expression is kind of
> misleading,
> >> > > what I actually mean is: "if --force not specified, an exception
> saying
> >> > > there are still active members on broker side will be thrown and
> >> > > suggesting using StreamsResetter with --force", I just updated
> the KIP
> >> > > page.
> >> > > >
> >> > > > For 2)
> >> > > > I may also had some misleading expression previous, to
> clarify
> >> > :
> >> > > >
> >> > > > Also, it's more efficient to just send a single "clear the
> group"
> >> > > request vs sending a LeaveGroup
> >> > > > request for every single member. What do you think?
> >> > > > => the comparison is to send a single "clear the group" request
> vs
> >> > > sending a "get members" + a "remove members" request since the
> >> > > adminClient.removeMembersFromConsumerGroup support batch removal.
> We
> >> > > don't need to send lots of leaveGroup requests for every single
> member.
> >> > > >
> >> > > > I can understand your point, but I think we could reuse
> the
> >> > > current
> >> > > > adminClient.removeMembersFromConsumerGroup interface
> effectively with
> >> > > the KIP.
> >> > > > What do you think?
> >> > > >
> >> > > > Thanks!
> >> > > >
> >> > > > Feyman
> >> > > >
> >> > > >
> >> > > >
> ------------------------------------------------------------------
> >> > > > 发件人：Sophie Blee-Goldman <so...@confluent.io>
> >> > > > 发送时间：2020年3月10日(星期二) 03:02
> >> > > > 收件人：dev <de...@kafka.apache.org>; feyman2009 <
> feyman2009@aliyun.com>
> >> > > > 主 题：Re: 回复：回复：[Vote] KIP-571: Add option to force remove
> >> > > members in StreamsResetter
> >> > > >
> >> > > > Hey Feyman,
> >> > > >
> >> > > > 1) Regarding point 2 in your last email, if I understand
> correctly you
> >> > > propose to change
> >> > > > the current behavior of the reset tool when --force is not
> specified,
> >> > > and wait for (up to)
> >> > > > the session timeout for all members to be removed. I'm not sure
> we
> >> > > should change this,
> >> > > > especially now that we have a better way to handle the case
> when the
> >> > > group is not empty:
> >> > > > we should continue to throw an exception and fail fast, but can
> print
> >> > > a message suggesting
> >> > > > to use the new --force option to remove remaining group
> members. Why
> >> > > make users wait
> >> > > > for the session timeout when we've just added a new feature
> that means
> >> > > they don't have to?
> >> > > >
> >> > > > 2) Regarding Matthias' question:
> >> > > >
> >> > > >> I am really wondering, if for a static group, we should allow
> users
> >> > > toremove individual members? For a dynamic group this feature
> would not
> >> > > > make much sense IMHO, because the `memberId` is not know by the
> user.
> >> > > >
> >> > > > I think his point is similar to what I was trying to get at
> earlier,
> >> > > with the proposal to add a new
> >> > > > #removeAllMembers API rather than an API to remove individual
> members
> >> > > according to their
> >> > > > memberId. As he explained, removing based on memberId is likely
> not
> >> > > that useful in general.
> >> > > > Also, it's not actually what we want to do here; maybe we
> should avoid
> >> > > adding a new API
> >> > > > that we think may be useful in other contexts (remove individual
> >> > > member based on memberId),
> >> > > > and just add the API we actually need (remove all members from
> group)
> >> > > in this KIP? We can
> >> > > > always add the "remove individual member by memberId" API at a
> later
> >> > > point, if it turns out to
> >> > > > actually be requested for specific reasons?
> >> > > >
> >> > > > Also, it's more efficient to just send a single "clear the
> group"
> >> > > request vs sending a LeaveGroup
> >> > > > request for every single member. What do you think?
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Sat, Mar 7, 2020 at 1:41 AM feyman2009
> >> > > <fe...@aliyun.com.invalid> wrote:
> >> > > > Hi, Matthias
> >> > > > Thanks, I updated the KIP to mention the deprecated and
> newly
> >> > > added methods.
> >> > > >
> >> > > > 1. What happens is `groupInstanceId` is used for a dynamic
> group? What
> >> > > > happens if both parameters are specified? What happens if
> `memberId`
> >> > > > is specified for a static group?
> >> > > >
> >> > > > => my understanding is that the dynamic/static membership is
> member
> >> > > level other than group level, and I think above questions could be
> >> > > answered by the "Leave Group Logic Change" section in KIP-345:
> >> > >
> >> > >
> >> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> >> > > ,
> >> > > this KIP stays consistent with KIP-345.
> >> > > >
> >> > > > 2. About the `--force` option. Currently, StreamsResetter
> fails with
> >> > an
> >> > > > error if the consumer group is not empty. You state in your
> KIP that:
> >> > > >
> >> > > > > without --force, we need to wait for session timeout.
> >> > > >
> >> > > > Is this an intended behavior change if `--force` is not used
> or is the
> >> > > > KIP description incorrect?
> >> > > >
> >> > > > => This is the intended behavior. For this part, I think there
> are
> >> > > two ways to go:
> >> > > > 1) (the implicit way) Not introducing the new "--force"
> option, with
> >> > > this KIP, StreamsResetter will by default remove active
> members(with
> >> > > long session timeout configured) on broker side
> >> > > > 2) (the explicit way) Introduce the new "--force" option,
> users need
> >> > > to explicitly specify --force to remove the active members. If
> --force
> >> > > not specified, the StreamsResetter behaviour is as previous
> versions'.
> >> > > >
> >> > > > I think the two alternatives above are both feasible,
> personally I
> >> > > prefer way 2.
> >> > > >
> >> > > > 3. For my own understanding: with the `--force` option, we
> intend to
> >> > get
> >> > > > all `memberIds` and send a "remove member" request for each
> with
> >> > > > corresponding `memberId` to remove the member from the group
> >> > > > (independent is the group is static or dynamic)?
> >> > > >
> >> > > > => Yeah, minor thing to mention is we will send the "remove
> member"
> >> > > request for each member(could be dynamic member or static member)
> to
> >> > > remove them from group
> >> > > > for dynamic members, both "group.instance.id" and "member.id"
> will be
> >> > > specified
> >> > > > for dynamic members, only "member.id" will be specified
> >> > > >
> >> > > > 4. I am really wondering, if for a static group, we should
> allow users
> >> > > to
> >> > > > remove individual members? For a dynamic group this feature
> would not
> >> > > > make much sense IMHO, because the `memberId` is not know by
> the user.
> >> > > >
> >> > > > => KIP-345 introduced the batch removal feature for both static
> >> > > member and dynamic member, my understanding is that "allow users
> to
> >> > > > remove individual members" could be useful for rolling bounce
> and
> >> > > scale down accoding to KIP-345. KafkaAdminClient currently only
> support
> >> > > static member removal and this KIP-571 enables dynamic member
> removal
> >> > > for it, which is also consistent with the broker side logic.
> Users could
> >> > > get the member.id (and group.instance.id if for static member) by
> >> > > adminClient.describeConsumerGroups.
> >> > > >
> >> > > > Furthermore, I don't have an assumption that a consumer group
> should
> >> > > contain only static or dynamic members, and I think KIP-345 and
> this KIP
> >> > > don't need to be based on this assumption.
> >> > > > You could correct me if I have the wrong understanding :)
> >> > > >
> >> > > > Thanks!
> >> > > > Feyman
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> ------------------------------------------------------------------
> >> > > > 发件人：Matthias J. Sax <mj...@apache.org>
> >> > > > 发送时间：2020年3月6日(星期五) 02:20
> >> > > > 收件人：dev <de...@kafka.apache.org>
> >> > > > 主 题：Re: 回复：回复：[Vote] KIP-571: Add option to force remove
> >> > > members in StreamsResetter
> >> > > >
> >> > > > Feyman,
> >> > > >
> >> > > > thanks a lot for the KIP. Overall LGTM. I have a few more
> comment and
> >> > > > questions (sorry for the late reply):
> >> > > >
> >> > > >
> >> > > > The KIP mentions that some constructors will be deprecated.
> Those
> >> > should
> >> > > > be listed explicitly. For example:
> >> > > >
> >> > > >
> >> > > > public class MemberToRemove {
> >> > > >
> >> > > > // deprecated methods
> >> > > >
> >> > > > @Deprecated
> >> > > > public MemberToRemove(String groupInstanceId);
> >> > > >
> >> > > > // new methods
> >> > > >
> >> > > > public MemberToRemove()
> >> > > >
> >> > > > public MemberToRemove withGroupInstanceId(String
> groupInstanceId)
> >> > > >
> >> > > > public MemberToRemove withMemberId(String memberId)
> >> > > > }
> >> > > >
> >> > > > What happens is `groupInstanceId` is used for a dynamic group?
> What
> >> > > > happens if both parameters are specified? What happens if
> `memberId`
> >> > > > is specified for a static group?
> >> > > >
> >> > > >
> >> > > > About the `--force` option. Currently, StreamsResetter fails
> with an
> >> > > > error if the consumer group is not empty. You state in your KIP
> that:
> >> > > >
> >> > > >> without --force, we need to wait for session timeout.
> >> > > >
> >> > > > Is this an intended behavior change if `--force` is not used or
> is the
> >> > > > KIP description incorrect?
> >> > > >
> >> > > > For my own understanding: with the `--force` option, we intend
> to get
> >> > > > all `memberIds` and send a "remove member" request for each with
> >> > > > corresponding `memberId` to remove the member from the group
> >> > > > (independent is the group is static or dynamic)?
> >> > > >
> >> > > > I am really wondering, if for a static group, we should allow
> users to
> >> > > > remove individual members? For a dynamic group this feature
> would not
> >> > > > make much sense IMHO, because the `memberId` is not know by the
> user.
> >> > > >
> >> > > >
> >> > > >
> >> > > > -Matthias
> >> > > >
> >> > > >
> >> > > > On 3/5/20 7:15 AM, Bill Bejeck wrote:
> >> > > >> Thanks for the KIP. +1 (binding).
> >> > > >
> >> > > >> -Bill
> >> > > >
> >> > > >
> >> > > >
> >> > > >> On Wed, Mar 4, 2020 at 12:40 AM Guozhang Wang <
> wangguoz@gmail.com>
> >> > > >> wrote:
> >> > > >
> >> > > >>> Thanks, +1 from me (binding).
> >> > > >>>
> >> > > >>> On Tue, Mar 3, 2020 at 9:39 PM feyman2009 <
> feyman2009@aliyun.com>
> >> > > >>> wrote:
> >> > > >>>
> >> > > >>>> Hi, Guozhang Thanks a lot for the advice, that make sense! I
> >> > > >>>> have updated the KIP page with the operational steps of
> >> > > >>>> StreamsResetter.
> >> > > >>>>
> >> > > >>>> Thanks! Feyman
> >> > > >>>>
> >> > > >>>>
> ------------------------------------------------------------------
> >> > > >>>>
> >> > > >>>>
> >> > > > 发件人：Guozhang Wang <wa...@gmail.com>
> >> > > >>>> 发送时间：2020年3月3日(星期二) 14:22 收件人：dev <de...@kafka.apache.org>;
> >> > > >>>> feyman2009 <fe...@aliyun.com> 主 题：Re: 回复：回复：[Vote]
> >> > > >>>> KIP-571: Add option to force remove members in
> StreamsResetter
> >> > > >>>>
> >> > > >>>> Hello Feyman, thanks for the proposal!
> >> > > >>>>
> >> > > >>>> I read through the doc and overall it looks good to me.
> >> > > >>>>
> >> > > >>>> One minor thing I'd still like to point out is that, the
> >> > > >>>> "removeMembersFromConsumerGroup" only sends a leave-group
> >> > > >>>> request to the coordinator to let it remove the member,
> >> > > >>>> however, if the member is still there alive and running then
> it
> >> > > >>>> would soon be notified that it is no
> >> > > >>> longer
> >> > > >>>> a legal member of the group via heartbeats, and then
> >> > > >>>> automatically tries
> >> > > >>> to
> >> > > >>>> re-join the group. So on the operational side, it is still
> >> > > >>>> required that the following steps:
> >> > > >>>>
> >> > > >>>> 1) first stop the consumers (of streams instances), wait
> until
> >> > > >>>> the shutdown is complete. 2) then use admin client in case
> the
> >> > > >>>> stopped consumers are still registered at the broker side and
> >> > > >>>> we do not want to wait for session timeout.
> >> > > >>>>
> >> > > >>>> Even with this KIP, people should still not skip step 1)
> above,
> >> > > >>>> since otherwise the consumers would re-connect and re-join
> the
> >> > > >>>> group
> >> > > >>> immediately
> >> > > >>>> still.
> >> > > >>>>
> >> > > >>>> In your doc you've already mentioned "Furthermore, users
> should
> >> > > >>>> make sure all the stream applications are shutdown when
> running
> >> > > >>>> StreamsResetter
> >> > > >>> with
> >> > > >>>> --force, otherwise it might trigger unexpected rebalance. "
> >> > > >>>> What I'd want to clarify is that no matter if "--force"
> option
> >> > > >>>> is enabled, this is
> >> > > >>> always
> >> > > >>>> the case that users should shutdown the streams instances
> >> > > >>>> first, and then use the streams resetter :)
> >> > > >>>>
> >> > > >>>> As long as that is clarified in the proposal documentation,
> I'm
> >> > > >>>> +1 on
> >> > > >>> this
> >> > > >>>> KIP.
> >> > > >>>>
> >> > > >>>>
> >> > > >>>> Thanks again for the contribution, Guozhang
> >> > > >>>>
> >> > > >>>>
> >> > > >>>> On Mon, Mar 2, 2020 at 6:31 AM feyman2009
> >> > > >>>> <feyman2009@aliyun.com.invalid
> >> > > >>>>
> >> > > >>>> wrote: Hi, John Sorry, I have mistaken the KIP approval
> >> > > >>>> standard, anyway, I will
> >> > > >>> start
> >> > > >>>> the PR soon and waiting for more binding approvals.
> >> > > >>>>
> >> > > >>>> Thanks! Feyman
> >> > > >>>>
> >> > > >>>>
> >> > > >>>>
> ------------------------------------------------------------------
> >> > > >>>>
> >> > > >>>>
> >> > > > 发件人：John Roesler <vv...@apache.org>
> >> > > >>>> 发送时间：2020年3月2日(星期一) 22:00 收件人：dev
> >> > > > <de...@kafka.apache.org> 主
> >> > > >>>> 题：Re: 回复：回复：[Vote] KIP-571: Add option to force remove
> members
> >> > > >>>> in StreamsResetter
> >> > > >>>>
> >> > > >>>> Hi Feyman,
> >> > > >>>>
> >> > > >>>> Sorry, but we actually need 3 binding votes for the KIP to
> >> > > >>>> pass. Please feel free to keep bumping the thread until some
> >> > > >>>> more committers can take
> >> > > >>> a
> >> > > >>>> look.
> >> > > >>>>
> >> > > >>>> By the way, you can totally start a PR, but we can’t merge it
> >> > > >>>> until the KIP passes the vote.
> >> > > >>>>
> >> > > >>>> Thanks! John
> >> > > >>>>
> >> > > >>>> On Mon, Mar 2, 2020, at 00:24, feyman2009 wrote:
> >> > > >>>>> Hi,all Since currently we have 1 binding and two non-binding
> >> > > >>>>> +1, I will update the KIP-571 as adopted and initiate a PR
> >> > > >>>>> shortly
> >> > > >>>>>
> >> > > >>>>> Thanks! Feyman
> >> > > >>>>>
> >> > > >>>>>
> >> > > >>>>>
> ------------------------------------------------------------------
> >> > > >>>>>
> >> > > >>>>>
> >> > > > 发件人：Sophie Blee-Goldman <so...@confluent.io>
> >> > > >>>>> 发送时间：2020年2月28日(星期五) 10:17 收件人：dev
> >> > > > <de...@kafka.apache.org> 主
> >> > > >>>>> 题：Re: 回复：[Vote] KIP-571: Add option to force remove members
> >> > > >>>>> in
> >> > > >>>> StreamsResetter
> >> > > >>>>>
> >> > > >>>>> Thanks for the KIP, +1 (non-binding)
> >> > > >>>>>
> >> > > >>>>> On Thu, Feb 27, 2020 at 12:40 PM Boyang Chen <
> >> > > >>> reluctanthero104@gmail.com
> >> > > >>>>>
> >> > > >>>>> wrote:
> >> > > >>>>>
> >> > > >>>>>> Thanks Feyman, +1 (non-binding)
> >> > > >>>>>>
> >> > > >>>>>> On Thu, Feb 27, 2020 at 9:25 AM John Roesler
> >> > > >>>>>> <vv...@apache.org>
> >> > > >>>> wrote:
> >> > > >>>>>>
> >> > > >>>>>>> Thanks for the proposal!
> >> > > >>>>>>>
> >> > > >>>>>>> I'm +1 (binding) -John
> >> > > >>>>>>>
> >> > > >>>>>>> On Wed, Feb 26, 2020, at 19:41, feyman2009 wrote:
> >> > > >>>>>>>> Updated with the KIP link:
> >> > > >>>>>>>>
> >> > > >>>>>>>
> >> > > >>>>>>
> >> > > >>>>
> >> > > >>>
> >> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-571%3A+Add+opti
> >> > > > on+to+force+remove+members+in+StreamsResetter
> >> > > >>>>>>>>
> >> > > >>>>>>>>
> >> > > >>>>>>>>
> >> > > >>>
> >> > > >>>
> >> > > >
> ------------------------------------------------------------------
> >> > > >>>>>>>> 发件人：feyman2009 <fe...@aliyun.com.INVALID> 发送时
> >> > > >>>>>>>> 间：2020年2月27日(星期四) 09:38 收件人：dev <de...@kafka.apache.org>
> >> > > >>>>>>>> 主 题：[Vote] KIP-571: Add option to force remove members
> >> > > >>>>>>>> in
> >> > > >>>>>> StreamsResetter
> >> > > >>>>>>>>
> >> > > >>>>>>>>
> >> > > >>>>>>>> Hi, all I would like to start a vote on KIP-571: Add
> >> > > >>>>>>>> option to force
> >> > > >>>> remove
> >> > > >>>>>>>> members in StreamsResetter .
> >> > > >>>>>>>>
> >> > > >>>>>>>> Thanks! Feyman
> >> > > >>>>>>>>
> >> > > >>>>>>>>
> >> > > >>>>>>>
> >> > > >>>>>>
> >> > > >>>>>
> >> > > >>>>>
> >> > > >>>>
> >> > > >>>>
> >> > > >>>>
> >> > > >>>> -- -- Guozhang
> >> > > >>>>
> >> > > >>>>
> >> > > >>>>
> >> > > >>>
> >> > > >>> -- -- Guozhang
> >> > > >>>
> >> > > >
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> >
> >> > --
> >> > -- Guozhang
> >> >
> >>
> >>
> >>
> >>
> >>
> >
> >
>
>
回复：回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in StreamsResetter
Posted by feyman2009 <fe...@aliyun.com.INVALID>.
Hi Boyang,
Thanks for reminding me of that!
I'm not sure about the convention, I thought it would need to re-collect votes if the KIP has changed~
Let's leave the vote thread here for 2 days, if no objection, I will take it as approved and update the PR accordingly.
Thanks!
Feyman
------------------------------------------------------------------
发件人：Boyang Chen <re...@gmail.com>
发送时间：2020年4月8日(星期三) 12:42
收件人：dev <de...@kafka.apache.org>; feyman2009 <fe...@aliyun.com>
主 题：Re: 回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in StreamsResetter
You should already get enough votes if I'm counting correctly (Guozhang, John, Matthias)
On Tue, Apr 7, 2020 at 6:59 PM feyman2009 <fe...@aliyun.com.invalid> wrote:
Hi, Boyang&Matthias
I think Matthias's proposal makes sense, but we can use the admin tool for this scenario as Boyang mentioned or follow up later, so I prefer to keep this KIP unchanged to minimize the scope.
Calling for vote ~
Thanks!
Feyman
------------------------------------------------------------------
发件人：Boyang Chen <re...@gmail.com>
发送时间：2020年4月8日(星期三) 02:15
收件人：dev <de...@kafka.apache.org>
主 题：Re: 回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in StreamsResetter
Hey Feyman,
I think Matthias' suggestion is optional, and we could just use admin tool
to remove single static members as well.
Boyang
On Tue, Apr 7, 2020 at 11:00 AM Matthias J. Sax <mj...@apache.org> wrote:
> > Would you mind to elaborate why we still need that
>
> Sure.
>
> For static memership, the session timeout it usually set quite high.
> This make scaling in an application tricky: if you shut down one
> instance, no rebalance would happen until `session.timeout.ms` hits.
> This is specific to Kafka Streams, because when a Kafka Stream client is
> closed, it does _not_ send a `LeaveGroupRequest`. Hence, the
> corresponding partitions would not be processed for a long time and
> thus, fall back.
>
> Given that each instance will have a unique `instance.id` provided by
> the user, we could allow users to remove the instance they want to
> decommission from the consumer group without the need to wait for
> `session.timeout.ms`.
>
> Hence, it's not an application reset scenario for which one wants to
> remove all members, but a scaling-in scenario. For dynamic membership,
> this issue usually does not occur because the `session.timeout.ms` is
> set to a fairly low value and a rebalance would happen quickly after an
> instance is decommissioned.
>
> Does this make sense?
>
> As said before, we may or may not include this in this KIP. It's up to
> you if you want to address it or not.
>
>
> -Matthias
>
>
>
> On 4/7/20 7:12 AM, feyman2009 wrote:
> > Hi, Matthias
> > Thanks a lot!
> > So you do not plan so support removing a _single static_ member via
> `StreamsResetter`?
> > =>
> > Would you mind to elaborate why we still need that if we are
> able to batch remove active members with adminClient?
> >
> > Thanks!
> >
> > Feyman
> > ------------------------------------------------------------------
> > 发件人：Matthias J. Sax <mj...@apache.org>
> > 发送时间：2020年4月7日(星期二) 08:25
> > 收件人：dev <de...@kafka.apache.org>
> > 主 题：Re: 回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members
> in StreamsResetter
> >
> > Overall LGTM.
> >
> > +1 (binding)
> >
> > So you do not plan so support removing a _single static_ member via
> > `StreamsResetter`? We can of course still add this as a follow up but it
> > might be nice to just add it to this KIP right away. Up to you if you
> > want to include it or not.
> >
> >
> > -Matthias
> >
> >
> >
> > On 3/30/20 8:16 AM, feyman2009 wrote:
> >> Hi, Boyang
> >> Thanks a lot, that make sense, we should not expose member.id in
> the MemberToRemove anymore, I have fixed it in the KIP.
> >>
> >>
> >> Feyman
> >> ------------------------------------------------------------------
> >> 发件人：Boyang Chen <re...@gmail.com>
> >> 发送时间：2020年3月29日(星期日) 01:45
> >> 收件人：dev <de...@kafka.apache.org>; feyman2009 <fe...@aliyun.com>
> >> 主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove members in
> StreamsResetter
> >>
> >> Hey Feyman,
> >>
> >> thanks for the update. I assume we would rely entirely on the internal
> changes for `removeMemberFromGroup` by sending a DescribeGroup request
> first. With that in mind, I don't think we need to add member.id to
> MemberToRemove anymore, as it is only facing public where users will only
> configure group.instance.id?
> >> On Fri, Mar 27, 2020 at 5:04 PM feyman2009
> <fe...@aliyun.com.invalid> wrote:
> >> Bump, can anyone kindly take a look at the updated KIP-571? Thanks!
> >>
> >>
> >> ------------------------------------------------------------------
> >> 发件人：feyman2009 <fe...@aliyun.com.INVALID>
> >> 发送时间：2020年3月23日(星期一) 08:51
> >> 收件人：dev <de...@kafka.apache.org>
> >> 主 题：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in
> StreamsResetter
> >>
> >> Hi, team
> >> I have updated the KIP-571 according to our latest discussion
> results, would you mind to take a look? Thanks!
> >>
> >> Feyman
> >>
> >>
> >> ------------------------------------------------------------------
> >> 发件人：Boyang Chen <re...@gmail.com>
> >> 发送时间：2020年3月19日(星期四) 13:41
> >> 收件人：dev <de...@kafka.apache.org>; feyman2009 <fe...@aliyun.com>
> >> 主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove members in
> StreamsResetter
> >>
> >> Thanks for the insight Feyman. I personally feel adding another admin
> client command is redundant, so picking option 1). The memberInfos struct
> is internal and just used for result reference purposes. I think it could
> still work even we overload with `removeAll` option, if that makes sense.
> >>
> >> Boyang
> >> On Wed, Mar 18, 2020 at 8:51 PM feyman2009
> <fe...@aliyun.com.invalid> wrote:
> >> Hi, team
> >> Before going too far on the KIP update, I would like to hear your
> opinions about how we would change the interface of AdminClient, the two
> alternatives I could think of are:
> >> 1) Extend adminClient.removeMembersFromConsumerGroup to support
> remove all
> >> As Guochang suggested, we could add some flag param in
> RemoveMembersFromConsumerGroupOptions to indicated the "remove all" logic.
> >> 2) Add a new API like
> adminClient.removeAllMembersFromConsumerGroup(groupId, options)
> >>
> >> I think 1) will be more compact from the API perspective, but
> looking at the code, I found that the if we are going to remove all, then
> the RemoveMembersFromConsumerGroupResult#memberInfos/memberResult()/all()
> should be changed accordingly, and they seem not that meaningful under the
> "remove all" scenario.
> >>
> >> A minor thought about the
> adminClient.removeMembersFromConsumerGroup API is:
> >> Looking at some other deleteXX APIs, like deleteTopics,
> deleteRecords, the results contains only a Map<A, Future<B>>, I think it's
> enough to describe the related results, is it make sense that we may remove
> memberInfos in RemoveMembersFromConsumerGroupResult ? This KIP has no
> dependency on this if we choose alternative 2)
> >>
> >> Could you advise? Thanks!
> >>
> >> Feyman
> >>
> >>
> >> 送时间：2020年3月15日(星期日) 10:11
> >> 收件人：dev <de...@kafka.apache.org>
> >> 主 题：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in
> StreamsResetter
> >>
> >> Hi, all
> >> Thanks a lot for your feedback!
> >> According to the discussion, it seems we don't have some valid
> use cases for removing specific dynamic members, I think it makes sense to
> encapsulate the "get and delete" logic in adminClient. I will update the
> KIP shortly!
> >>
> >> Thanks!
> >>
> >> Feyman
> >>
> >>
> >> ------------------------------------------------------------------
> >> 发件人：Boyang Chen <re...@gmail.com>
> >> 发送时间：2020年3月14日(星期六) 00:39
> >> 收件人：dev <de...@kafka.apache.org>
> >> 主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove members
> in StreamsResetter
> >>
> >> Thanks Matthias and Guozhang for the feedback. I'm not worrying too
> much
> >> about the member.id exposure as we have done so in a couple of
> areas. As
> >> for the recommended admin client change, I think it makes sense in an
> >> encapsulation perspective. Maybe I'm still a bit hesitant as we are
> losing
> >> the flexibility of closing only a subset of `dynamic members`
> potentially,
> >> but we could always get back and address it if some user feels
> necessary to
> >> have it.
> >>
> >> My short answer would be, LGTM :)
> >>
> >> Boyang
> >>
> >> On Thu, Mar 12, 2020 at 5:26 PM Guozhang Wang <wa...@gmail.com>
> wrote:
> >>
> >> > Hi Matthias,
> >> >
> >> > About the AdminClient param API: that's a great point here. I think
> overall
> >> > if users want to just "remove all members" they should not need to
> first
> >> > get all the member.ids themselves, but instead internally the admin
> client
> >> > can first issue a describe-group request to get all the member.ids,
> and
> >> > then use them in the next issued leave-group request, all
> abstracted away
> >> > from the users. With that in mind, maybe in
> >> > RemoveMembersFromConsumerGroupOptions we can just introduce an
> overloaded
> >> > flag param besides "members" that indicate "remove all"?
> >> >
> >> > Guozhang
> >> >
> >> > On Thu, Mar 12, 2020 at 2:59 PM Matthias J. Sax <mj...@apache.org>
> wrote:
> >> >
> >> > > Feyman,
> >> > >
> >> > > some more comments/questions:
> >> > >
> >> > > The description of `LeaveGroupRequest` is clear but it's unclear
> how
> >> > > `MemberToRemove` should behave. Which parameter is required?
> Which is
> >> > > optional? What is the relationship between both.
> >> > >
> >> > > The `LeaveGroupRequest` description clearly states that
> specifying a
> >> > > `memberId` is optional if the `groupInstanceId` is provided. If
> >> > > `MemberToRemove` applies the same pattern, it must be explicitly
> defined
> >> > > in the KIP (and explained in the JavaDocs of `MemberToRemove`)
> because
> >> > > we cannot expect that an admin-client users knows that internally
> a
> >> > > `LeaveGroupRequest` is used nor what the semantics of a
> >> > > `LeaveGroupRequest` are.
> >> > >
> >> > >
> >> > > About Admin API:
> >> > >
> >> > > In general, I am also confused that we allow so specify a
> `memberId` at
> >> > > all, because the `memberId` is an internal id that is not really
> exposed
> >> > > to the user. Hence, from a AdminClient point of view, accepting a
> >> > > `memberId` as input seems questionable to me? Of course,
> `memberId` can
> >> > > be collected via `describeConsumerGroups()` but it will return the
> >> > > `memberId` of _all_ consumer in the group and thus how would a
> user know
> >> > > which member should be removed for a dynamic group (if an
> individual
> >> > > member should be removed)?
> >> > >
> >> > > Hence, how can any user get to know the `memberId` of an
> individual
> >> > > client in a programtic way?
> >> > >
> >> > > Also I am wondering in general, why the removal of single dynamic
> member
> >> > > is important? In general, I would expect a short
> `session.timeout` for
> >> > > dynamic groups and thus removing a specific member from the group
> seems
> >> > > not to be an important feature -- for static groups we expect a
> long
> >> > > `session.timeout` and a user can also identify individual clients
> via
> >> > > `groupInstandId`, hence the feature makes sense for this case and
> is
> >> > > straight forward to use.
> >> > >
> >> > >
> >> > > About StreamsResetter:
> >> > >
> >> > > For this case we just say "remove all members" and thus the
> >> > > `describeConsumerGroup` approach works. However, it seems to be a
> >> > > special case?
> >> > >
> >> > > Or, if we expected that the "remove all members" use case is the
> norm,
> >> > > why can't we make a change admin-client to directly accept a `
> group.id`?
> >> > > The admin-client can internal first do a `DescribeGroupRequest`
> and
> >> > > afterward corresponding `LeaveGroupRequest` -- i.e., instead of
> building
> >> > > this pattern in `StreamsResetter` we build it directly into
> >> > `AdminClient`.
> >> > >
> >> > > Last, for static group the main use case seems to be to remove an
> >> > > individual member from the group but this feature is not covered
> by the
> >> > > KIP: I think using `--force` to remove all members makes sense,
> but an
> >> > > important second feature to remove an individual static member
> would
> >> > > require it's own flag to specify a single `group.instance.id`.
> >> > >
> >> > >
> >> > > Thoughts?
> >> > >
> >> > >
> >> > > -Matthias
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > On 3/11/20 8:43 PM, feyman2009 wrote:
> >> > > > Hi, Sophie
> >> > > > For 1) Sorry, I found that my expression is kind of
> misleading,
> >> > > what I actually mean is: "if --force not specified, an exception
> saying
> >> > > there are still active members on broker side will be thrown and
> >> > > suggesting using StreamsResetter with --force", I just updated
> the KIP
> >> > > page.
> >> > > >
> >> > > > For 2)
> >> > > > I may also had some misleading expression previous, to
> clarify
> >> > :
> >> > > >
> >> > > > Also, it's more efficient to just send a single "clear the
> group"
> >> > > request vs sending a LeaveGroup
> >> > > > request for every single member. What do you think?
> >> > > > => the comparison is to send a single "clear the group" request
> vs
> >> > > sending a "get members" + a "remove members" request since the
> >> > > adminClient.removeMembersFromConsumerGroup support batch removal.
> We
> >> > > don't need to send lots of leaveGroup requests for every single
> member.
> >> > > >
> >> > > > I can understand your point, but I think we could reuse
> the
> >> > > current
> >> > > > adminClient.removeMembersFromConsumerGroup interface
> effectively with
> >> > > the KIP.
> >> > > > What do you think?
> >> > > >
> >> > > > Thanks!
> >> > > >
> >> > > > Feyman
> >> > > >
> >> > > >
> >> > > >
> ------------------------------------------------------------------
> >> > > > 发件人：Sophie Blee-Goldman <so...@confluent.io>
> >> > > > 发送时间：2020年3月10日(星期二) 03:02
> >> > > > 收件人：dev <de...@kafka.apache.org>; feyman2009 <
> feyman2009@aliyun.com>
> >> > > > 主 题：Re: 回复：回复：[Vote] KIP-571: Add option to force remove
> >> > > members in StreamsResetter
> >> > > >
> >> > > > Hey Feyman,
> >> > > >
> >> > > > 1) Regarding point 2 in your last email, if I understand
> correctly you
> >> > > propose to change
> >> > > > the current behavior of the reset tool when --force is not
> specified,
> >> > > and wait for (up to)
> >> > > > the session timeout for all members to be removed. I'm not sure
> we
> >> > > should change this,
> >> > > > especially now that we have a better way to handle the case
> when the
> >> > > group is not empty:
> >> > > > we should continue to throw an exception and fail fast, but can
> print
> >> > > a message suggesting
> >> > > > to use the new --force option to remove remaining group
> members. Why
> >> > > make users wait
> >> > > > for the session timeout when we've just added a new feature
> that means
> >> > > they don't have to?
> >> > > >
> >> > > > 2) Regarding Matthias' question:
> >> > > >
> >> > > >> I am really wondering, if for a static group, we should allow
> users
> >> > > toremove individual members? For a dynamic group this feature
> would not
> >> > > > make much sense IMHO, because the `memberId` is not know by the
> user.
> >> > > >
> >> > > > I think his point is similar to what I was trying to get at
> earlier,
> >> > > with the proposal to add a new
> >> > > > #removeAllMembers API rather than an API to remove individual
> members
> >> > > according to their
> >> > > > memberId. As he explained, removing based on memberId is likely
> not
> >> > > that useful in general.
> >> > > > Also, it's not actually what we want to do here; maybe we
> should avoid
> >> > > adding a new API
> >> > > > that we think may be useful in other contexts (remove individual
> >> > > member based on memberId),
> >> > > > and just add the API we actually need (remove all members from
> group)
> >> > > in this KIP? We can
> >> > > > always add the "remove individual member by memberId" API at a
> later
> >> > > point, if it turns out to
> >> > > > actually be requested for specific reasons?
> >> > > >
> >> > > > Also, it's more efficient to just send a single "clear the
> group"
> >> > > request vs sending a LeaveGroup
> >> > > > request for every single member. What do you think?
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Sat, Mar 7, 2020 at 1:41 AM feyman2009
> >> > > <fe...@aliyun.com.invalid> wrote:
> >> > > > Hi, Matthias
> >> > > > Thanks, I updated the KIP to mention the deprecated and
> newly
> >> > > added methods.
> >> > > >
> >> > > > 1. What happens is `groupInstanceId` is used for a dynamic
> group? What
> >> > > > happens if both parameters are specified? What happens if
> `memberId`
> >> > > > is specified for a static group?
> >> > > >
> >> > > > => my understanding is that the dynamic/static membership is
> member
> >> > > level other than group level, and I think above questions could be
> >> > > answered by the "Leave Group Logic Change" section in KIP-345:
> >> > >
> >> > >
> >> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> >> > > ,
> >> > > this KIP stays consistent with KIP-345.
> >> > > >
> >> > > > 2. About the `--force` option. Currently, StreamsResetter
> fails with
> >> > an
> >> > > > error if the consumer group is not empty. You state in your
> KIP that:
> >> > > >
> >> > > > > without --force, we need to wait for session timeout.
> >> > > >
> >> > > > Is this an intended behavior change if `--force` is not used
> or is the
> >> > > > KIP description incorrect?
> >> > > >
> >> > > > => This is the intended behavior. For this part, I think there
> are
> >> > > two ways to go:
> >> > > > 1) (the implicit way) Not introducing the new "--force"
> option, with
> >> > > this KIP, StreamsResetter will by default remove active
> members(with
> >> > > long session timeout configured) on broker side
> >> > > > 2) (the explicit way) Introduce the new "--force" option,
> users need
> >> > > to explicitly specify --force to remove the active members. If
> --force
> >> > > not specified, the StreamsResetter behaviour is as previous
> versions'.
> >> > > >
> >> > > > I think the two alternatives above are both feasible,
> personally I
> >> > > prefer way 2.
> >> > > >
> >> > > > 3. For my own understanding: with the `--force` option, we
> intend to
> >> > get
> >> > > > all `memberIds` and send a "remove member" request for each
> with
> >> > > > corresponding `memberId` to remove the member from the group
> >> > > > (independent is the group is static or dynamic)?
> >> > > >
> >> > > > => Yeah, minor thing to mention is we will send the "remove
> member"
> >> > > request for each member(could be dynamic member or static member)
> to
> >> > > remove them from group
> >> > > > for dynamic members, both "group.instance.id" and "member.id"
> will be
> >> > > specified
> >> > > > for dynamic members, only "member.id" will be specified
> >> > > >
> >> > > > 4. I am really wondering, if for a static group, we should
> allow users
> >> > > to
> >> > > > remove individual members? For a dynamic group this feature
> would not
> >> > > > make much sense IMHO, because the `memberId` is not know by
> the user.
> >> > > >
> >> > > > => KIP-345 introduced the batch removal feature for both static
> >> > > member and dynamic member, my understanding is that "allow users
> to
> >> > > > remove individual members" could be useful for rolling bounce
> and
> >> > > scale down accoding to KIP-345. KafkaAdminClient currently only
> support
> >> > > static member removal and this KIP-571 enables dynamic member
> removal
> >> > > for it, which is also consistent with the broker side logic.
> Users could
> >> > > get the member.id (and group.instance.id if for static member) by
> >> > > adminClient.describeConsumerGroups.
> >> > > >
> >> > > > Furthermore, I don't have an assumption that a consumer group
> should
> >> > > contain only static or dynamic members, and I think KIP-345 and
> this KIP
> >> > > don't need to be based on this assumption.
> >> > > > You could correct me if I have the wrong understanding :)
> >> > > >
> >> > > > Thanks!
> >> > > > Feyman
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> ------------------------------------------------------------------
> >> > > > 发件人：Matthias J. Sax <mj...@apache.org>
> >> > > > 发送时间：2020年3月6日(星期五) 02:20
> >> > > > 收件人：dev <de...@kafka.apache.org>
> >> > > > 主 题：Re: 回复：回复：[Vote] KIP-571: Add option to force remove
> >> > > members in StreamsResetter
> >> > > >
> >> > > > Feyman,
> >> > > >
> >> > > > thanks a lot for the KIP. Overall LGTM. I have a few more
> comment and
> >> > > > questions (sorry for the late reply):
> >> > > >
> >> > > >
> >> > > > The KIP mentions that some constructors will be deprecated.
> Those
> >> > should
> >> > > > be listed explicitly. For example:
> >> > > >
> >> > > >
> >> > > > public class MemberToRemove {
> >> > > >
> >> > > > // deprecated methods
> >> > > >
> >> > > > @Deprecated
> >> > > > public MemberToRemove(String groupInstanceId);
> >> > > >
> >> > > > // new methods
> >> > > >
> >> > > > public MemberToRemove()
> >> > > >
> >> > > > public MemberToRemove withGroupInstanceId(String
> groupInstanceId)
> >> > > >
> >> > > > public MemberToRemove withMemberId(String memberId)
> >> > > > }
> >> > > >
> >> > > > What happens is `groupInstanceId` is used for a dynamic group?
> What
> >> > > > happens if both parameters are specified? What happens if
> `memberId`
> >> > > > is specified for a static group?
> >> > > >
> >> > > >
> >> > > > About the `--force` option. Currently, StreamsResetter fails
> with an
> >> > > > error if the consumer group is not empty. You state in your KIP
> that:
> >> > > >
> >> > > >> without --force, we need to wait for session timeout.
> >> > > >
> >> > > > Is this an intended behavior change if `--force` is not used or
> is the
> >> > > > KIP description incorrect?
> >> > > >
> >> > > > For my own understanding: with the `--force` option, we intend
> to get
> >> > > > all `memberIds` and send a "remove member" request for each with
> >> > > > corresponding `memberId` to remove the member from the group
> >> > > > (independent is the group is static or dynamic)?
> >> > > >
> >> > > > I am really wondering, if for a static group, we should allow
> users to
> >> > > > remove individual members? For a dynamic group this feature
> would not
> >> > > > make much sense IMHO, because the `memberId` is not know by the
> user.
> >> > > >
> >> > > >
> >> > > >
> >> > > > -Matthias
> >> > > >
> >> > > >
> >> > > > On 3/5/20 7:15 AM, Bill Bejeck wrote:
> >> > > >> Thanks for the KIP. +1 (binding).
> >> > > >
> >> > > >> -Bill
> >> > > >
> >> > > >
> >> > > >
> >> > > >> On Wed, Mar 4, 2020 at 12:40 AM Guozhang Wang <
> wangguoz@gmail.com>
> >> > > >> wrote:
> >> > > >
> >> > > >>> Thanks, +1 from me (binding).
> >> > > >>>
> >> > > >>> On Tue, Mar 3, 2020 at 9:39 PM feyman2009 <
> feyman2009@aliyun.com>
> >> > > >>> wrote:
> >> > > >>>
> >> > > >>>> Hi, Guozhang Thanks a lot for the advice, that make sense! I
> >> > > >>>> have updated the KIP page with the operational steps of
> >> > > >>>> StreamsResetter.
> >> > > >>>>
> >> > > >>>> Thanks! Feyman
> >> > > >>>>
> >> > > >>>>
> ------------------------------------------------------------------
> >> > > >>>>
> >> > > >>>>
> >> > > > 发件人：Guozhang Wang <wa...@gmail.com>
> >> > > >>>> 发送时间：2020年3月3日(星期二) 14:22 收件人：dev <de...@kafka.apache.org>;
> >> > > >>>> feyman2009 <fe...@aliyun.com> 主 题：Re: 回复：回复：[Vote]
> >> > > >>>> KIP-571: Add option to force remove members in
> StreamsResetter
> >> > > >>>>
> >> > > >>>> Hello Feyman, thanks for the proposal!
> >> > > >>>>
> >> > > >>>> I read through the doc and overall it looks good to me.
> >> > > >>>>
> >> > > >>>> One minor thing I'd still like to point out is that, the
> >> > > >>>> "removeMembersFromConsumerGroup" only sends a leave-group
> >> > > >>>> request to the coordinator to let it remove the member,
> >> > > >>>> however, if the member is still there alive and running then
> it
> >> > > >>>> would soon be notified that it is no
> >> > > >>> longer
> >> > > >>>> a legal member of the group via heartbeats, and then
> >> > > >>>> automatically tries
> >> > > >>> to
> >> > > >>>> re-join the group. So on the operational side, it is still
> >> > > >>>> required that the following steps:
> >> > > >>>>
> >> > > >>>> 1) first stop the consumers (of streams instances), wait
> until
> >> > > >>>> the shutdown is complete. 2) then use admin client in case
> the
> >> > > >>>> stopped consumers are still registered at the broker side and
> >> > > >>>> we do not want to wait for session timeout.
> >> > > >>>>
> >> > > >>>> Even with this KIP, people should still not skip step 1)
> above,
> >> > > >>>> since otherwise the consumers would re-connect and re-join
> the
> >> > > >>>> group
> >> > > >>> immediately
> >> > > >>>> still.
> >> > > >>>>
> >> > > >>>> In your doc you've already mentioned "Furthermore, users
> should
> >> > > >>>> make sure all the stream applications are shutdown when
> running
> >> > > >>>> StreamsResetter
> >> > > >>> with
> >> > > >>>> --force, otherwise it might trigger unexpected rebalance. "
> >> > > >>>> What I'd want to clarify is that no matter if "--force"
> option
> >> > > >>>> is enabled, this is
> >> > > >>> always
> >> > > >>>> the case that users should shutdown the streams instances
> >> > > >>>> first, and then use the streams resetter :)
> >> > > >>>>
> >> > > >>>> As long as that is clarified in the proposal documentation,
> I'm
> >> > > >>>> +1 on
> >> > > >>> this
> >> > > >>>> KIP.
> >> > > >>>>
> >> > > >>>>
> >> > > >>>> Thanks again for the contribution, Guozhang
> >> > > >>>>
> >> > > >>>>
> >> > > >>>> On Mon, Mar 2, 2020 at 6:31 AM feyman2009
> >> > > >>>> <feyman2009@aliyun.com.invalid
> >> > > >>>>
> >> > > >>>> wrote: Hi, John Sorry, I have mistaken the KIP approval
> >> > > >>>> standard, anyway, I will
> >> > > >>> start
> >> > > >>>> the PR soon and waiting for more binding approvals.
> >> > > >>>>
> >> > > >>>> Thanks! Feyman
> >> > > >>>>
> >> > > >>>>
> >> > > >>>>
> ------------------------------------------------------------------
> >> > > >>>>
> >> > > >>>>
> >> > > > 发件人：John Roesler <vv...@apache.org>
> >> > > >>>> 发送时间：2020年3月2日(星期一) 22:00 收件人：dev
> >> > > > <de...@kafka.apache.org> 主
> >> > > >>>> 题：Re: 回复：回复：[Vote] KIP-571: Add option to force remove
> members
> >> > > >>>> in StreamsResetter
> >> > > >>>>
> >> > > >>>> Hi Feyman,
> >> > > >>>>
> >> > > >>>> Sorry, but we actually need 3 binding votes for the KIP to
> >> > > >>>> pass. Please feel free to keep bumping the thread until some
> >> > > >>>> more committers can take
> >> > > >>> a
> >> > > >>>> look.
> >> > > >>>>
> >> > > >>>> By the way, you can totally start a PR, but we can’t merge it
> >> > > >>>> until the KIP passes the vote.
> >> > > >>>>
> >> > > >>>> Thanks! John
> >> > > >>>>
> >> > > >>>> On Mon, Mar 2, 2020, at 00:24, feyman2009 wrote:
> >> > > >>>>> Hi,all Since currently we have 1 binding and two non-binding
> >> > > >>>>> +1, I will update the KIP-571 as adopted and initiate a PR
> >> > > >>>>> shortly
> >> > > >>>>>
> >> > > >>>>> Thanks! Feyman
> >> > > >>>>>
> >> > > >>>>>
> >> > > >>>>>
> ------------------------------------------------------------------
> >> > > >>>>>
> >> > > >>>>>
> >> > > > 发件人：Sophie Blee-Goldman <so...@confluent.io>
> >> > > >>>>> 发送时间：2020年2月28日(星期五) 10:17 收件人：dev
> >> > > > <de...@kafka.apache.org> 主
> >> > > >>>>> 题：Re: 回复：[Vote] KIP-571: Add option to force remove members
> >> > > >>>>> in
> >> > > >>>> StreamsResetter
> >> > > >>>>>
> >> > > >>>>> Thanks for the KIP, +1 (non-binding)
> >> > > >>>>>
> >> > > >>>>> On Thu, Feb 27, 2020 at 12:40 PM Boyang Chen <
> >> > > >>> reluctanthero104@gmail.com
> >> > > >>>>>
> >> > > >>>>> wrote:
> >> > > >>>>>
> >> > > >>>>>> Thanks Feyman, +1 (non-binding)
> >> > > >>>>>>
> >> > > >>>>>> On Thu, Feb 27, 2020 at 9:25 AM John Roesler
> >> > > >>>>>> <vv...@apache.org>
> >> > > >>>> wrote:
> >> > > >>>>>>
> >> > > >>>>>>> Thanks for the proposal!
> >> > > >>>>>>>
> >> > > >>>>>>> I'm +1 (binding) -John
> >> > > >>>>>>>
> >> > > >>>>>>> On Wed, Feb 26, 2020, at 19:41, feyman2009 wrote:
> >> > > >>>>>>>> Updated with the KIP link:
> >> > > >>>>>>>>
> >> > > >>>>>>>
> >> > > >>>>>>
> >> > > >>>>
> >> > > >>>
> >> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-571%3A+Add+opti
> >> > > > on+to+force+remove+members+in+StreamsResetter
> >> > > >>>>>>>>
> >> > > >>>>>>>>
> >> > > >>>>>>>>
> >> > > >>>
> >> > > >>>
> >> > > >
> ------------------------------------------------------------------
> >> > > >>>>>>>> 发件人：feyman2009 <fe...@aliyun.com.INVALID> 发送时
> >> > > >>>>>>>> 间：2020年2月27日(星期四) 09:38 收件人：dev <de...@kafka.apache.org>
> >> > > >>>>>>>> 主 题：[Vote] KIP-571: Add option to force remove members
> >> > > >>>>>>>> in
> >> > > >>>>>> StreamsResetter
> >> > > >>>>>>>>
> >> > > >>>>>>>>
> >> > > >>>>>>>> Hi, all I would like to start a vote on KIP-571: Add
> >> > > >>>>>>>> option to force
> >> > > >>>> remove
> >> > > >>>>>>>> members in StreamsResetter .
> >> > > >>>>>>>>
> >> > > >>>>>>>> Thanks! Feyman
> >> > > >>>>>>>>
> >> > > >>>>>>>>
> >> > > >>>>>>>
> >> > > >>>>>>
> >> > > >>>>>
> >> > > >>>>>
> >> > > >>>>
> >> > > >>>>
> >> > > >>>>
> >> > > >>>> -- -- Guozhang
> >> > > >>>>
> >> > > >>>>
> >> > > >>>>
> >> > > >>>
> >> > > >>> -- -- Guozhang
> >> > > >>>
> >> > > >
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> >
> >> > --
> >> > -- Guozhang
> >> >
> >>
> >>
> >>
> >>
> >>
> >
> >
>
>
Re: 回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in StreamsResetter
Posted by Boyang Chen <re...@gmail.com>.
You should already get enough votes if I'm counting correctly (Guozhang,
John, Matthias)
On Tue, Apr 7, 2020 at 6:59 PM feyman2009 <fe...@aliyun.com.invalid>
wrote:
> Hi, Boyang&Matthias
> I think Matthias's proposal makes sense, but we can use the admin tool
> for this scenario as Boyang mentioned or follow up later, so I prefer to
> keep this KIP unchanged to minimize the scope.
> Calling for vote ~
>
> Thanks!
> Feyman
>
> ------------------------------------------------------------------
> 发件人：Boyang Chen <re...@gmail.com>
> 发送时间：2020年4月8日(星期三) 02:15
> 收件人：dev <de...@kafka.apache.org>
> 主 题：Re: 回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members
> in StreamsResetter
>
> Hey Feyman,
>
> I think Matthias' suggestion is optional, and we could just use admin tool
> to remove single static members as well.
>
> Boyang
>
> On Tue, Apr 7, 2020 at 11:00 AM Matthias J. Sax <mj...@apache.org> wrote:
>
> > > Would you mind to elaborate why we still need that
> >
> > Sure.
> >
> > For static memership, the session timeout it usually set quite high.
> > This make scaling in an application tricky: if you shut down one
> > instance, no rebalance would happen until `session.timeout.ms` hits.
> > This is specific to Kafka Streams, because when a Kafka Stream client is
> > closed, it does _not_ send a `LeaveGroupRequest`. Hence, the
> > corresponding partitions would not be processed for a long time and
> > thus, fall back.
> >
> > Given that each instance will have a unique `instance.id` provided by
> > the user, we could allow users to remove the instance they want to
> > decommission from the consumer group without the need to wait for
> > `session.timeout.ms`.
> >
> > Hence, it's not an application reset scenario for which one wants to
> > remove all members, but a scaling-in scenario. For dynamic membership,
> > this issue usually does not occur because the `session.timeout.ms` is
> > set to a fairly low value and a rebalance would happen quickly after an
> > instance is decommissioned.
> >
> > Does this make sense?
> >
> > As said before, we may or may not include this in this KIP. It's up to
> > you if you want to address it or not.
> >
> >
> > -Matthias
> >
> >
> >
> > On 4/7/20 7:12 AM, feyman2009 wrote:
> > > Hi, Matthias
> > > Thanks a lot!
> > > So you do not plan so support removing a _single static_ member via
> > `StreamsResetter`?
> > > =>
> > > Would you mind to elaborate why we still need that if we are
> > able to batch remove active members with adminClient?
> > >
> > > Thanks!
> > >
> > > Feyman
> > > ------------------------------------------------------------------
> > > 发件人：Matthias J. Sax <mj...@apache.org>
> > > 发送时间：2020年4月7日(星期二) 08:25
> > > 收件人：dev <de...@kafka.apache.org>
> > > 主 题：Re: 回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members
> > in StreamsResetter
> > >
> > > Overall LGTM.
> > >
> > > +1 (binding)
> > >
> > > So you do not plan so support removing a _single static_ member via
> > > `StreamsResetter`? We can of course still add this as a follow up but
> it
> > > might be nice to just add it to this KIP right away. Up to you if you
> > > want to include it or not.
> > >
> > >
> > > -Matthias
> > >
> > >
> > >
> > > On 3/30/20 8:16 AM, feyman2009 wrote:
> > >> Hi, Boyang
> > >> Thanks a lot, that make sense, we should not expose member.id in
> > the MemberToRemove anymore, I have fixed it in the KIP.
> > >>
> > >>
> > >> Feyman
> > >> ------------------------------------------------------------------
> > >> 发件人：Boyang Chen <re...@gmail.com>
> > >> 发送时间：2020年3月29日(星期日) 01:45
> > >> 收件人：dev <de...@kafka.apache.org>; feyman2009 <fe...@aliyun.com>
> > >> 主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove members in
> > StreamsResetter
> > >>
> > >> Hey Feyman,
> > >>
> > >> thanks for the update. I assume we would rely entirely on the internal
> > changes for `removeMemberFromGroup` by sending a DescribeGroup request
> > first. With that in mind, I don't think we need to add member.id to
> > MemberToRemove anymore, as it is only facing public where users will only
> > configure group.instance.id?
> > >> On Fri, Mar 27, 2020 at 5:04 PM feyman2009
> > <fe...@aliyun.com.invalid> wrote:
> > >> Bump, can anyone kindly take a look at the updated KIP-571? Thanks!
> > >>
> > >>
> > >> ------------------------------------------------------------------
> > >> 发件人：feyman2009 <fe...@aliyun.com.INVALID>
> > >> 发送时间：2020年3月23日(星期一) 08:51
> > >> 收件人：dev <de...@kafka.apache.org>
> > >> 主 题：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in
> > StreamsResetter
> > >>
> > >> Hi, team
> > >> I have updated the KIP-571 according to our latest discussion
> > results, would you mind to take a look? Thanks!
> > >>
> > >> Feyman
> > >>
> > >>
> > >> ------------------------------------------------------------------
> > >> 发件人：Boyang Chen <re...@gmail.com>
> > >> 发送时间：2020年3月19日(星期四) 13:41
> > >> 收件人：dev <de...@kafka.apache.org>; feyman2009 <fe...@aliyun.com>
> > >> 主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove members
> in
> > StreamsResetter
> > >>
> > >> Thanks for the insight Feyman. I personally feel adding another admin
> > client command is redundant, so picking option 1). The memberInfos struct
> > is internal and just used for result reference purposes. I think it could
> > still work even we overload with `removeAll` option, if that makes sense.
> > >>
> > >> Boyang
> > >> On Wed, Mar 18, 2020 at 8:51 PM feyman2009
> > <fe...@aliyun.com.invalid> wrote:
> > >> Hi, team
> > >> Before going too far on the KIP update, I would like to hear
> your
> > opinions about how we would change the interface of AdminClient, the two
> > alternatives I could think of are:
> > >> 1) Extend adminClient.removeMembersFromConsumerGroup to support
> > remove all
> > >> As Guochang suggested, we could add some flag param in
> > RemoveMembersFromConsumerGroupOptions to indicated the "remove all"
> logic.
> > >> 2) Add a new API like
> > adminClient.removeAllMembersFromConsumerGroup(groupId, options)
> > >>
> > >> I think 1) will be more compact from the API perspective, but
> > looking at the code, I found that the if we are going to remove all, then
> > the RemoveMembersFromConsumerGroupResult#memberInfos/memberResult()/all()
> > should be changed accordingly, and they seem not that meaningful under
> the
> > "remove all" scenario.
> > >>
> > >> A minor thought about the
> > adminClient.removeMembersFromConsumerGroup API is:
> > >> Looking at some other deleteXX APIs, like deleteTopics,
> > deleteRecords, the results contains only a Map<A, Future<B>>, I think
> it's
> > enough to describe the related results, is it make sense that we may
> remove
> > memberInfos in RemoveMembersFromConsumerGroupResult ? This KIP has no
> > dependency on this if we choose alternative 2)
> > >>
> > >> Could you advise? Thanks!
> > >>
> > >> Feyman
> > >>
> > >>
> > >> 送时间：2020年3月15日(星期日) 10:11
> > >> 收件人：dev <de...@kafka.apache.org>
> > >> 主 题：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members
> in
> > StreamsResetter
> > >>
> > >> Hi, all
> > >> Thanks a lot for your feedback!
> > >> According to the discussion, it seems we don't have some valid
> > use cases for removing specific dynamic members, I think it makes sense
> to
> > encapsulate the "get and delete" logic in adminClient. I will update the
> > KIP shortly!
> > >>
> > >> Thanks!
> > >>
> > >> Feyman
> > >>
> > >>
> > >> ------------------------------------------------------------------
> > >> 发件人：Boyang Chen <re...@gmail.com>
> > >> 发送时间：2020年3月14日(星期六) 00:39
> > >> 收件人：dev <de...@kafka.apache.org>
> > >> 主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove members
> > in StreamsResetter
> > >>
> > >> Thanks Matthias and Guozhang for the feedback. I'm not worrying too
> > much
> > >> about the member.id exposure as we have done so in a couple of
> > areas. As
> > >> for the recommended admin client change, I think it makes sense in
> an
> > >> encapsulation perspective. Maybe I'm still a bit hesitant as we are
> > losing
> > >> the flexibility of closing only a subset of `dynamic members`
> > potentially,
> > >> but we could always get back and address it if some user feels
> > necessary to
> > >> have it.
> > >>
> > >> My short answer would be, LGTM :)
> > >>
> > >> Boyang
> > >>
> > >> On Thu, Mar 12, 2020 at 5:26 PM Guozhang Wang <wa...@gmail.com>
> > wrote:
> > >>
> > >> > Hi Matthias,
> > >> >
> > >> > About the AdminClient param API: that's a great point here. I
> think
> > overall
> > >> > if users want to just "remove all members" they should not need to
> > first
> > >> > get all the member.ids themselves, but instead internally the
> admin
> > client
> > >> > can first issue a describe-group request to get all the
> member.ids,
> > and
> > >> > then use them in the next issued leave-group request, all
> > abstracted away
> > >> > from the users. With that in mind, maybe in
> > >> > RemoveMembersFromConsumerGroupOptions we can just introduce an
> > overloaded
> > >> > flag param besides "members" that indicate "remove all"?
> > >> >
> > >> > Guozhang
> > >> >
> > >> > On Thu, Mar 12, 2020 at 2:59 PM Matthias J. Sax <mjsax@apache.org
> >
> > wrote:
> > >> >
> > >> > > Feyman,
> > >> > >
> > >> > > some more comments/questions:
> > >> > >
> > >> > > The description of `LeaveGroupRequest` is clear but it's unclear
> > how
> > >> > > `MemberToRemove` should behave. Which parameter is required?
> > Which is
> > >> > > optional? What is the relationship between both.
> > >> > >
> > >> > > The `LeaveGroupRequest` description clearly states that
> > specifying a
> > >> > > `memberId` is optional if the `groupInstanceId` is provided. If
> > >> > > `MemberToRemove` applies the same pattern, it must be explicitly
> > defined
> > >> > > in the KIP (and explained in the JavaDocs of `MemberToRemove`)
> > because
> > >> > > we cannot expect that an admin-client users knows that
> internally
> > a
> > >> > > `LeaveGroupRequest` is used nor what the semantics of a
> > >> > > `LeaveGroupRequest` are.
> > >> > >
> > >> > >
> > >> > > About Admin API:
> > >> > >
> > >> > > In general, I am also confused that we allow so specify a
> > `memberId` at
> > >> > > all, because the `memberId` is an internal id that is not really
> > exposed
> > >> > > to the user. Hence, from a AdminClient point of view, accepting
> a
> > >> > > `memberId` as input seems questionable to me? Of course,
> > `memberId` can
> > >> > > be collected via `describeConsumerGroups()` but it will return
> the
> > >> > > `memberId` of _all_ consumer in the group and thus how would a
> > user know
> > >> > > which member should be removed for a dynamic group (if an
> > individual
> > >> > > member should be removed)?
> > >> > >
> > >> > > Hence, how can any user get to know the `memberId` of an
> > individual
> > >> > > client in a programtic way?
> > >> > >
> > >> > > Also I am wondering in general, why the removal of single
> dynamic
> > member
> > >> > > is important? In general, I would expect a short
> > `session.timeout` for
> > >> > > dynamic groups and thus removing a specific member from the
> group
> > seems
> > >> > > not to be an important feature -- for static groups we expect a
> > long
> > >> > > `session.timeout` and a user can also identify individual
> clients
> > via
> > >> > > `groupInstandId`, hence the feature makes sense for this case
> and
> > is
> > >> > > straight forward to use.
> > >> > >
> > >> > >
> > >> > > About StreamsResetter:
> > >> > >
> > >> > > For this case we just say "remove all members" and thus the
> > >> > > `describeConsumerGroup` approach works. However, it seems to be
> a
> > >> > > special case?
> > >> > >
> > >> > > Or, if we expected that the "remove all members" use case is the
> > norm,
> > >> > > why can't we make a change admin-client to directly accept a `
> > group.id`?
> > >> > > The admin-client can internal first do a `DescribeGroupRequest`
> > and
> > >> > > afterward corresponding `LeaveGroupRequest` -- i.e., instead of
> > building
> > >> > > this pattern in `StreamsResetter` we build it directly into
> > >> > `AdminClient`.
> > >> > >
> > >> > > Last, for static group the main use case seems to be to remove
> an
> > >> > > individual member from the group but this feature is not covered
> > by the
> > >> > > KIP: I think using `--force` to remove all members makes sense,
> > but an
> > >> > > important second feature to remove an individual static member
> > would
> > >> > > require it's own flag to specify a single `group.instance.id`.
> > >> > >
> > >> > >
> > >> > > Thoughts?
> > >> > >
> > >> > >
> > >> > > -Matthias
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > On 3/11/20 8:43 PM, feyman2009 wrote:
> > >> > > > Hi, Sophie
> > >> > > > For 1) Sorry, I found that my expression is kind of
> > misleading,
> > >> > > what I actually mean is: "if --force not specified, an exception
> > saying
> > >> > > there are still active members on broker side will be thrown and
> > >> > > suggesting using StreamsResetter with --force", I just updated
> > the KIP
> > >> > > page.
> > >> > > >
> > >> > > > For 2)
> > >> > > > I may also had some misleading expression previous, to
> > clarify
> > >> > :
> > >> > > >
> > >> > > > Also, it's more efficient to just send a single "clear the
> > group"
> > >> > > request vs sending a LeaveGroup
> > >> > > > request for every single member. What do you think?
> > >> > > > => the comparison is to send a single "clear the group"
> request
> > vs
> > >> > > sending a "get members" + a "remove members" request since the
> > >> > > adminClient.removeMembersFromConsumerGroup support batch
> removal.
> > We
> > >> > > don't need to send lots of leaveGroup requests for every single
> > member.
> > >> > > >
> > >> > > > I can understand your point, but I think we could reuse
> > the
> > >> > > current
> > >> > > > adminClient.removeMembersFromConsumerGroup interface
> > effectively with
> > >> > > the KIP.
> > >> > > > What do you think?
> > >> > > >
> > >> > > > Thanks!
> > >> > > >
> > >> > > > Feyman
> > >> > > >
> > >> > > >
> > >> > > >
> > ------------------------------------------------------------------
> > >> > > > 发件人：Sophie Blee-Goldman <so...@confluent.io>
> > >> > > > 发送时间：2020年3月10日(星期二) 03:02
> > >> > > > 收件人：dev <de...@kafka.apache.org>; feyman2009 <
> > feyman2009@aliyun.com>
> > >> > > > 主 题：Re: 回复：回复：[Vote] KIP-571: Add option to force remove
> > >> > > members in StreamsResetter
> > >> > > >
> > >> > > > Hey Feyman,
> > >> > > >
> > >> > > > 1) Regarding point 2 in your last email, if I understand
> > correctly you
> > >> > > propose to change
> > >> > > > the current behavior of the reset tool when --force is not
> > specified,
> > >> > > and wait for (up to)
> > >> > > > the session timeout for all members to be removed. I'm not
> sure
> > we
> > >> > > should change this,
> > >> > > > especially now that we have a better way to handle the case
> > when the
> > >> > > group is not empty:
> > >> > > > we should continue to throw an exception and fail fast, but
> can
> > print
> > >> > > a message suggesting
> > >> > > > to use the new --force option to remove remaining group
> > members. Why
> > >> > > make users wait
> > >> > > > for the session timeout when we've just added a new feature
> > that means
> > >> > > they don't have to?
> > >> > > >
> > >> > > > 2) Regarding Matthias' question:
> > >> > > >
> > >> > > >> I am really wondering, if for a static group, we should allow
> > users
> > >> > > toremove individual members? For a dynamic group this feature
> > would not
> > >> > > > make much sense IMHO, because the `memberId` is not know by
> the
> > user.
> > >> > > >
> > >> > > > I think his point is similar to what I was trying to get at
> > earlier,
> > >> > > with the proposal to add a new
> > >> > > > #removeAllMembers API rather than an API to remove individual
> > members
> > >> > > according to their
> > >> > > > memberId. As he explained, removing based on memberId is
> likely
> > not
> > >> > > that useful in general.
> > >> > > > Also, it's not actually what we want to do here; maybe we
> > should avoid
> > >> > > adding a new API
> > >> > > > that we think may be useful in other contexts (remove
> individual
> > >> > > member based on memberId),
> > >> > > > and just add the API we actually need (remove all members from
> > group)
> > >> > > in this KIP? We can
> > >> > > > always add the "remove individual member by memberId" API at a
> > later
> > >> > > point, if it turns out to
> > >> > > > actually be requested for specific reasons?
> > >> > > >
> > >> > > > Also, it's more efficient to just send a single "clear the
> > group"
> > >> > > request vs sending a LeaveGroup
> > >> > > > request for every single member. What do you think?
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On Sat, Mar 7, 2020 at 1:41 AM feyman2009
> > >> > > <fe...@aliyun.com.invalid> wrote:
> > >> > > > Hi, Matthias
> > >> > > > Thanks, I updated the KIP to mention the deprecated and
> > newly
> > >> > > added methods.
> > >> > > >
> > >> > > > 1. What happens is `groupInstanceId` is used for a dynamic
> > group? What
> > >> > > > happens if both parameters are specified? What happens if
> > `memberId`
> > >> > > > is specified for a static group?
> > >> > > >
> > >> > > > => my understanding is that the dynamic/static membership is
> > member
> > >> > > level other than group level, and I think above questions could
> be
> > >> > > answered by the "Leave Group Logic Change" section in KIP-345:
> > >> > >
> > >> > >
> > >> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> > >> > > ,
> > >> > > this KIP stays consistent with KIP-345.
> > >> > > >
> > >> > > > 2. About the `--force` option. Currently, StreamsResetter
> > fails with
> > >> > an
> > >> > > > error if the consumer group is not empty. You state in your
> > KIP that:
> > >> > > >
> > >> > > > > without --force, we need to wait for session timeout.
> > >> > > >
> > >> > > > Is this an intended behavior change if `--force` is not used
> > or is the
> > >> > > > KIP description incorrect?
> > >> > > >
> > >> > > > => This is the intended behavior. For this part, I think
> there
> > are
> > >> > > two ways to go:
> > >> > > > 1) (the implicit way) Not introducing the new "--force"
> > option, with
> > >> > > this KIP, StreamsResetter will by default remove active
> > members(with
> > >> > > long session timeout configured) on broker side
> > >> > > > 2) (the explicit way) Introduce the new "--force" option,
> > users need
> > >> > > to explicitly specify --force to remove the active members. If
> > --force
> > >> > > not specified, the StreamsResetter behaviour is as previous
> > versions'.
> > >> > > >
> > >> > > > I think the two alternatives above are both feasible,
> > personally I
> > >> > > prefer way 2.
> > >> > > >
> > >> > > > 3. For my own understanding: with the `--force` option, we
> > intend to
> > >> > get
> > >> > > > all `memberIds` and send a "remove member" request for each
> > with
> > >> > > > corresponding `memberId` to remove the member from the group
> > >> > > > (independent is the group is static or dynamic)?
> > >> > > >
> > >> > > > => Yeah, minor thing to mention is we will send the "remove
> > member"
> > >> > > request for each member(could be dynamic member or static
> member)
> > to
> > >> > > remove them from group
> > >> > > > for dynamic members, both "group.instance.id" and "member.id
> "
> > will be
> > >> > > specified
> > >> > > > for dynamic members, only "member.id" will be specified
> > >> > > >
> > >> > > > 4. I am really wondering, if for a static group, we should
> > allow users
> > >> > > to
> > >> > > > remove individual members? For a dynamic group this feature
> > would not
> > >> > > > make much sense IMHO, because the `memberId` is not know by
> > the user.
> > >> > > >
> > >> > > > => KIP-345 introduced the batch removal feature for both
> static
> > >> > > member and dynamic member, my understanding is that "allow users
> > to
> > >> > > > remove individual members" could be useful for rolling bounce
> > and
> > >> > > scale down accoding to KIP-345. KafkaAdminClient currently only
> > support
> > >> > > static member removal and this KIP-571 enables dynamic member
> > removal
> > >> > > for it, which is also consistent with the broker side logic.
> > Users could
> > >> > > get the member.id (and group.instance.id if for static member)
> by
> > >> > > adminClient.describeConsumerGroups.
> > >> > > >
> > >> > > > Furthermore, I don't have an assumption that a consumer group
> > should
> > >> > > contain only static or dynamic members, and I think KIP-345 and
> > this KIP
> > >> > > don't need to be based on this assumption.
> > >> > > > You could correct me if I have the wrong understanding :)
> > >> > > >
> > >> > > > Thanks!
> > >> > > > Feyman
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > ------------------------------------------------------------------
> > >> > > > 发件人：Matthias J. Sax <mj...@apache.org>
> > >> > > > 发送时间：2020年3月6日(星期五) 02:20
> > >> > > > 收件人：dev <de...@kafka.apache.org>
> > >> > > > 主 题：Re: 回复：回复：[Vote] KIP-571: Add option to force remove
> > >> > > members in StreamsResetter
> > >> > > >
> > >> > > > Feyman,
> > >> > > >
> > >> > > > thanks a lot for the KIP. Overall LGTM. I have a few more
> > comment and
> > >> > > > questions (sorry for the late reply):
> > >> > > >
> > >> > > >
> > >> > > > The KIP mentions that some constructors will be deprecated.
> > Those
> > >> > should
> > >> > > > be listed explicitly. For example:
> > >> > > >
> > >> > > >
> > >> > > > public class MemberToRemove {
> > >> > > >
> > >> > > > // deprecated methods
> > >> > > >
> > >> > > > @Deprecated
> > >> > > > public MemberToRemove(String groupInstanceId);
> > >> > > >
> > >> > > > // new methods
> > >> > > >
> > >> > > > public MemberToRemove()
> > >> > > >
> > >> > > > public MemberToRemove withGroupInstanceId(String
> > groupInstanceId)
> > >> > > >
> > >> > > > public MemberToRemove withMemberId(String memberId)
> > >> > > > }
> > >> > > >
> > >> > > > What happens is `groupInstanceId` is used for a dynamic group?
> > What
> > >> > > > happens if both parameters are specified? What happens if
> > `memberId`
> > >> > > > is specified for a static group?
> > >> > > >
> > >> > > >
> > >> > > > About the `--force` option. Currently, StreamsResetter fails
> > with an
> > >> > > > error if the consumer group is not empty. You state in your
> KIP
> > that:
> > >> > > >
> > >> > > >> without --force, we need to wait for session timeout.
> > >> > > >
> > >> > > > Is this an intended behavior change if `--force` is not used
> or
> > is the
> > >> > > > KIP description incorrect?
> > >> > > >
> > >> > > > For my own understanding: with the `--force` option, we intend
> > to get
> > >> > > > all `memberIds` and send a "remove member" request for each
> with
> > >> > > > corresponding `memberId` to remove the member from the group
> > >> > > > (independent is the group is static or dynamic)?
> > >> > > >
> > >> > > > I am really wondering, if for a static group, we should allow
> > users to
> > >> > > > remove individual members? For a dynamic group this feature
> > would not
> > >> > > > make much sense IMHO, because the `memberId` is not know by
> the
> > user.
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > -Matthias
> > >> > > >
> > >> > > >
> > >> > > > On 3/5/20 7:15 AM, Bill Bejeck wrote:
> > >> > > >> Thanks for the KIP. +1 (binding).
> > >> > > >
> > >> > > >> -Bill
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >> On Wed, Mar 4, 2020 at 12:40 AM Guozhang Wang <
> > wangguoz@gmail.com>
> > >> > > >> wrote:
> > >> > > >
> > >> > > >>> Thanks, +1 from me (binding).
> > >> > > >>>
> > >> > > >>> On Tue, Mar 3, 2020 at 9:39 PM feyman2009 <
> > feyman2009@aliyun.com>
> > >> > > >>> wrote:
> > >> > > >>>
> > >> > > >>>> Hi, Guozhang Thanks a lot for the advice, that make sense!
> I
> > >> > > >>>> have updated the KIP page with the operational steps of
> > >> > > >>>> StreamsResetter.
> > >> > > >>>>
> > >> > > >>>> Thanks! Feyman
> > >> > > >>>>
> > >> > > >>>>
> > ------------------------------------------------------------------
> > >> > > >>>>
> > >> > > >>>>
> > >> > > > 发件人：Guozhang Wang <wa...@gmail.com>
> > >> > > >>>> 发送时间：2020年3月3日(星期二) 14:22 收件人：dev <de...@kafka.apache.org>;
> > >> > > >>>> feyman2009 <fe...@aliyun.com> 主 题：Re: 回复：回复：[Vote]
> > >> > > >>>> KIP-571: Add option to force remove members in
> > StreamsResetter
> > >> > > >>>>
> > >> > > >>>> Hello Feyman, thanks for the proposal!
> > >> > > >>>>
> > >> > > >>>> I read through the doc and overall it looks good to me.
> > >> > > >>>>
> > >> > > >>>> One minor thing I'd still like to point out is that, the
> > >> > > >>>> "removeMembersFromConsumerGroup" only sends a leave-group
> > >> > > >>>> request to the coordinator to let it remove the member,
> > >> > > >>>> however, if the member is still there alive and running
> then
> > it
> > >> > > >>>> would soon be notified that it is no
> > >> > > >>> longer
> > >> > > >>>> a legal member of the group via heartbeats, and then
> > >> > > >>>> automatically tries
> > >> > > >>> to
> > >> > > >>>> re-join the group. So on the operational side, it is still
> > >> > > >>>> required that the following steps:
> > >> > > >>>>
> > >> > > >>>> 1) first stop the consumers (of streams instances), wait
> > until
> > >> > > >>>> the shutdown is complete. 2) then use admin client in case
> > the
> > >> > > >>>> stopped consumers are still registered at the broker side
> and
> > >> > > >>>> we do not want to wait for session timeout.
> > >> > > >>>>
> > >> > > >>>> Even with this KIP, people should still not skip step 1)
> > above,
> > >> > > >>>> since otherwise the consumers would re-connect and re-join
> > the
> > >> > > >>>> group
> > >> > > >>> immediately
> > >> > > >>>> still.
> > >> > > >>>>
> > >> > > >>>> In your doc you've already mentioned "Furthermore, users
> > should
> > >> > > >>>> make sure all the stream applications are shutdown when
> > running
> > >> > > >>>> StreamsResetter
> > >> > > >>> with
> > >> > > >>>> --force, otherwise it might trigger unexpected rebalance. "
> > >> > > >>>> What I'd want to clarify is that no matter if "--force"
> > option
> > >> > > >>>> is enabled, this is
> > >> > > >>> always
> > >> > > >>>> the case that users should shutdown the streams instances
> > >> > > >>>> first, and then use the streams resetter :)
> > >> > > >>>>
> > >> > > >>>> As long as that is clarified in the proposal documentation,
> > I'm
> > >> > > >>>> +1 on
> > >> > > >>> this
> > >> > > >>>> KIP.
> > >> > > >>>>
> > >> > > >>>>
> > >> > > >>>> Thanks again for the contribution, Guozhang
> > >> > > >>>>
> > >> > > >>>>
> > >> > > >>>> On Mon, Mar 2, 2020 at 6:31 AM feyman2009
> > >> > > >>>> <feyman2009@aliyun.com.invalid
> > >> > > >>>>
> > >> > > >>>> wrote: Hi, John Sorry, I have mistaken the KIP approval
> > >> > > >>>> standard, anyway, I will
> > >> > > >>> start
> > >> > > >>>> the PR soon and waiting for more binding approvals.
> > >> > > >>>>
> > >> > > >>>> Thanks! Feyman
> > >> > > >>>>
> > >> > > >>>>
> > >> > > >>>>
> > ------------------------------------------------------------------
> > >> > > >>>>
> > >> > > >>>>
> > >> > > > 发件人：John Roesler <vv...@apache.org>
> > >> > > >>>> 发送时间：2020年3月2日(星期一) 22:00 收件人：dev
> > >> > > > <de...@kafka.apache.org> 主
> > >> > > >>>> 题：Re: 回复：回复：[Vote] KIP-571: Add option to force remove
> > members
> > >> > > >>>> in StreamsResetter
> > >> > > >>>>
> > >> > > >>>> Hi Feyman,
> > >> > > >>>>
> > >> > > >>>> Sorry, but we actually need 3 binding votes for the KIP to
> > >> > > >>>> pass. Please feel free to keep bumping the thread until
> some
> > >> > > >>>> more committers can take
> > >> > > >>> a
> > >> > > >>>> look.
> > >> > > >>>>
> > >> > > >>>> By the way, you can totally start a PR, but we can’t merge
> it
> > >> > > >>>> until the KIP passes the vote.
> > >> > > >>>>
> > >> > > >>>> Thanks! John
> > >> > > >>>>
> > >> > > >>>> On Mon, Mar 2, 2020, at 00:24, feyman2009 wrote:
> > >> > > >>>>> Hi,all Since currently we have 1 binding and two
> non-binding
> > >> > > >>>>> +1, I will update the KIP-571 as adopted and initiate a PR
> > >> > > >>>>> shortly
> > >> > > >>>>>
> > >> > > >>>>> Thanks! Feyman
> > >> > > >>>>>
> > >> > > >>>>>
> > >> > > >>>>>
> > ------------------------------------------------------------------
> > >> > > >>>>>
> > >> > > >>>>>
> > >> > > > 发件人：Sophie Blee-Goldman <so...@confluent.io>
> > >> > > >>>>> 发送时间：2020年2月28日(星期五) 10:17 收件人：dev
> > >> > > > <de...@kafka.apache.org> 主
> > >> > > >>>>> 题：Re: 回复：[Vote] KIP-571: Add option to force remove
> members
> > >> > > >>>>> in
> > >> > > >>>> StreamsResetter
> > >> > > >>>>>
> > >> > > >>>>> Thanks for the KIP, +1 (non-binding)
> > >> > > >>>>>
> > >> > > >>>>> On Thu, Feb 27, 2020 at 12:40 PM Boyang Chen <
> > >> > > >>> reluctanthero104@gmail.com
> > >> > > >>>>>
> > >> > > >>>>> wrote:
> > >> > > >>>>>
> > >> > > >>>>>> Thanks Feyman, +1 (non-binding)
> > >> > > >>>>>>
> > >> > > >>>>>> On Thu, Feb 27, 2020 at 9:25 AM John Roesler
> > >> > > >>>>>> <vv...@apache.org>
> > >> > > >>>> wrote:
> > >> > > >>>>>>
> > >> > > >>>>>>> Thanks for the proposal!
> > >> > > >>>>>>>
> > >> > > >>>>>>> I'm +1 (binding) -John
> > >> > > >>>>>>>
> > >> > > >>>>>>> On Wed, Feb 26, 2020, at 19:41, feyman2009 wrote:
> > >> > > >>>>>>>> Updated with the KIP link:
> > >> > > >>>>>>>>
> > >> > > >>>>>>>
> > >> > > >>>>>>
> > >> > > >>>>
> > >> > > >>>
> > >> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-571%3A+Add+opti
> > >> > > > on+to+force+remove+members+in+StreamsResetter
> > >> > > >>>>>>>>
> > >> > > >>>>>>>>
> > >> > > >>>>>>>>
> > >> > > >>>
> > >> > > >>>
> > >> > > >
> > ------------------------------------------------------------------
> > >> > > >>>>>>>> 发件人：feyman2009 <fe...@aliyun.com.INVALID> 发送时
> > >> > > >>>>>>>> 间：2020年2月27日(星期四) 09:38 收件人：dev <de...@kafka.apache.org>
> > >> > > >>>>>>>> 主 题：[Vote] KIP-571: Add option to force remove members
> > >> > > >>>>>>>> in
> > >> > > >>>>>> StreamsResetter
> > >> > > >>>>>>>>
> > >> > > >>>>>>>>
> > >> > > >>>>>>>> Hi, all I would like to start a vote on KIP-571: Add
> > >> > > >>>>>>>> option to force
> > >> > > >>>> remove
> > >> > > >>>>>>>> members in StreamsResetter .
> > >> > > >>>>>>>>
> > >> > > >>>>>>>> Thanks! Feyman
> > >> > > >>>>>>>>
> > >> > > >>>>>>>>
> > >> > > >>>>>>>
> > >> > > >>>>>>
> > >> > > >>>>>
> > >> > > >>>>>
> > >> > > >>>>
> > >> > > >>>>
> > >> > > >>>>
> > >> > > >>>> -- -- Guozhang
> > >> > > >>>>
> > >> > > >>>>
> > >> > > >>>>
> > >> > > >>>
> > >> > > >>> -- -- Guozhang
> > >> > > >>>
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > >
> > >> > >
> > >> >
> > >> > --
> > >> > -- Guozhang
> > >> >
> > >>
> > >>
> > >>
> > >>
> > >>
> > >
> > >
> >
> >
>
>