You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Frank Yang <ya...@gmail.com> on 2024/11/01 06:39:46 UTC
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Hi Sean / Andrew / Lianet,
Thanks for all your review and suggestions.
AS1, LM1, LM4: Change to add KIP-848 information when users give —verbose option.
—describe —verbose: shows group epoch and target assignment epoch.
—describe —members —verbose: shows above information, member epoch, and target assignment.
AS2: Change to use MEMBER-EPOCH to align with KIP-848 definition.
LM2: For classic group, it doesn’t have epoch, so I use Optional fields in ConsumerGroupDescription.
For share group, it relies on KIP-848. It must have epoch, so I use int fields in ShareGroupDescription.
LM3: Remove —state change. We can get group level information by —describe —verbose.
SQ1: Add LEADER-EPOCH when users give —describe —offsets —verbose.
Thanks.
PoAn
> On Nov 1, 2024, at 5:08 AM, Lianet M. <li...@gmail.com> wrote:
>
> Hello Frank, thanks for the KIP! A few comments:
>
> LM1. I strongly agree with Andrew's suggestion of moving this into a
> --verbose option. I expect someone would only attempt to make sense of the
> epochs while debugging an issue, not while trying to view the group or
> member's info. (Also member-epoch makes more sense to me, even more if we
> end up agreeing in a --verbose). Related to both issues, my take is that
> whoever is close to the protocol will expect member-epoch, whoever is not
> will probably won't even need to see the epochs at all.
>
> LM2. Why are the epochs defined as Optional (don't we expect to always have
> them? with 0 initially, defined on the broker side for the group ones, and
> client side for the member)
>
> LM3. Why is the KIP including the –state option in the proposed changes? (
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
> ).
> I get that the output would change in that example, but it’s not because of
> any change to the –state option itself. It's because of the change proposed
> to the –described (required with the --state), and the changes to
> --describe are already explained above (seems confusing, got me looking for
> a change to the state filter which seemed unrelated).
>
> LM4. Group epoch and target assignment epoch are conceptually at the group
> level. vs member epoch that lands at a member level. So wonder if we should
> show them accordingly (ex. using the --verbose option)
> –describe –verbose => shows group epoch and target assignment epoch
> –describe –members –verbose => shows member epoch (along
> with group epoch and target assignment epoch)
>
> Thanks!
>
> Lianet
>
> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield <
> andrew_schofield_jira@outlook.com> wrote:
>
>> Hi PoAn,
>> Thanks for the KIP. I have a few comments.
>>
>> AS1: It seems to me that these new additions are most useful to people
>> trying to understand
>> the progress of rebalancing in quite some detail. The information is
>> really not understandable
>> for most users who do not have deep knowledge of KIP-848/932.
>>
>> As a result, I suggest for kafka-share-groups.sh that you add a --members
>> --verbose option
>> and only include the new information in the output for that option, rather
>> than changing the
>> non-verbose --members output.
>>
>> I also make a similar suggestion for kafka-consumer-groups.sh --members
>> and only add the
>> new information for the --verbose output.
>>
>> AS2: I strongly suggest that you use MEMBER-EPOCH instead of
>> CONSUMER-EPOCH.
>> I think it's more confusing not following the terminology of the KIPs. For
>> one thing,
>> we already have "member" in the admin client such as MemberDescription.
>> It's not a
>> new term.
>>
>>
>> Thanks,
>> Andrew
>> ________________________________________
>> From: PoAn Yang <pa...@apache.org>
>> Sent: 25 October 2024 13:55
>> To: dev@kafka.apache.org <de...@kafka.apache.org>
>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line
>> tool to support new consumer group
>>
>> Hi Lucas,
>>
>> Thanks for the review!
>>
>> 1) Yes, I add related change for kafka-share-groups.sh to the KIP. Could
>> you take a look? Thanks for the suggestion.
>>
>> 2) We use CONSUMER-ID as member ID. If we use MEMBER-EPOCH here, users may
>> confuse what is different between CONSUMER and MEMBER.
>>
>> Thanks,
>> PoAn
>>
>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
>>> Hi Frank,
>>>
>>> thanks for the KIP!
>>>
>>> 1) For consistency, should we do the same for
>>> kafka-share-groups.sh, ShareGroupDescription, etc. ? Even if we do not
>>> implement it right now if the share group implementation may still be
>>> incomplete, it may make sense to include it in the KIP.
>>>
>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH? That would seem more
>>> consistent.
>>>
>>> Cheers,
>>> Lucas
>>>
>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <ya...@gmail.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I would like to kick off the discussion of KIP-1099. This KIP enhances
>> the
>>>> kafka-consumer-groups tools to include state which is introduced by
>> KIP-848.
>>>>
>>>> KIP-1099: Extend kafka-consumer-groups command line tool to support new
>>>> consumer group - Apache Kafka - Apache Software Foundation
>>>> <
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>
>>>> cwiki.apache.org
>>>> <
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>
>>>> [image: favicon.ico]
>>>> <
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>
>>>> <
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>
>>>>
>>>> Thank you,
>>>> PoAn
>>>>
>>>
>>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by Andrew Schofield <an...@outlook.com>.
Hi David,
Yes, it would be an option to have a ShareMemberDescription. Because
ShareGroupDescription is still an evolving interface, it's acceptable
to change its definition in a minor release, so no rush to get this done
in the next 2 weeks.
I suggest that I change KIP-932 to introduce ShareMemberDescription
to replace the use of MemberDescription in ShareGroupDescription.
While I'm at it, it's probably prudent to have a ShareMemberAssignment
also because I have plans in mind which take assignments beyond
what MemberAssignment can represent.
I'll start just by defining new classes which simply remove the bit of
cruft which does not apply to share groups.
Thanks,
Andrew
________________________________________
From: David Jacot <dj...@confluent.io.INVALID>
Sent: 25 November 2024 15:08
To: dev@kafka.apache.org <de...@kafka.apache.org>
Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Hi PoAn,
Thanks for the update.
Regarding the CLI, I wonder whether we should rather have a column called
"Upgraded" which is only displayed when some members use the classic
protocol. The displayed value would be True for consumer members and False
for classic members. I think that this may be more useful for end users.
What do you think?
It is a bit unfortunate that we reused MemberDescription for share groups
in the admin client because they are unrelated fields now. I wonder whether
we should have a ShareMemberDescription. Andrew, would it be an option?
Best,
David
On Mon, Nov 25, 2024 at 3:50 PM Andrew Schofield <
andrew_schofield_jira@outlook.com> wrote:
> Hi PoAn,
> I've taken another pass over the KIP and have some more comments around
> the share
> group support.
>
> AS4: Share groups do not have the concept of a target assignment. As a
> result,
> the verbose state TARGET-ASSIGNMENT-EPOCH is a bit misleading from my
> point of view. ASSIGNMENT-EPOCH is better.
>
> AS5: Similarly for the verbose members output, the TARGET-EPOCH and
> TARGET-ASSIGNMENT are not really appropriate. Personally, I would prefer
> just MEMBER-EPOCH and ASSIGNMENT, and remove the "target" fields.
>
> Thanks,
> Andrew
>
> ________________________________________
> From: PoAn Yang <ya...@gmail.com>
> Sent: 24 November 2024 15:08
> To: dev@kafka.apache.org <de...@kafka.apache.org>
> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line
> tool to support new consumer group
>
> Hi Chia-Ping,
>
> Thanks for the suggestion.
>
> The “isClassic” field in ConsumerGroupDescribeResponse is a boolean value,
> so we can’t use it as the protocol string.
> I change the “PROTOCOL” column to “IS-CLASSIC” in kafka-consumer-groups.sh.
> It returns “true” if a member is in a classic group or it’s a classic
> member in a consumer group.
> In other cases, it returns “false”.
>
> I have a draft implementation in
> https://github.com/apache/kafka/pull/17872.
>
> Thanks,
> PoAn
>
> > On Nov 23, 2024, at 1:05 PM, Chia-Ping Tsai <ch...@gmail.com> wrote:
> >
> > hi PoAn
> >
> > `isClassic` is good to me :)
> >
> > Best,
> > Chia-Ping
> >
> > PoAn Yang <ya...@gmail.com> 於 2024年11月23日 週六 下午12:35寫道：
> >
> >> Hi Chia-Ping / David,
> >>
> >> Thanks for the great suggestion.
> >>
> >> How about using “isClassic”?
> >> The term like “isLegacy” may be equivocal in the future.
> >> For example, if we have a newer version of consumer / share consumer,
> the
> >> meaning will be different for different cases.
> >> If we use “isClassic”, for MemberDescription in ClassicGroupDescription,
> >> the value can be always “true”.
> >> In ConsumerGroupDescription, it’s “true” when it’s a classic member.
> >> In ShareGroupDescription, it’s always “false”.
> >>
> >> Thanks,
> >> PoAn
> >>
> >>> On Nov 23, 2024, at 1:02 AM, Chia-Ping Tsai <ch...@gmail.com>
> wrote:
> >>>
> >>> hi DJ
> >>>
> >>>> My goal was to have a way to tell the user that his (new) consumer
> group
> >>> may have non-upgraded members yet. I wonder if we could use a boolean
> to
> >>> signal this, e.g. isLegacy. Then, in the CLI we could have a column
> which
> >>> is only displayed when they are non upgraded members. What do you
> think?
> >>>
> >>> Yes, this is a good approach, but the downside is that the new isLegacy
> >>> field in MemberDescription would be irrelevant for classic and shared
> >>> groups. This would require thorough documentation to clarify its
> purpose.
> >>>
> >>> Additionally, we could use a boolean instead of a string for the new
> >> field
> >>> in the RPC to reduce the request size.
> >>> Best,
> >>> Chia-Ping
> >>>
> >>>
> >>>
> >>> David Jacot <dj...@confluent.io.invalid> 於 2024年11月23日 週六 上午12:41寫道：
> >>>
> >>>> Hi both,
> >>>>
> >>>> Ah, I understand what you mean now. You're saying that we have two
> >> protocol
> >>>> fields. One at the group level and one at the member level (the new
> >> one).
> >>>> The two fields are actually two different things. The former is the
> >> name of
> >>>> the embedded protocol used in the classic protocol whereas the latter
> is
> >>>> the name of the protocol/api used. This is indeed confusing.
> >>>>
> >>>> My goal was to have a way to tell the user that his (new) consumer
> group
> >>>> may have non-upgraded members yet. I wonder if we could use a boolean
> to
> >>>> signal this, e.g. isLegacy. Then, in the CLI we could have a column
> >> which
> >>>> is only displayed when they are non upgraded members. What do you
> think?
> >>>>
> >>>> If we cannot find a good way to represent this, we could drop it from
> >> this
> >>>> KIP too.
> >>>>
> >>>> Best,
> >>>> David
> >>>>
> >>>> On Fri, Nov 22, 2024 at 5:12 PM PoAn Yang <ya...@gmail.com> wrote:
> >>>>
> >>>>> Hi Chia-Ping / David,
> >>>>>
> >>>>> Thanks for the review and suggestions.
> >>>>>
> >>>>>>> I don't fully follow your comment so please excuse me if I say
> >>>> something
> >>>>>>> wrong.
> >>>>>
> >>>>> I think Chia-Ping wants to say: originally, the default protocol in
> >>>>> classic group is “consumer”.
> >>>>> After we have a new consumer group, the protocol of a classic member
> in
> >>>>> consumer group is “classic”.
> >>>>> The MemberDescription is used by both ClassicGroupDescription and
> >>>>> ConsumerGroupDescription.
> >>>>> If we add the protocol field to MemberDescription, the protocol of a
> >>>>> classic member in classic group is “consumer”, but it’s “classic” in
> >>>>> consumer group.
> >>>>> To avoid misleading word, we can add a new collection of member
> >> protocols
> >>>>> to ConsumerGroupDescription only.
> >>>>> The original goal is to show member protocol during consumer
> migration,
> >>>> so
> >>>>> we don’t need to add this information to ClassicGroupDescription.
> >>>>>
> >>>>> Please correct me if I misunderstood anything.
> >>>>>
> >>>>> Thanks,
> >>>>> PoAn
> >>>>>
> >>>>>> On Nov 22, 2024, at 7:00 PM, Chia-Ping Tsai <ch...@gmail.com>
> >>>> wrote:
> >>>>>>
> >>>>>> hi DJ
> >>>>>>
> >>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and
> >>>> change
> >>>>>> return type of ModernGroup#protocolType?
> >>>>>>
> >>>>>> This is not my comment ... sorry for the incorrect quote :(
> >>>>>>
> >>>>>>> A member in a classic group should report "classic".
> >>>>>>
> >>>>>> I completely agree. However, in this case,
> >>>>> ClassicGroupDescription#protocol
> >>>>>> returns 'consumer', while
> ClassicGroupDescription#members[x].protocol
> >>>>>> returns 'classic'. This inconsistency feels a bit strange to me.
> >>>>>>
> >>>>>> Best,
> >>>>>> Chia-Ping
> >>>>>>
> >>>>>>
> >>>>>> David Jacot <dj...@confluent.io.invalid> 於 2024年11月22日 週五
> 下午6:51寫道：
> >>>>>>
> >>>>>>> Hi Chia,
> >>>>>>>
> >>>>>>>>> LM4: I agree it’s better to use a type like GroupProtocol enum,
> but
> >>>> I
> >>>>>>> would like to align the format with kafka-groups.sh.
> >>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and
> >>>> change
> >>>>>>> return type of ModernGroup#protocolType?
> >>>>>>>
> >>>>>>> My understanding is that `GroupProtocol` is used to drive the
> config
> >>>> of
> >>>>> the
> >>>>>>> `Consumer`. Hence adding `Share` seems wrong to me. Or are you
> >>>>> referring to
> >>>>>>> another GroupProtocol enum? Perhaps GroupType enum?
> >>>>>>>
> >>>>>>>> In fact, I'm wondering if we should even add 'protocol' to
> >>>>>>> MemberDescription,
> >>>>>>> since the field has different values between classic and consumer
> >>>>> groups—it
> >>>>>>> shows 'consumer' in classic groups, whereas it shows 'classic' in
> >>>>> consumer
> >>>>>>> groups. Yes, it seems like a typo, but it's true. This
> inconsistency
> >>>>> feels
> >>>>>>> confusing to me. For example, users call
> Admin#describeConsumerGroups
> >>>>> and
> >>>>>>> see that a member has the 'consumer' protocol when it's in a
> classic
> >>>>> group.
> >>>>>>> Then, if the group is upgraded to a consumer group, calling
> >>>>>>> Admin#describeConsumerGroups again shows the protocol as 'classic'
> >> for
> >>>>> the
> >>>>>>> same member.
> >>>>>>>
> >>>>>>> I don't fully follow your comment so please excuse me if I say
> >>>> something
> >>>>>>> wrong.
> >>>>>>>
> >>>>>>> I think that it should work as follows:
> >>>>>>> * A member in a classic group should report "classic".
> >>>>>>> * A member using the consumer protocol in a consumer group should
> >>>> report
> >>>>>>> "consumer".
> >>>>>>> * A member using the classic protocol in a consumer group should
> >>>> report
> >>>>>>> "classic".
> >>>>>>> * A member in a share group should report "share".
> >>>>>>>
> >>>>>>>> If all we want to do is distinguish classic members in the
> >>>>>>> ConsumerGroupDescription, then we shouldn't modify the common
> >>>>>>> MemberDescription structure. Instead, we could add a collection to
> >>>>>>> ConsumerGroupDescription to trace members using the classic
> protocol,
> >>>>> which
> >>>>>>> would allow us to enrich the output of command-line tools.
> >>>>>>>
> >>>>>>> Personally, I believe that using different collections per protocol
> >>>>> types
> >>>>>>> is not necessary. A field in the member is more than enough.
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> DJ
> >>>>>>>
> >>>>>>> On Fri, Nov 22, 2024 at 12:01 PM Chia-Ping Tsai <
> chia7712@gmail.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> hi PoAn
> >>>>>>>>
> >>>>>>>>> LM4: I agree it’s better to use a type like GroupProtocol enum,
> but
> >>>> I
> >>>>>>>> would like to align the format with kafka-groups.sh.
> >>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and
> >>>>> change
> >>>>>>>> return type of ModernGroup#protocolType?
> >>>>>>>>
> >>>>>>>> MemberDescription is also used by ClassicGroupDescription, and in
> >>>>> classic
> >>>>>>>> groups, the protocol can be an arbitrary string, so we can't use
> an
> >>>>> enum.
> >>>>>>>>
> >>>>>>>> In fact, I'm wondering if we should even add 'protocol' to
> >>>>>>>> MemberDescription,
> >>>>>>>> since the field has different values between classic and consumer
> >>>>>>> groups—it
> >>>>>>>> shows 'consumer' in classic groups, whereas it shows 'classic' in
> >>>>>>> consumer
> >>>>>>>> groups. Yes, it seems like a typo, but it's true. This
> inconsistency
> >>>>>>> feels
> >>>>>>>> confusing to me. For example, users call
> >> Admin#describeConsumerGroups
> >>>>> and
> >>>>>>>> see that a member has the 'consumer' protocol when it's in a
> classic
> >>>>>>> group.
> >>>>>>>> Then, if the group is upgraded to a consumer group, calling
> >>>>>>>> Admin#describeConsumerGroups again shows the protocol as 'classic'
> >>>> for
> >>>>>>> the
> >>>>>>>> same member.
> >>>>>>>>
> >>>>>>>> If all we want to do is distinguish classic members in the
> >>>>>>>> ConsumerGroupDescription, then we shouldn't modify the common
> >>>>>>>> MemberDescription structure. Instead, we could add a collection to
> >>>>>>>> ConsumerGroupDescription to trace members using the classic
> >> protocol,
> >>>>>>> which
> >>>>>>>> would allow us to enrich the output of command-line tools.
> >>>>>>>> Best,
> >>>>>>>> Chia-Ping
> >>>>>>>>
> >>>>>>>> PoAn Yang <ya...@gmail.com> 於 2024年11月22日 週五 下午3:46寫道：
> >>>>>>>>
> >>>>>>>>> Hi Lianet / Jeff,
> >>>>>>>>>
> >>>>>>>>> Thanks for the review.
> >>>>>>>>>
> >>>>>>>>> LM5: The kafka-share-groups.sh also uses “—describe” to show
> >> offsets
> >>>>>>>>> information.
> >>>>>>>>> Change to use “—describe —state —verbose” to show group level
> >>>>>>> information
> >>>>>>>>> in kafka-share-groups.sh.
> >>>>>>>>>
> >>>>>>>>> LM6: Update the example.
> >>>>>>>>>
> >>>>>>>>> LM7: Add a description to mention “—verbose” is a new option in
> >>>>>>>>> ShareGroupCommandOptions.
> >>>>>>>>>
> >>>>>>>>> JK2: Yes, add the statement about format change for ASSIGNMENT
> >>>> value.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> PoAn
> >>>>>>>>>
> >>>>>>>>>> On Nov 22, 2024, at 7:35 AM, Jeff Kim <je...@apache.org>
> >>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hi PoAn,
> >>>>>>>>>>
> >>>>>>>>>> JK2: Can we state in the KIP that we will reformat the existing
> >>>>>>>>>> classic group's `--members --verbose` as well, specifically the
> >>>>>>>>>> "ASSIGNMENT" column to include the topic name?
> >>>>>>>>>>
> >>>>>>>>>> Thanks!
> >>>>>>>>>>
> >>>>>>>>>> On 2024/11/21 14:33:37 "Lianet M." wrote:
> >>>>>>>>>>> Thanks for the updates PoAn!
> >>>>>>>>>>>
> >>>>>>>>>>> LM6. Just a nit, under the --describe --state --verbose the
> >> header
> >>>>>>> is
> >>>>>>>>> fine
> >>>>>>>>>>> but the example is still missing the --state argument.
> >>>>>>>>>>>
> >>>>>>>>>>> LM7. Under the Proposed changes for kafka-share-groups.sh,
> should
> >>>> we
> >>>>>>>>>>> clearly mention that the KIP adds a new --verbose option? I
> think
> >>>>>>> it's
> >>>>>>>>>>> confusing because it's presented as if the option already
> exists
> >>>> and
> >>>>>>>>> we're
> >>>>>>>>>>> only changing the output (which is the case for
> >>>>>>> kafka-consumer-groups
> >>>>>>>>> but
> >>>>>>>>>>> not for kafka-share-groups)
> >>>>>>>>>>>
> >>>>>>>>>>> That's all on my side. Thanks!
> >>>>>>>>>>>
> >>>>>>>>>>> Lianet
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Nov 20, 2024 at 11:12 PM PoAn Yang <yangpoan@gmail.com
> >
> >>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi Lianet / Jeff,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks for review and suggestions.
> >>>>>>>>>>>>
> >>>>>>>>>>>> LM4: At group level, the protocol field is from
> >>>>>>>>> ModernGroup#protocolType.
> >>>>>>>>>>>> It’s a string.
> >>>>>>>>>>>> The value is defined in different places like
> >>>>>>>> ShareGroup#PROTOCOL_TYPE
> >>>>>>>>> and
> >>>>>>>>>>>> ConsumerProtocol#PROTOCOL_TYPE.
> >>>>>>>>>>>> In KIP-1043 <
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=305171038#KIP1043:Administrationofgroups-kafka-groups.sh
> >>>>>>>>>> ,
> >>>>>>>>>>>> the kafka-groups.sh also shows protocol like “classic” /
> >>>>>>> “consumer” /
> >>>>>>>>>>>> “share”.
> >>>>>>>>>>>> I agree it’s better to use a type like GroupProtocol enum,
> but I
> >>>>>>>> would
> >>>>>>>>>>>> like to align the format with kafka-groups.sh.
> >>>>>>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol
> >> and
> >>>>>>>>> change
> >>>>>>>>>>>> return type of ModernGroup#protocolType?
> >>>>>>>>>>>>
> >>>>>>>>>>>> LM5 / JK1: Sorry, I misunderstood LM3, so I deleted —state in
> >> the
> >>>>>>>>> title.
> >>>>>>>>>>>> You’re correct. I would like to use `—describe —state
> —verbose`
> >>>> to
> >>>>>>>> show
> >>>>>>>>>>>> group level information.
> >>>>>>>>>>>> For `—describe —verbose`, it will show output as same as
> >>>> `—describe
> >>>>>>>>>>>> —offsets —verbose`,
> >>>>>>>>>>>> because `—describe` also shows output as same as `—describe
> >>>>>>>> —offsets`.
> >>>>>>>>>>>>
> >>>>>>>>>>>> JK2: Yes, I only use consumer group in the sample output, but
> >> the
> >>>>>>>>> classic
> >>>>>>>>>>>> group will use the same format as well.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>> PoAn
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On Nov 21, 2024, at 8:28 AM, Jeff Kim <je...@apache.org>
> >>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for the KIP. I have some questions:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> JK1: (Realized this is similar to LM5) On `--describe
> >> --verbose`
> >>>>>>>>>>>> proposed changes - doesn't `--describe $group` default to
> >>>> printing
> >>>>>>>> the
> >>>>>>>>>>>> offsets? Perhaps you're referring to the `--state` argument?
> >>>> Also,
> >>>>>>>>> would
> >>>>>>>>>>>> that mean the default `--describe $group --verbose` command
> >> would
> >>>>>>> not
> >>>>>>>>> print
> >>>>>>>>>>>> the added field to `--offsets verbose` (leader-epoch) or would
> >>>> it?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> JK2: Looking at ConsumerGroupCommand.java, the existing
> >>>>>>> "ASSIGNMENT"
> >>>>>>>>>>>> column under `--members --verbose` does group the topic
> >>>> partitions
> >>>>>>> by
> >>>>>>>>> topic
> >>>>>>>>>>>> but does not prefix the grouping with the topic name like
> you're
> >>>>>>>>> proposal:
> >>>>>>>>>>>> "my_topic:0,1;new_topic:0,1". Should we do apply the same
> format
> >>>>>>> for
> >>>>>>>>> the
> >>>>>>>>>>>> classic group as well?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 2024/11/20 21:32:41 "Lianet M." wrote:
> >>>>>>>>>>>>>> Hello PoAn, just of couple more minor comments:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> LM4. Regarding the new “protocol” field added to
> >>>>>>> MemberDescription.
> >>>>>>>>>>>> Should
> >>>>>>>>>>>>>> we consider reusing the existing GroupProtocol enum instead
> of
> >>>>>>>>> String?
> >>>>>>>>>>>>>> (It’s the one we use from the consumer side to refer to the
> >>>>>>>> protocol
> >>>>>>>>> in
> >>>>>>>>>>>>>> use, just missing Share I notice).
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> LM5. Regarding the change to the output for
> >>>>>>> kafka-consumer-groups,
> >>>>>>>>> the
> >>>>>>>>>>>>>> command shown does not include the —state option, but the
> >>>> output
> >>>>>>>>> shows
> >>>>>>>>>>>>>> state info (state, #members, epochs). I would guess that we
> >>>> want
> >>>>>>> to
> >>>>>>>>>>>> modify
> >>>>>>>>>>>>>> the output only when we describe a group with the —state
> >>>> —verbose
> >>>>>>>>>>>> option,
> >>>>>>>>>>>>>> is my understanding right? If my understanding is right
> we’re
> >>>>>>> just
> >>>>>>>>>>>> missing
> >>>>>>>>>>>>>> adding the —state in the example, and the KIP does not
> >>>> introduce
> >>>>>>>> any
> >>>>>>>>>>>>>> changes to the —describe —verbose option. (If not, it would
> >>>> mean
> >>>>>>> a
> >>>>>>>>>>>> bigger
> >>>>>>>>>>>>>> change to the output of —describe —verbose which I expect is
> >>>> not
> >>>>>>>> the
> >>>>>>>>>>>>>> intention?)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Lianet
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Wed, Nov 20, 2024 at 2:12 AM PoAn Yang <
> yangpoan@gmail.com
> >>>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi Chia-Ping,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks for the review and suggestions.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> CI0: Thanks for the reminder. Update validVersions in
> >>>>>>>>>>>>>>> ConsumerGroupDescribeRequest to 0-1.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> CI1: Yes, I use `ConsumerGroupMember#useClassicProtocol` to
> >>>>>>> check
> >>>>>>>>>>>> whether
> >>>>>>>>>>>>>>> a member in consumer group uses “classic” or “consumer”
> >>>>>>> protocol.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> CI2: Yes, a member in share group always uses “share”
> >>>> protocol.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> CI3: Add a table to show meaning of “classic”, “consumer”,
> >> and
> >>>>>>>>> “share”
> >>>>>>>>>>>>>>> protocol.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> BTW, the vote thread is in
> >>>>>>>>>>>>>>>
> >>>>>>> https://lists.apache.org/thread/rb25tf75tzf4c7jqqldlo5jh9w8chsq6.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Nov 20, 2024, at 11:46 AM, Chia-Ping Tsai <
> >>>>>>>> chia7712@apache.org>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> hi PoAn
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> CI0: We have to bump the version of
> >>>>>>> ConsumerGroupDescribeRequest
> >>>>>>>>> as
> >>>>>>>>>>>>>>> well, so server can distinguish the new and old behavior.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> CI1: the type of new filed is string, so I guess you plan
> to
> >>>>>>> use
> >>>>>>>>>>>>>>> `ConsumerGroupMember#useClassicProtocol` [0] flag to return
> >>>>>>> either
> >>>>>>>>>>>>>>> "classic" or "consumer", right?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> CI2: `MemberDescription` is used by
> `ShareGroupDescription`
> >>>>>>> too,
> >>>>>>>> so
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>> new filed protocol in shared group is always "shared",
> right?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> CI3: Could you consider adding a table to show the value
> of
> >>>> the
> >>>>>>>>>>>> protocol
> >>>>>>>>>>>>>>> field in each case? Andrew has a beautiful table in
> KIP-1043
> >>>>>>> that
> >>>>>>>>>>>> lists all
> >>>>>>>>>>>>>>> possible protocol names.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> [0]
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> https://github.com/apache/kafka/blob/441a6d0b790f4a17b454caeea7588a6b90fbd9db/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/consumer/ConsumerGroupMember.java#L454
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On 2024/11/19 15:45:16 PoAn Yang wrote:
> >>>>>>>>>>>>>>>>> Hi David,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks for the reminder.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> DJ4: Add related content to the KIP.
> >>>>>>>>>>>>>>>>> Bump validVersions in ConsumerGroupDescribeResponse to
> >>>> “0-1”.
> >>>>>>>>>>>>>>>>> Add a new field “Protocol” to
> >> ConsumerGroupDescribeResponse.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Nov 19, 2024, at 3:57 PM, David Jacot
> >>>>>>>>>>>> <dj...@confluent.io.INVALID>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> DJ4: Could you please also include the required changes
> to
> >>>>>>> the
> >>>>>>>>>>>>>>>>>> ConsumerGroupDescribed API in the public interface
> >> section?
> >>>>>>> We
> >>>>>>>>>>>>>>> basically
> >>>>>>>>>>>>>>>>>> need to bump the version, add the new field, etc.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>> David
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang <
> >>>>>>> yangpoan@gmail.com>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi David,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> DJ4: Add a new field protocol to MemberDescription,
> >>>>>>>>>>>>>>>>>>> so the command line tool can show protocol information
> >>>> when
> >>>>>>>>> users
> >>>>>>>>>>>>>>> describe
> >>>>>>>>>>>>>>>>>>> members.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> If there is no further suggestion, I will start a vote
> >>>>>>> thread
> >>>>>>>>>>>> today.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Nov 15, 2024, at 11:50 PM, David Jacot
> >>>>>>>>>>>>>>> <dj...@confluent.io.INVALID>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> DJ2: Using "-" sounds good to me.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> DJ3: That seems reasonable to me.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> DJ4: Why not add it right now? I don't want to change
> >> the
> >>>>>>>>> output
> >>>>>>>>>>>> of
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> tool too many times.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>> David
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang <
> >>>>>>>> yangpoan@gmail.com>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hi David / Andrew,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks for review. Thanks Andrew for picking up
> >>>>>>>>>>>>>>> kafka-share-groups.sh
> >>>>>>>>>>>>>>>>>>>>> implementation.
> >>>>>>>>>>>>>>>>>>>>> I will handle kafka-consumer-groups.sh.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> DJ3: After discussing with @Chia-Ping Tsai, we think
> >>>> that
> >>>>>>>>> using
> >>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>> format
> >>>>>>>>>>>>>>>>>>>>> is more clear.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> The new format will be like
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Using colon(:) to concat topic name and partition
> IDs.
> >>>>>>>>>>>>>>>>>>>>> Using comma(,) to concat partition IDs within same
> >> topic
> >>>>>>>> name.
> >>>>>>>>>>>>>>>>>>>>> Using semicolon(;) to concat topic strings.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> AS3: Fix it with kafka-share-groups.sh. Thanks.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> If there is no further suggestion, I will start a
> vote
> >>>>>>>> thread
> >>>>>>>>>>>> next
> >>>>>>>>>>>>>>>>>>> Monday.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield <
> >>>>>>>>>>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>>>>>>>>>>>>> DJ2: I was just going to comment that "-" would be a
> >>>> more
> >>>>>>>>>>>>>>> appropriate
> >>>>>>>>>>>>>>>>>>>>> missing value, but
> >>>>>>>>>>>>>>>>>>>>>> you got there first.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> AS3: The examples for kafka-share-groups.sh include
> >>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups.sh in the
> >>>>>>>>>>>>>>>>>>>>>> command line.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> If this is accepted in time, I'm happy to pick up
> the
> >>>>>>>>>>>>>>> implementation of
> >>>>>>>>>>>>>>>>>>>>> the share groups
> >>>>>>>>>>>>>>>>>>>>>> part of this if it helps.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>> Andrew
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>>>>>>>>>> From: Frank Yang <ya...@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>> Sent: 14 November 2024 10:48
> >>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
> >>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
> >>>>>>>> kafka-consumer-groups
> >>>>>>>>>>>>>>> command
> >>>>>>>>>>>>>>>>>>>>> line tool to support new consumer group
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Hi David,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Thanks for the review and suggestion! I would like
> to
> >>>> get
> >>>>>>>>> this
> >>>>>>>>>>>> in
> >>>>>>>>>>>>>>> AK
> >>>>>>>>>>>>>>>>>>> 4.0
> >>>>>>>>>>>>>>>>>>>>> as well. I will do my best.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> DJ1: Update KIP to put GROUP-EPOCH and
> >>>>>>>>> TARGET-ASSIGNMENT-EPOCH
> >>>>>>>>>>>>>>> before
> >>>>>>>>>>>>>>>>>>>>> #MEMBERS.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> DJ2: I prefer to follow current missing column value
> >>>> “-“.
> >>>>>>>>>>>>>>> (reference <
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92
> >>>>>>>>>>>>>>>>>>>>>> )
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> DJ3: Update KIP to use CURRENT-EPOCH
> >> CURRENT-ASSIGNMENT
> >>>>>>>>>>>>>>> TARGET-EPOCH
> >>>>>>>>>>>>>>>>>>>>> TARGET-ASSIGNMENT.
> >>>>>>>>>>>>>>>>>>>>>> Remove GROUP-EPOCH.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> For assignment value, it follows current output
> >>>>>>> (reference
> >>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418
> >>>>>>>>>>>>>>>>>>>> ).
> >>>>>>>>>>>>>>>>>>>>> I think the form of `topicid-partitionid` is more
> >> clear.
> >>>>>>>>>>>>>>>>>>>>>> If we would like to use this form, I will update
> both
> >>>>>>>> output
> >>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups.sh and kafka-share-groups.sh.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> DJ4: It looks like DescribeGroupsResponseData only
> has
> >>>>>>>>> protocol
> >>>>>>>>>>>>>>> type at
> >>>>>>>>>>>>>>>>>>>>> group level.
> >>>>>>>>>>>>>>>>>>>>>> Both DescribeGroupsResponseData and
> >>>>>>>>>>>>>>> ConsumerGroupDescribeResponseData
> >>>>>>>>>>>>>>>>>>>>> don’t have protocol at member level.
> >>>>>>>>>>>>>>>>>>>>>> Could we use a followup to add it?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> DJ5: Update KIP to put LEADER-EPOCH before
> >>>>>>> CURRENT-OFFSET.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot
> >>>>>>>>>>>>>>> <djacot@confluent.io.INVALID
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP! I have a few minor
> >>>>>>>> comments/suggestions:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> DJ1: In the output of `--describe --verbose`, I
> would
> >>>>>>>>> suggest
> >>>>>>>>>>>>>>> putting
> >>>>>>>>>>>>>>>>>>>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before
> >>>>>>>>> `#MEMBERS`.
> >>>>>>>>>>>>>>>>>>>>>>> DJ2: Continuing on the above, I assume that we will
> >>>>>>> print
> >>>>>>>>> out
> >>>>>>>>>>>> N/A
> >>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> fields not supported by classic groups. Is this
> >>>> correct?
> >>>>>>>>>>>>>>>>>>>>>>> DJ3: In the output of `--describe --members
> >>>> --verbose`,
> >>>>>>> I
> >>>>>>>>>>>> wonder
> >>>>>>>>>>>>>>> if we
> >>>>>>>>>>>>>>>>>>>>>>> should use the following order and terms:
> >>>> CURRENT-EPOCH
> >>>>>>>>>>>>>>>>>>>>> CURRENT-ASSIGNMENT
> >>>>>>>>>>>>>>>>>>>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the
> >>>>>>>>> GROUP-EPOCH
> >>>>>>>>>>>>>>>>>>> because
> >>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>> is already in the group description. The value
> `(0)`
> >>>>>>> used
> >>>>>>>>> for
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> assignment is incorrect. Here I suppose that we
> will
> >>>>>>> print
> >>>>>>>>> out
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> list
> >>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>> partitions in the form of `topicid-partitionid`.
> >>>>>>>>>>>>>>>>>>>>>>> DJ4: Continuing on the above, I wonder if we should
> >>>> also
> >>>>>>>> add
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> protocol
> >>>>>>>>>>>>>>>>>>>>>>> used `classic` or `consumer`. For context, it is
> >>>>>>> possible
> >>>>>>>> to
> >>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>>>> `classic`
> >>>>>>>>>>>>>>>>>>>>>>> members and `consumer` members in a `consumer`
> group
> >>>>>>>> during
> >>>>>>>>> an
> >>>>>>>>>>>>>>> online
> >>>>>>>>>>>>>>>>>>>>>>> upgrade from the classic protocol to the consumer
> >>>>>>>> protocol.
> >>>>>>>>>>>> Having
> >>>>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>>> information may be handy for administrators. What
> do
> >>>> you
> >>>>>>>>> think?
> >>>>>>>>>>>>>>>>>>>>>>> DJ5: In the output of `--describe --offsets
> >>>> --verbose`,
> >>>>>>> I
> >>>>>>>>> would
> >>>>>>>>>>>>>>>>>>> suggest
> >>>>>>>>>>>>>>>>>>>>>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> It would be great if we could get this one in AK
> 4.0
> >>>> as
> >>>>>>> it
> >>>>>>>>>>>>>>> changes the
> >>>>>>>>>>>>>>>>>>>>>>> output of the command.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>> DJ
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <
> >>>>>>>>> yangpoan@gmail.com>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Hi Sean / Andrew / Lianet,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks for all your review and suggestions.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> AS1, LM1, LM4: Change to add KIP-848 information
> >> when
> >>>>>>>> users
> >>>>>>>>>>>> give
> >>>>>>>>>>>>>>>>>>>>> —verbose
> >>>>>>>>>>>>>>>>>>>>>>>> option.
> >>>>>>>>>>>>>>>>>>>>>>>> —describe —verbose: shows group epoch and target
> >>>>>>>> assignment
> >>>>>>>>>>>>>>> epoch.
> >>>>>>>>>>>>>>>>>>>>>>>> —describe —members —verbose: shows above
> >> information,
> >>>>>>>>> member
> >>>>>>>>>>>>>>> epoch,
> >>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>> target assignment.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> AS2: Change to use MEMBER-EPOCH to align with
> >> KIP-848
> >>>>>>>>>>>> definition.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> LM2: For classic group, it doesn’t have epoch, so
> I
> >>>> use
> >>>>>>>>>>>> Optional
> >>>>>>>>>>>>>>>>>>>>> fields in
> >>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupDescription.
> >>>>>>>>>>>>>>>>>>>>>>>> For share group, it relies on KIP-848. It must
> have
> >>>>>>>> epoch,
> >>>>>>>>> so
> >>>>>>>>>>>> I
> >>>>>>>>>>>>>>> use
> >>>>>>>>>>>>>>>>>>> int
> >>>>>>>>>>>>>>>>>>>>>>>> fields in ShareGroupDescription.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> LM3: Remove —state change. We can get group level
> >>>>>>>>> information
> >>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>>>> —describe
> >>>>>>>>>>>>>>>>>>>>>>>> —verbose.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> SQ1: Add LEADER-EPOCH when users give —describe
> >>>>>>> —offsets
> >>>>>>>>>>>>>>> —verbose.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks.
> >>>>>>>>>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <
> >>>>>>>> lianetmr@gmail.com
> >>>>>>>>>>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Hello Frank, thanks for the KIP! A few comments:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> LM1. I strongly agree with Andrew's suggestion of
> >>>>>>> moving
> >>>>>>>>> this
> >>>>>>>>>>>>>>> into a
> >>>>>>>>>>>>>>>>>>>>>>>>> --verbose option. I expect someone would only
> >>>> attempt
> >>>>>>> to
> >>>>>>>>> make
> >>>>>>>>>>>>>>> sense
> >>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> epochs while debugging an issue, not while trying
> >> to
> >>>>>>>> view
> >>>>>>>>> the
> >>>>>>>>>>>>>>> group
> >>>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>>>> member's info. (Also member-epoch makes more
> sense
> >>>> to
> >>>>>>>> me,
> >>>>>>>>>>>> even
> >>>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>>>>> if we
> >>>>>>>>>>>>>>>>>>>>>>>>> end up agreeing in a --verbose). Related to both
> >>>>>>> issues,
> >>>>>>>>> my
> >>>>>>>>>>>>>>> take is
> >>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>> whoever is close to the protocol will expect
> >>>>>>>> member-epoch,
> >>>>>>>>>>>>>>> whoever
> >>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>> will probably won't even need to see the epochs
> at
> >>>>>>> all.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> LM2. Why are the epochs defined as Optional
> (don't
> >>>> we
> >>>>>>>>> expect
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> always
> >>>>>>>>>>>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>>>>>>>> them? with 0 initially, defined on the broker
> side
> >>>> for
> >>>>>>>> the
> >>>>>>>>>>>> group
> >>>>>>>>>>>>>>>>>>> ones,
> >>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>> client side for the member)
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> LM3. Why is the KIP including the –state option
> in
> >>>> the
> >>>>>>>>>>>> proposed
> >>>>>>>>>>>>>>>>>>>>> changes?
> >>>>>>>>>>>>>>>>>>>>>>>> (
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
> >>>>>>>>>>>>>>>>>>>>>>>>> ).
> >>>>>>>>>>>>>>>>>>>>>>>>> I get that the output would change in that
> example,
> >>>>>>> but
> >>>>>>>>> it’s
> >>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>> because
> >>>>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>> any change to the –state option itself. It's
> >> because
> >>>>>>> of
> >>>>>>>>> the
> >>>>>>>>>>>>>>> change
> >>>>>>>>>>>>>>>>>>>>>>>> proposed
> >>>>>>>>>>>>>>>>>>>>>>>>> to the –described (required with the --state),
> and
> >>>> the
> >>>>>>>>>>>> changes
> >>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>> --describe are already explained above (seems
> >>>>>>> confusing,
> >>>>>>>>> got
> >>>>>>>>>>>> me
> >>>>>>>>>>>>>>>>>>>>> looking
> >>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>> a change to the state filter which seemed
> >>>> unrelated).
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> LM4. Group epoch and target assignment epoch are
> >>>>>>>>> conceptually
> >>>>>>>>>>>>>>> at the
> >>>>>>>>>>>>>>>>>>>>>>>> group
> >>>>>>>>>>>>>>>>>>>>>>>>> level. vs member epoch that lands at a member
> >> level.
> >>>>>>> So
> >>>>>>>>>>>> wonder
> >>>>>>>>>>>>>>> if we
> >>>>>>>>>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>>>>>>> show them accordingly (ex. using the --verbose
> >>>> option)
> >>>>>>>>>>>>>>>>>>>>>>>>> –describe –verbose => shows group epoch and
> target
> >>>>>>>>> assignment
> >>>>>>>>>>>>>>> epoch
> >>>>>>>>>>>>>>>>>>>>>>>>> –describe –members –verbose => shows member
> >>>>>>>> epoch
> >>>>>>>>>>>>>>> (along
> >>>>>>>>>>>>>>>>>>>>>>>>> with group epoch and target assignment epoch)
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Lianet
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield
> <
> >>>>>>>>>>>>>>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I have a few comments.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> AS1: It seems to me that these new additions are
> >>>> most
> >>>>>>>>>>>> useful to
> >>>>>>>>>>>>>>>>>>>>> people
> >>>>>>>>>>>>>>>>>>>>>>>>>> trying to understand
> >>>>>>>>>>>>>>>>>>>>>>>>>> the progress of rebalancing in quite some
> detail.
> >>>> The
> >>>>>>>>>>>>>>> information
> >>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>> really not understandable
> >>>>>>>>>>>>>>>>>>>>>>>>>> for most users who do not have deep knowledge of
> >>>>>>>>>>>> KIP-848/932.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> As a result, I suggest for kafka-share-groups.sh
> >>>> that
> >>>>>>>> you
> >>>>>>>>>>>> add a
> >>>>>>>>>>>>>>>>>>>>>>>> --members
> >>>>>>>>>>>>>>>>>>>>>>>>>> --verbose option
> >>>>>>>>>>>>>>>>>>>>>>>>>> and only include the new information in the
> output
> >>>>>>> for
> >>>>>>>>> that
> >>>>>>>>>>>>>>> option,
> >>>>>>>>>>>>>>>>>>>>>>>> rather
> >>>>>>>>>>>>>>>>>>>>>>>>>> than changing the
> >>>>>>>>>>>>>>>>>>>>>>>>>> non-verbose --members output.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> I also make a similar suggestion for
> >>>>>>>>>>>> kafka-consumer-groups.sh
> >>>>>>>>>>>>>>>>>>>>> --members
> >>>>>>>>>>>>>>>>>>>>>>>>>> and only add the
> >>>>>>>>>>>>>>>>>>>>>>>>>> new information for the --verbose output.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> AS2: I strongly suggest that you use
> MEMBER-EPOCH
> >>>>>>>>> instead of
> >>>>>>>>>>>>>>>>>>>>>>>>>> CONSUMER-EPOCH.
> >>>>>>>>>>>>>>>>>>>>>>>>>> I think it's more confusing not following the
> >>>>>>>>> terminology of
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> KIPs.
> >>>>>>>>>>>>>>>>>>>>>>>> For
> >>>>>>>>>>>>>>>>>>>>>>>>>> one thing,
> >>>>>>>>>>>>>>>>>>>>>>>>>> we already have "member" in the admin client
> such
> >>>> as
> >>>>>>>>>>>>>>>>>>>>> MemberDescription.
> >>>>>>>>>>>>>>>>>>>>>>>>>> It's not a
> >>>>>>>>>>>>>>>>>>>>>>>>>> new term.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>> Andrew
> >>>>>>>>>>>>>>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>>>>>>>>>>>>>> From: PoAn Yang <pa...@apache.org>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Sent: 25 October 2024 13:55
> >>>>>>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
> >>>>>>>>>>>> kafka-consumer-groups
> >>>>>>>>>>>>>>>>>>> command
> >>>>>>>>>>>>>>>>>>>>>>>> line
> >>>>>>>>>>>>>>>>>>>>>>>>>> tool to support new consumer group
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi Lucas,
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the review!
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> 1) Yes, I add related change for
> >>>>>>> kafka-share-groups.sh
> >>>>>>>> to
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>> KIP.
> >>>>>>>>>>>>>>>>>>>>> Could
> >>>>>>>>>>>>>>>>>>>>>>>>>> you take a look? Thanks for the suggestion.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use
> >>>>>>>>> MEMBER-EPOCH
> >>>>>>>>>>>>>>> here,
> >>>>>>>>>>>>>>>>>>>>> users
> >>>>>>>>>>>>>>>>>>>>>>>> may
> >>>>>>>>>>>>>>>>>>>>>>>>>> confuse what is different between CONSUMER and
> >>>>>>> MEMBER.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Frank,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> thanks for the KIP!
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 1) For consistency, should we do the same for
> >>>>>>>>>>>>>>>>>>>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription,
> >>>> etc. ?
> >>>>>>>>> Even
> >>>>>>>>>>>> if
> >>>>>>>>>>>>>>> we do
> >>>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>>>> implement it right now if the share group
> >>>>>>>> implementation
> >>>>>>>>>>>> may
> >>>>>>>>>>>>>>> still
> >>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>> incomplete, it may make sense to include it in
> >> the
> >>>>>>>> KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not
> MEMBER-EPOCH?
> >>>>>>> That
> >>>>>>>>> would
> >>>>>>>>>>>>>>> seem
> >>>>>>>>>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>>>>>>>>>>> consistent.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Lucas
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <
> >>>>>>>>>>>>>>> yangpoan@gmail.com>
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to kick off the discussion of
> >>>>>>> KIP-1099.
> >>>>>>>>> This
> >>>>>>>>>>>> KIP
> >>>>>>>>>>>>>>>>>>>>> enhances
> >>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups tools to include state
> >>>> which
> >>>>>>> is
> >>>>>>>>>>>>>>> introduced
> >>>>>>>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>>>>>>>>> KIP-848.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups command
> >>>> line
> >>>>>>>>> tool
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> support
> >>>>>>>>>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer group - Apache Kafka - Apache
> Software
> >>>>>>>>> Foundation
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> cwiki.apache.org
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> [image: favicon.ico]
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>
> >>
> >>
>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by David Jacot <dj...@confluent.io.INVALID>.
Hi PoAn,
Thanks for the update.
Regarding the CLI, I wonder whether we should rather have a column called
"Upgraded" which is only displayed when some members use the classic
protocol. The displayed value would be True for consumer members and False
for classic members. I think that this may be more useful for end users.
What do you think?
It is a bit unfortunate that we reused MemberDescription for share groups
in the admin client because they are unrelated fields now. I wonder whether
we should have a ShareMemberDescription. Andrew, would it be an option?
Best,
David
On Mon, Nov 25, 2024 at 3:50 PM Andrew Schofield <
andrew_schofield_jira@outlook.com> wrote:
> Hi PoAn,
> I've taken another pass over the KIP and have some more comments around
> the share
> group support.
>
> AS4: Share groups do not have the concept of a target assignment. As a
> result,
> the verbose state TARGET-ASSIGNMENT-EPOCH is a bit misleading from my
> point of view. ASSIGNMENT-EPOCH is better.
>
> AS5: Similarly for the verbose members output, the TARGET-EPOCH and
> TARGET-ASSIGNMENT are not really appropriate. Personally, I would prefer
> just MEMBER-EPOCH and ASSIGNMENT, and remove the "target" fields.
>
> Thanks,
> Andrew
>
> ________________________________________
> From: PoAn Yang <ya...@gmail.com>
> Sent: 24 November 2024 15:08
> To: dev@kafka.apache.org <de...@kafka.apache.org>
> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line
> tool to support new consumer group
>
> Hi Chia-Ping,
>
> Thanks for the suggestion.
>
> The “isClassic” field in ConsumerGroupDescribeResponse is a boolean value,
> so we can’t use it as the protocol string.
> I change the “PROTOCOL” column to “IS-CLASSIC” in kafka-consumer-groups.sh.
> It returns “true” if a member is in a classic group or it’s a classic
> member in a consumer group.
> In other cases, it returns “false”.
>
> I have a draft implementation in
> https://github.com/apache/kafka/pull/17872.
>
> Thanks,
> PoAn
>
> > On Nov 23, 2024, at 1:05 PM, Chia-Ping Tsai <ch...@gmail.com> wrote:
> >
> > hi PoAn
> >
> > `isClassic` is good to me :)
> >
> > Best,
> > Chia-Ping
> >
> > PoAn Yang <ya...@gmail.com> 於 2024年11月23日 週六 下午12:35寫道：
> >
> >> Hi Chia-Ping / David,
> >>
> >> Thanks for the great suggestion.
> >>
> >> How about using “isClassic”?
> >> The term like “isLegacy” may be equivocal in the future.
> >> For example, if we have a newer version of consumer / share consumer,
> the
> >> meaning will be different for different cases.
> >> If we use “isClassic”, for MemberDescription in ClassicGroupDescription,
> >> the value can be always “true”.
> >> In ConsumerGroupDescription, it’s “true” when it’s a classic member.
> >> In ShareGroupDescription, it’s always “false”.
> >>
> >> Thanks,
> >> PoAn
> >>
> >>> On Nov 23, 2024, at 1:02 AM, Chia-Ping Tsai <ch...@gmail.com>
> wrote:
> >>>
> >>> hi DJ
> >>>
> >>>> My goal was to have a way to tell the user that his (new) consumer
> group
> >>> may have non-upgraded members yet. I wonder if we could use a boolean
> to
> >>> signal this, e.g. isLegacy. Then, in the CLI we could have a column
> which
> >>> is only displayed when they are non upgraded members. What do you
> think?
> >>>
> >>> Yes, this is a good approach, but the downside is that the new isLegacy
> >>> field in MemberDescription would be irrelevant for classic and shared
> >>> groups. This would require thorough documentation to clarify its
> purpose.
> >>>
> >>> Additionally, we could use a boolean instead of a string for the new
> >> field
> >>> in the RPC to reduce the request size.
> >>> Best,
> >>> Chia-Ping
> >>>
> >>>
> >>>
> >>> David Jacot <dj...@confluent.io.invalid> 於 2024年11月23日 週六 上午12:41寫道：
> >>>
> >>>> Hi both,
> >>>>
> >>>> Ah, I understand what you mean now. You're saying that we have two
> >> protocol
> >>>> fields. One at the group level and one at the member level (the new
> >> one).
> >>>> The two fields are actually two different things. The former is the
> >> name of
> >>>> the embedded protocol used in the classic protocol whereas the latter
> is
> >>>> the name of the protocol/api used. This is indeed confusing.
> >>>>
> >>>> My goal was to have a way to tell the user that his (new) consumer
> group
> >>>> may have non-upgraded members yet. I wonder if we could use a boolean
> to
> >>>> signal this, e.g. isLegacy. Then, in the CLI we could have a column
> >> which
> >>>> is only displayed when they are non upgraded members. What do you
> think?
> >>>>
> >>>> If we cannot find a good way to represent this, we could drop it from
> >> this
> >>>> KIP too.
> >>>>
> >>>> Best,
> >>>> David
> >>>>
> >>>> On Fri, Nov 22, 2024 at 5:12 PM PoAn Yang <ya...@gmail.com> wrote:
> >>>>
> >>>>> Hi Chia-Ping / David,
> >>>>>
> >>>>> Thanks for the review and suggestions.
> >>>>>
> >>>>>>> I don't fully follow your comment so please excuse me if I say
> >>>> something
> >>>>>>> wrong.
> >>>>>
> >>>>> I think Chia-Ping wants to say: originally, the default protocol in
> >>>>> classic group is “consumer”.
> >>>>> After we have a new consumer group, the protocol of a classic member
> in
> >>>>> consumer group is “classic”.
> >>>>> The MemberDescription is used by both ClassicGroupDescription and
> >>>>> ConsumerGroupDescription.
> >>>>> If we add the protocol field to MemberDescription, the protocol of a
> >>>>> classic member in classic group is “consumer”, but it’s “classic” in
> >>>>> consumer group.
> >>>>> To avoid misleading word, we can add a new collection of member
> >> protocols
> >>>>> to ConsumerGroupDescription only.
> >>>>> The original goal is to show member protocol during consumer
> migration,
> >>>> so
> >>>>> we don’t need to add this information to ClassicGroupDescription.
> >>>>>
> >>>>> Please correct me if I misunderstood anything.
> >>>>>
> >>>>> Thanks,
> >>>>> PoAn
> >>>>>
> >>>>>> On Nov 22, 2024, at 7:00 PM, Chia-Ping Tsai <ch...@gmail.com>
> >>>> wrote:
> >>>>>>
> >>>>>> hi DJ
> >>>>>>
> >>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and
> >>>> change
> >>>>>> return type of ModernGroup#protocolType?
> >>>>>>
> >>>>>> This is not my comment ... sorry for the incorrect quote :(
> >>>>>>
> >>>>>>> A member in a classic group should report "classic".
> >>>>>>
> >>>>>> I completely agree. However, in this case,
> >>>>> ClassicGroupDescription#protocol
> >>>>>> returns 'consumer', while
> ClassicGroupDescription#members[x].protocol
> >>>>>> returns 'classic'. This inconsistency feels a bit strange to me.
> >>>>>>
> >>>>>> Best,
> >>>>>> Chia-Ping
> >>>>>>
> >>>>>>
> >>>>>> David Jacot <dj...@confluent.io.invalid> 於 2024年11月22日 週五
> 下午6:51寫道：
> >>>>>>
> >>>>>>> Hi Chia,
> >>>>>>>
> >>>>>>>>> LM4: I agree it’s better to use a type like GroupProtocol enum,
> but
> >>>> I
> >>>>>>> would like to align the format with kafka-groups.sh.
> >>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and
> >>>> change
> >>>>>>> return type of ModernGroup#protocolType?
> >>>>>>>
> >>>>>>> My understanding is that `GroupProtocol` is used to drive the
> config
> >>>> of
> >>>>> the
> >>>>>>> `Consumer`. Hence adding `Share` seems wrong to me. Or are you
> >>>>> referring to
> >>>>>>> another GroupProtocol enum? Perhaps GroupType enum?
> >>>>>>>
> >>>>>>>> In fact, I'm wondering if we should even add 'protocol' to
> >>>>>>> MemberDescription,
> >>>>>>> since the field has different values between classic and consumer
> >>>>> groups—it
> >>>>>>> shows 'consumer' in classic groups, whereas it shows 'classic' in
> >>>>> consumer
> >>>>>>> groups. Yes, it seems like a typo, but it's true. This
> inconsistency
> >>>>> feels
> >>>>>>> confusing to me. For example, users call
> Admin#describeConsumerGroups
> >>>>> and
> >>>>>>> see that a member has the 'consumer' protocol when it's in a
> classic
> >>>>> group.
> >>>>>>> Then, if the group is upgraded to a consumer group, calling
> >>>>>>> Admin#describeConsumerGroups again shows the protocol as 'classic'
> >> for
> >>>>> the
> >>>>>>> same member.
> >>>>>>>
> >>>>>>> I don't fully follow your comment so please excuse me if I say
> >>>> something
> >>>>>>> wrong.
> >>>>>>>
> >>>>>>> I think that it should work as follows:
> >>>>>>> * A member in a classic group should report "classic".
> >>>>>>> * A member using the consumer protocol in a consumer group should
> >>>> report
> >>>>>>> "consumer".
> >>>>>>> * A member using the classic protocol in a consumer group should
> >>>> report
> >>>>>>> "classic".
> >>>>>>> * A member in a share group should report "share".
> >>>>>>>
> >>>>>>>> If all we want to do is distinguish classic members in the
> >>>>>>> ConsumerGroupDescription, then we shouldn't modify the common
> >>>>>>> MemberDescription structure. Instead, we could add a collection to
> >>>>>>> ConsumerGroupDescription to trace members using the classic
> protocol,
> >>>>> which
> >>>>>>> would allow us to enrich the output of command-line tools.
> >>>>>>>
> >>>>>>> Personally, I believe that using different collections per protocol
> >>>>> types
> >>>>>>> is not necessary. A field in the member is more than enough.
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> DJ
> >>>>>>>
> >>>>>>> On Fri, Nov 22, 2024 at 12:01 PM Chia-Ping Tsai <
> chia7712@gmail.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> hi PoAn
> >>>>>>>>
> >>>>>>>>> LM4: I agree it’s better to use a type like GroupProtocol enum,
> but
> >>>> I
> >>>>>>>> would like to align the format with kafka-groups.sh.
> >>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and
> >>>>> change
> >>>>>>>> return type of ModernGroup#protocolType?
> >>>>>>>>
> >>>>>>>> MemberDescription is also used by ClassicGroupDescription, and in
> >>>>> classic
> >>>>>>>> groups, the protocol can be an arbitrary string, so we can't use
> an
> >>>>> enum.
> >>>>>>>>
> >>>>>>>> In fact, I'm wondering if we should even add 'protocol' to
> >>>>>>>> MemberDescription,
> >>>>>>>> since the field has different values between classic and consumer
> >>>>>>> groups—it
> >>>>>>>> shows 'consumer' in classic groups, whereas it shows 'classic' in
> >>>>>>> consumer
> >>>>>>>> groups. Yes, it seems like a typo, but it's true. This
> inconsistency
> >>>>>>> feels
> >>>>>>>> confusing to me. For example, users call
> >> Admin#describeConsumerGroups
> >>>>> and
> >>>>>>>> see that a member has the 'consumer' protocol when it's in a
> classic
> >>>>>>> group.
> >>>>>>>> Then, if the group is upgraded to a consumer group, calling
> >>>>>>>> Admin#describeConsumerGroups again shows the protocol as 'classic'
> >>>> for
> >>>>>>> the
> >>>>>>>> same member.
> >>>>>>>>
> >>>>>>>> If all we want to do is distinguish classic members in the
> >>>>>>>> ConsumerGroupDescription, then we shouldn't modify the common
> >>>>>>>> MemberDescription structure. Instead, we could add a collection to
> >>>>>>>> ConsumerGroupDescription to trace members using the classic
> >> protocol,
> >>>>>>> which
> >>>>>>>> would allow us to enrich the output of command-line tools.
> >>>>>>>> Best,
> >>>>>>>> Chia-Ping
> >>>>>>>>
> >>>>>>>> PoAn Yang <ya...@gmail.com> 於 2024年11月22日 週五 下午3:46寫道：
> >>>>>>>>
> >>>>>>>>> Hi Lianet / Jeff,
> >>>>>>>>>
> >>>>>>>>> Thanks for the review.
> >>>>>>>>>
> >>>>>>>>> LM5: The kafka-share-groups.sh also uses “—describe” to show
> >> offsets
> >>>>>>>>> information.
> >>>>>>>>> Change to use “—describe —state —verbose” to show group level
> >>>>>>> information
> >>>>>>>>> in kafka-share-groups.sh.
> >>>>>>>>>
> >>>>>>>>> LM6: Update the example.
> >>>>>>>>>
> >>>>>>>>> LM7: Add a description to mention “—verbose” is a new option in
> >>>>>>>>> ShareGroupCommandOptions.
> >>>>>>>>>
> >>>>>>>>> JK2: Yes, add the statement about format change for ASSIGNMENT
> >>>> value.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> PoAn
> >>>>>>>>>
> >>>>>>>>>> On Nov 22, 2024, at 7:35 AM, Jeff Kim <je...@apache.org>
> >>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hi PoAn,
> >>>>>>>>>>
> >>>>>>>>>> JK2: Can we state in the KIP that we will reformat the existing
> >>>>>>>>>> classic group's `--members --verbose` as well, specifically the
> >>>>>>>>>> "ASSIGNMENT" column to include the topic name?
> >>>>>>>>>>
> >>>>>>>>>> Thanks!
> >>>>>>>>>>
> >>>>>>>>>> On 2024/11/21 14:33:37 "Lianet M." wrote:
> >>>>>>>>>>> Thanks for the updates PoAn!
> >>>>>>>>>>>
> >>>>>>>>>>> LM6. Just a nit, under the --describe --state --verbose the
> >> header
> >>>>>>> is
> >>>>>>>>> fine
> >>>>>>>>>>> but the example is still missing the --state argument.
> >>>>>>>>>>>
> >>>>>>>>>>> LM7. Under the Proposed changes for kafka-share-groups.sh,
> should
> >>>> we
> >>>>>>>>>>> clearly mention that the KIP adds a new --verbose option? I
> think
> >>>>>>> it's
> >>>>>>>>>>> confusing because it's presented as if the option already
> exists
> >>>> and
> >>>>>>>>> we're
> >>>>>>>>>>> only changing the output (which is the case for
> >>>>>>> kafka-consumer-groups
> >>>>>>>>> but
> >>>>>>>>>>> not for kafka-share-groups)
> >>>>>>>>>>>
> >>>>>>>>>>> That's all on my side. Thanks!
> >>>>>>>>>>>
> >>>>>>>>>>> Lianet
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Nov 20, 2024 at 11:12 PM PoAn Yang <yangpoan@gmail.com
> >
> >>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi Lianet / Jeff,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks for review and suggestions.
> >>>>>>>>>>>>
> >>>>>>>>>>>> LM4: At group level, the protocol field is from
> >>>>>>>>> ModernGroup#protocolType.
> >>>>>>>>>>>> It’s a string.
> >>>>>>>>>>>> The value is defined in different places like
> >>>>>>>> ShareGroup#PROTOCOL_TYPE
> >>>>>>>>> and
> >>>>>>>>>>>> ConsumerProtocol#PROTOCOL_TYPE.
> >>>>>>>>>>>> In KIP-1043 <
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=305171038#KIP1043:Administrationofgroups-kafka-groups.sh
> >>>>>>>>>> ,
> >>>>>>>>>>>> the kafka-groups.sh also shows protocol like “classic” /
> >>>>>>> “consumer” /
> >>>>>>>>>>>> “share”.
> >>>>>>>>>>>> I agree it’s better to use a type like GroupProtocol enum,
> but I
> >>>>>>>> would
> >>>>>>>>>>>> like to align the format with kafka-groups.sh.
> >>>>>>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol
> >> and
> >>>>>>>>> change
> >>>>>>>>>>>> return type of ModernGroup#protocolType?
> >>>>>>>>>>>>
> >>>>>>>>>>>> LM5 / JK1: Sorry, I misunderstood LM3, so I deleted —state in
> >> the
> >>>>>>>>> title.
> >>>>>>>>>>>> You’re correct. I would like to use `—describe —state
> —verbose`
> >>>> to
> >>>>>>>> show
> >>>>>>>>>>>> group level information.
> >>>>>>>>>>>> For `—describe —verbose`, it will show output as same as
> >>>> `—describe
> >>>>>>>>>>>> —offsets —verbose`,
> >>>>>>>>>>>> because `—describe` also shows output as same as `—describe
> >>>>>>>> —offsets`.
> >>>>>>>>>>>>
> >>>>>>>>>>>> JK2: Yes, I only use consumer group in the sample output, but
> >> the
> >>>>>>>>> classic
> >>>>>>>>>>>> group will use the same format as well.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>> PoAn
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On Nov 21, 2024, at 8:28 AM, Jeff Kim <je...@apache.org>
> >>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for the KIP. I have some questions:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> JK1: (Realized this is similar to LM5) On `--describe
> >> --verbose`
> >>>>>>>>>>>> proposed changes - doesn't `--describe $group` default to
> >>>> printing
> >>>>>>>> the
> >>>>>>>>>>>> offsets? Perhaps you're referring to the `--state` argument?
> >>>> Also,
> >>>>>>>>> would
> >>>>>>>>>>>> that mean the default `--describe $group --verbose` command
> >> would
> >>>>>>> not
> >>>>>>>>> print
> >>>>>>>>>>>> the added field to `--offsets verbose` (leader-epoch) or would
> >>>> it?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> JK2: Looking at ConsumerGroupCommand.java, the existing
> >>>>>>> "ASSIGNMENT"
> >>>>>>>>>>>> column under `--members --verbose` does group the topic
> >>>> partitions
> >>>>>>> by
> >>>>>>>>> topic
> >>>>>>>>>>>> but does not prefix the grouping with the topic name like
> you're
> >>>>>>>>> proposal:
> >>>>>>>>>>>> "my_topic:0,1;new_topic:0,1". Should we do apply the same
> format
> >>>>>>> for
> >>>>>>>>> the
> >>>>>>>>>>>> classic group as well?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 2024/11/20 21:32:41 "Lianet M." wrote:
> >>>>>>>>>>>>>> Hello PoAn, just of couple more minor comments:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> LM4. Regarding the new “protocol” field added to
> >>>>>>> MemberDescription.
> >>>>>>>>>>>> Should
> >>>>>>>>>>>>>> we consider reusing the existing GroupProtocol enum instead
> of
> >>>>>>>>> String?
> >>>>>>>>>>>>>> (It’s the one we use from the consumer side to refer to the
> >>>>>>>> protocol
> >>>>>>>>> in
> >>>>>>>>>>>>>> use, just missing Share I notice).
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> LM5. Regarding the change to the output for
> >>>>>>> kafka-consumer-groups,
> >>>>>>>>> the
> >>>>>>>>>>>>>> command shown does not include the —state option, but the
> >>>> output
> >>>>>>>>> shows
> >>>>>>>>>>>>>> state info (state, #members, epochs). I would guess that we
> >>>> want
> >>>>>>> to
> >>>>>>>>>>>> modify
> >>>>>>>>>>>>>> the output only when we describe a group with the —state
> >>>> —verbose
> >>>>>>>>>>>> option,
> >>>>>>>>>>>>>> is my understanding right? If my understanding is right
> we’re
> >>>>>>> just
> >>>>>>>>>>>> missing
> >>>>>>>>>>>>>> adding the —state in the example, and the KIP does not
> >>>> introduce
> >>>>>>>> any
> >>>>>>>>>>>>>> changes to the —describe —verbose option. (If not, it would
> >>>> mean
> >>>>>>> a
> >>>>>>>>>>>> bigger
> >>>>>>>>>>>>>> change to the output of —describe —verbose which I expect is
> >>>> not
> >>>>>>>> the
> >>>>>>>>>>>>>> intention?)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Lianet
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Wed, Nov 20, 2024 at 2:12 AM PoAn Yang <
> yangpoan@gmail.com
> >>>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi Chia-Ping,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks for the review and suggestions.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> CI0: Thanks for the reminder. Update validVersions in
> >>>>>>>>>>>>>>> ConsumerGroupDescribeRequest to 0-1.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> CI1: Yes, I use `ConsumerGroupMember#useClassicProtocol` to
> >>>>>>> check
> >>>>>>>>>>>> whether
> >>>>>>>>>>>>>>> a member in consumer group uses “classic” or “consumer”
> >>>>>>> protocol.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> CI2: Yes, a member in share group always uses “share”
> >>>> protocol.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> CI3: Add a table to show meaning of “classic”, “consumer”,
> >> and
> >>>>>>>>> “share”
> >>>>>>>>>>>>>>> protocol.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> BTW, the vote thread is in
> >>>>>>>>>>>>>>>
> >>>>>>> https://lists.apache.org/thread/rb25tf75tzf4c7jqqldlo5jh9w8chsq6.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Nov 20, 2024, at 11:46 AM, Chia-Ping Tsai <
> >>>>>>>> chia7712@apache.org>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> hi PoAn
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> CI0: We have to bump the version of
> >>>>>>> ConsumerGroupDescribeRequest
> >>>>>>>>> as
> >>>>>>>>>>>>>>> well, so server can distinguish the new and old behavior.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> CI1: the type of new filed is string, so I guess you plan
> to
> >>>>>>> use
> >>>>>>>>>>>>>>> `ConsumerGroupMember#useClassicProtocol` [0] flag to return
> >>>>>>> either
> >>>>>>>>>>>>>>> "classic" or "consumer", right?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> CI2: `MemberDescription` is used by
> `ShareGroupDescription`
> >>>>>>> too,
> >>>>>>>> so
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>> new filed protocol in shared group is always "shared",
> right?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> CI3: Could you consider adding a table to show the value
> of
> >>>> the
> >>>>>>>>>>>> protocol
> >>>>>>>>>>>>>>> field in each case? Andrew has a beautiful table in
> KIP-1043
> >>>>>>> that
> >>>>>>>>>>>> lists all
> >>>>>>>>>>>>>>> possible protocol names.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> [0]
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> https://github.com/apache/kafka/blob/441a6d0b790f4a17b454caeea7588a6b90fbd9db/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/consumer/ConsumerGroupMember.java#L454
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On 2024/11/19 15:45:16 PoAn Yang wrote:
> >>>>>>>>>>>>>>>>> Hi David,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks for the reminder.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> DJ4: Add related content to the KIP.
> >>>>>>>>>>>>>>>>> Bump validVersions in ConsumerGroupDescribeResponse to
> >>>> “0-1”.
> >>>>>>>>>>>>>>>>> Add a new field “Protocol” to
> >> ConsumerGroupDescribeResponse.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Nov 19, 2024, at 3:57 PM, David Jacot
> >>>>>>>>>>>> <dj...@confluent.io.INVALID>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> DJ4: Could you please also include the required changes
> to
> >>>>>>> the
> >>>>>>>>>>>>>>>>>> ConsumerGroupDescribed API in the public interface
> >> section?
> >>>>>>> We
> >>>>>>>>>>>>>>> basically
> >>>>>>>>>>>>>>>>>> need to bump the version, add the new field, etc.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>> David
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang <
> >>>>>>> yangpoan@gmail.com>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi David,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> DJ4: Add a new field protocol to MemberDescription,
> >>>>>>>>>>>>>>>>>>> so the command line tool can show protocol information
> >>>> when
> >>>>>>>>> users
> >>>>>>>>>>>>>>> describe
> >>>>>>>>>>>>>>>>>>> members.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> If there is no further suggestion, I will start a vote
> >>>>>>> thread
> >>>>>>>>>>>> today.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Nov 15, 2024, at 11:50 PM, David Jacot
> >>>>>>>>>>>>>>> <dj...@confluent.io.INVALID>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> DJ2: Using "-" sounds good to me.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> DJ3: That seems reasonable to me.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> DJ4: Why not add it right now? I don't want to change
> >> the
> >>>>>>>>> output
> >>>>>>>>>>>> of
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> tool too many times.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>> David
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang <
> >>>>>>>> yangpoan@gmail.com>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hi David / Andrew,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks for review. Thanks Andrew for picking up
> >>>>>>>>>>>>>>> kafka-share-groups.sh
> >>>>>>>>>>>>>>>>>>>>> implementation.
> >>>>>>>>>>>>>>>>>>>>> I will handle kafka-consumer-groups.sh.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> DJ3: After discussing with @Chia-Ping Tsai, we think
> >>>> that
> >>>>>>>>> using
> >>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>> format
> >>>>>>>>>>>>>>>>>>>>> is more clear.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> The new format will be like
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Using colon(:) to concat topic name and partition
> IDs.
> >>>>>>>>>>>>>>>>>>>>> Using comma(,) to concat partition IDs within same
> >> topic
> >>>>>>>> name.
> >>>>>>>>>>>>>>>>>>>>> Using semicolon(;) to concat topic strings.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> AS3: Fix it with kafka-share-groups.sh. Thanks.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> If there is no further suggestion, I will start a
> vote
> >>>>>>>> thread
> >>>>>>>>>>>> next
> >>>>>>>>>>>>>>>>>>> Monday.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield <
> >>>>>>>>>>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>>>>>>>>>>>>> DJ2: I was just going to comment that "-" would be a
> >>>> more
> >>>>>>>>>>>>>>> appropriate
> >>>>>>>>>>>>>>>>>>>>> missing value, but
> >>>>>>>>>>>>>>>>>>>>>> you got there first.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> AS3: The examples for kafka-share-groups.sh include
> >>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups.sh in the
> >>>>>>>>>>>>>>>>>>>>>> command line.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> If this is accepted in time, I'm happy to pick up
> the
> >>>>>>>>>>>>>>> implementation of
> >>>>>>>>>>>>>>>>>>>>> the share groups
> >>>>>>>>>>>>>>>>>>>>>> part of this if it helps.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>> Andrew
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>>>>>>>>>> From: Frank Yang <ya...@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>> Sent: 14 November 2024 10:48
> >>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
> >>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
> >>>>>>>> kafka-consumer-groups
> >>>>>>>>>>>>>>> command
> >>>>>>>>>>>>>>>>>>>>> line tool to support new consumer group
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Hi David,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Thanks for the review and suggestion! I would like
> to
> >>>> get
> >>>>>>>>> this
> >>>>>>>>>>>> in
> >>>>>>>>>>>>>>> AK
> >>>>>>>>>>>>>>>>>>> 4.0
> >>>>>>>>>>>>>>>>>>>>> as well. I will do my best.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> DJ1: Update KIP to put GROUP-EPOCH and
> >>>>>>>>> TARGET-ASSIGNMENT-EPOCH
> >>>>>>>>>>>>>>> before
> >>>>>>>>>>>>>>>>>>>>> #MEMBERS.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> DJ2: I prefer to follow current missing column value
> >>>> “-“.
> >>>>>>>>>>>>>>> (reference <
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92
> >>>>>>>>>>>>>>>>>>>>>> )
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> DJ3: Update KIP to use CURRENT-EPOCH
> >> CURRENT-ASSIGNMENT
> >>>>>>>>>>>>>>> TARGET-EPOCH
> >>>>>>>>>>>>>>>>>>>>> TARGET-ASSIGNMENT.
> >>>>>>>>>>>>>>>>>>>>>> Remove GROUP-EPOCH.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> For assignment value, it follows current output
> >>>>>>> (reference
> >>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418
> >>>>>>>>>>>>>>>>>>>> ).
> >>>>>>>>>>>>>>>>>>>>> I think the form of `topicid-partitionid` is more
> >> clear.
> >>>>>>>>>>>>>>>>>>>>>> If we would like to use this form, I will update
> both
> >>>>>>>> output
> >>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups.sh and kafka-share-groups.sh.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> DJ4: It looks like DescribeGroupsResponseData only
> has
> >>>>>>>>> protocol
> >>>>>>>>>>>>>>> type at
> >>>>>>>>>>>>>>>>>>>>> group level.
> >>>>>>>>>>>>>>>>>>>>>> Both DescribeGroupsResponseData and
> >>>>>>>>>>>>>>> ConsumerGroupDescribeResponseData
> >>>>>>>>>>>>>>>>>>>>> don’t have protocol at member level.
> >>>>>>>>>>>>>>>>>>>>>> Could we use a followup to add it?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> DJ5: Update KIP to put LEADER-EPOCH before
> >>>>>>> CURRENT-OFFSET.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot
> >>>>>>>>>>>>>>> <djacot@confluent.io.INVALID
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP! I have a few minor
> >>>>>>>> comments/suggestions:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> DJ1: In the output of `--describe --verbose`, I
> would
> >>>>>>>>> suggest
> >>>>>>>>>>>>>>> putting
> >>>>>>>>>>>>>>>>>>>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before
> >>>>>>>>> `#MEMBERS`.
> >>>>>>>>>>>>>>>>>>>>>>> DJ2: Continuing on the above, I assume that we will
> >>>>>>> print
> >>>>>>>>> out
> >>>>>>>>>>>> N/A
> >>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> fields not supported by classic groups. Is this
> >>>> correct?
> >>>>>>>>>>>>>>>>>>>>>>> DJ3: In the output of `--describe --members
> >>>> --verbose`,
> >>>>>>> I
> >>>>>>>>>>>> wonder
> >>>>>>>>>>>>>>> if we
> >>>>>>>>>>>>>>>>>>>>>>> should use the following order and terms:
> >>>> CURRENT-EPOCH
> >>>>>>>>>>>>>>>>>>>>> CURRENT-ASSIGNMENT
> >>>>>>>>>>>>>>>>>>>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the
> >>>>>>>>> GROUP-EPOCH
> >>>>>>>>>>>>>>>>>>> because
> >>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>> is already in the group description. The value
> `(0)`
> >>>>>>> used
> >>>>>>>>> for
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> assignment is incorrect. Here I suppose that we
> will
> >>>>>>> print
> >>>>>>>>> out
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> list
> >>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>> partitions in the form of `topicid-partitionid`.
> >>>>>>>>>>>>>>>>>>>>>>> DJ4: Continuing on the above, I wonder if we should
> >>>> also
> >>>>>>>> add
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> protocol
> >>>>>>>>>>>>>>>>>>>>>>> used `classic` or `consumer`. For context, it is
> >>>>>>> possible
> >>>>>>>> to
> >>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>>>> `classic`
> >>>>>>>>>>>>>>>>>>>>>>> members and `consumer` members in a `consumer`
> group
> >>>>>>>> during
> >>>>>>>>> an
> >>>>>>>>>>>>>>> online
> >>>>>>>>>>>>>>>>>>>>>>> upgrade from the classic protocol to the consumer
> >>>>>>>> protocol.
> >>>>>>>>>>>> Having
> >>>>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>>> information may be handy for administrators. What
> do
> >>>> you
> >>>>>>>>> think?
> >>>>>>>>>>>>>>>>>>>>>>> DJ5: In the output of `--describe --offsets
> >>>> --verbose`,
> >>>>>>> I
> >>>>>>>>> would
> >>>>>>>>>>>>>>>>>>> suggest
> >>>>>>>>>>>>>>>>>>>>>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> It would be great if we could get this one in AK
> 4.0
> >>>> as
> >>>>>>> it
> >>>>>>>>>>>>>>> changes the
> >>>>>>>>>>>>>>>>>>>>>>> output of the command.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>> DJ
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <
> >>>>>>>>> yangpoan@gmail.com>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Hi Sean / Andrew / Lianet,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks for all your review and suggestions.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> AS1, LM1, LM4: Change to add KIP-848 information
> >> when
> >>>>>>>> users
> >>>>>>>>>>>> give
> >>>>>>>>>>>>>>>>>>>>> —verbose
> >>>>>>>>>>>>>>>>>>>>>>>> option.
> >>>>>>>>>>>>>>>>>>>>>>>> —describe —verbose: shows group epoch and target
> >>>>>>>> assignment
> >>>>>>>>>>>>>>> epoch.
> >>>>>>>>>>>>>>>>>>>>>>>> —describe —members —verbose: shows above
> >> information,
> >>>>>>>>> member
> >>>>>>>>>>>>>>> epoch,
> >>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>> target assignment.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> AS2: Change to use MEMBER-EPOCH to align with
> >> KIP-848
> >>>>>>>>>>>> definition.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> LM2: For classic group, it doesn’t have epoch, so
> I
> >>>> use
> >>>>>>>>>>>> Optional
> >>>>>>>>>>>>>>>>>>>>> fields in
> >>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupDescription.
> >>>>>>>>>>>>>>>>>>>>>>>> For share group, it relies on KIP-848. It must
> have
> >>>>>>>> epoch,
> >>>>>>>>> so
> >>>>>>>>>>>> I
> >>>>>>>>>>>>>>> use
> >>>>>>>>>>>>>>>>>>> int
> >>>>>>>>>>>>>>>>>>>>>>>> fields in ShareGroupDescription.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> LM3: Remove —state change. We can get group level
> >>>>>>>>> information
> >>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>>>> —describe
> >>>>>>>>>>>>>>>>>>>>>>>> —verbose.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> SQ1: Add LEADER-EPOCH when users give —describe
> >>>>>>> —offsets
> >>>>>>>>>>>>>>> —verbose.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks.
> >>>>>>>>>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <
> >>>>>>>> lianetmr@gmail.com
> >>>>>>>>>>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Hello Frank, thanks for the KIP! A few comments:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> LM1. I strongly agree with Andrew's suggestion of
> >>>>>>> moving
> >>>>>>>>> this
> >>>>>>>>>>>>>>> into a
> >>>>>>>>>>>>>>>>>>>>>>>>> --verbose option. I expect someone would only
> >>>> attempt
> >>>>>>> to
> >>>>>>>>> make
> >>>>>>>>>>>>>>> sense
> >>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> epochs while debugging an issue, not while trying
> >> to
> >>>>>>>> view
> >>>>>>>>> the
> >>>>>>>>>>>>>>> group
> >>>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>>>> member's info. (Also member-epoch makes more
> sense
> >>>> to
> >>>>>>>> me,
> >>>>>>>>>>>> even
> >>>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>>>>> if we
> >>>>>>>>>>>>>>>>>>>>>>>>> end up agreeing in a --verbose). Related to both
> >>>>>>> issues,
> >>>>>>>>> my
> >>>>>>>>>>>>>>> take is
> >>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>> whoever is close to the protocol will expect
> >>>>>>>> member-epoch,
> >>>>>>>>>>>>>>> whoever
> >>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>> will probably won't even need to see the epochs
> at
> >>>>>>> all.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> LM2. Why are the epochs defined as Optional
> (don't
> >>>> we
> >>>>>>>>> expect
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> always
> >>>>>>>>>>>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>>>>>>>> them? with 0 initially, defined on the broker
> side
> >>>> for
> >>>>>>>> the
> >>>>>>>>>>>> group
> >>>>>>>>>>>>>>>>>>> ones,
> >>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>> client side for the member)
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> LM3. Why is the KIP including the –state option
> in
> >>>> the
> >>>>>>>>>>>> proposed
> >>>>>>>>>>>>>>>>>>>>> changes?
> >>>>>>>>>>>>>>>>>>>>>>>> (
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
> >>>>>>>>>>>>>>>>>>>>>>>>> ).
> >>>>>>>>>>>>>>>>>>>>>>>>> I get that the output would change in that
> example,
> >>>>>>> but
> >>>>>>>>> it’s
> >>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>> because
> >>>>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>> any change to the –state option itself. It's
> >> because
> >>>>>>> of
> >>>>>>>>> the
> >>>>>>>>>>>>>>> change
> >>>>>>>>>>>>>>>>>>>>>>>> proposed
> >>>>>>>>>>>>>>>>>>>>>>>>> to the –described (required with the --state),
> and
> >>>> the
> >>>>>>>>>>>> changes
> >>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>> --describe are already explained above (seems
> >>>>>>> confusing,
> >>>>>>>>> got
> >>>>>>>>>>>> me
> >>>>>>>>>>>>>>>>>>>>> looking
> >>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>> a change to the state filter which seemed
> >>>> unrelated).
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> LM4. Group epoch and target assignment epoch are
> >>>>>>>>> conceptually
> >>>>>>>>>>>>>>> at the
> >>>>>>>>>>>>>>>>>>>>>>>> group
> >>>>>>>>>>>>>>>>>>>>>>>>> level. vs member epoch that lands at a member
> >> level.
> >>>>>>> So
> >>>>>>>>>>>> wonder
> >>>>>>>>>>>>>>> if we
> >>>>>>>>>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>>>>>>> show them accordingly (ex. using the --verbose
> >>>> option)
> >>>>>>>>>>>>>>>>>>>>>>>>> –describe –verbose => shows group epoch and
> target
> >>>>>>>>> assignment
> >>>>>>>>>>>>>>> epoch
> >>>>>>>>>>>>>>>>>>>>>>>>> –describe –members –verbose => shows member
> >>>>>>>> epoch
> >>>>>>>>>>>>>>> (along
> >>>>>>>>>>>>>>>>>>>>>>>>> with group epoch and target assignment epoch)
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Lianet
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield
> <
> >>>>>>>>>>>>>>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I have a few comments.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> AS1: It seems to me that these new additions are
> >>>> most
> >>>>>>>>>>>> useful to
> >>>>>>>>>>>>>>>>>>>>> people
> >>>>>>>>>>>>>>>>>>>>>>>>>> trying to understand
> >>>>>>>>>>>>>>>>>>>>>>>>>> the progress of rebalancing in quite some
> detail.
> >>>> The
> >>>>>>>>>>>>>>> information
> >>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>> really not understandable
> >>>>>>>>>>>>>>>>>>>>>>>>>> for most users who do not have deep knowledge of
> >>>>>>>>>>>> KIP-848/932.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> As a result, I suggest for kafka-share-groups.sh
> >>>> that
> >>>>>>>> you
> >>>>>>>>>>>> add a
> >>>>>>>>>>>>>>>>>>>>>>>> --members
> >>>>>>>>>>>>>>>>>>>>>>>>>> --verbose option
> >>>>>>>>>>>>>>>>>>>>>>>>>> and only include the new information in the
> output
> >>>>>>> for
> >>>>>>>>> that
> >>>>>>>>>>>>>>> option,
> >>>>>>>>>>>>>>>>>>>>>>>> rather
> >>>>>>>>>>>>>>>>>>>>>>>>>> than changing the
> >>>>>>>>>>>>>>>>>>>>>>>>>> non-verbose --members output.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> I also make a similar suggestion for
> >>>>>>>>>>>> kafka-consumer-groups.sh
> >>>>>>>>>>>>>>>>>>>>> --members
> >>>>>>>>>>>>>>>>>>>>>>>>>> and only add the
> >>>>>>>>>>>>>>>>>>>>>>>>>> new information for the --verbose output.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> AS2: I strongly suggest that you use
> MEMBER-EPOCH
> >>>>>>>>> instead of
> >>>>>>>>>>>>>>>>>>>>>>>>>> CONSUMER-EPOCH.
> >>>>>>>>>>>>>>>>>>>>>>>>>> I think it's more confusing not following the
> >>>>>>>>> terminology of
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> KIPs.
> >>>>>>>>>>>>>>>>>>>>>>>> For
> >>>>>>>>>>>>>>>>>>>>>>>>>> one thing,
> >>>>>>>>>>>>>>>>>>>>>>>>>> we already have "member" in the admin client
> such
> >>>> as
> >>>>>>>>>>>>>>>>>>>>> MemberDescription.
> >>>>>>>>>>>>>>>>>>>>>>>>>> It's not a
> >>>>>>>>>>>>>>>>>>>>>>>>>> new term.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>> Andrew
> >>>>>>>>>>>>>>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>>>>>>>>>>>>>> From: PoAn Yang <pa...@apache.org>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Sent: 25 October 2024 13:55
> >>>>>>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
> >>>>>>>>>>>> kafka-consumer-groups
> >>>>>>>>>>>>>>>>>>> command
> >>>>>>>>>>>>>>>>>>>>>>>> line
> >>>>>>>>>>>>>>>>>>>>>>>>>> tool to support new consumer group
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi Lucas,
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the review!
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> 1) Yes, I add related change for
> >>>>>>> kafka-share-groups.sh
> >>>>>>>> to
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>> KIP.
> >>>>>>>>>>>>>>>>>>>>> Could
> >>>>>>>>>>>>>>>>>>>>>>>>>> you take a look? Thanks for the suggestion.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use
> >>>>>>>>> MEMBER-EPOCH
> >>>>>>>>>>>>>>> here,
> >>>>>>>>>>>>>>>>>>>>> users
> >>>>>>>>>>>>>>>>>>>>>>>> may
> >>>>>>>>>>>>>>>>>>>>>>>>>> confuse what is different between CONSUMER and
> >>>>>>> MEMBER.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Frank,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> thanks for the KIP!
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 1) For consistency, should we do the same for
> >>>>>>>>>>>>>>>>>>>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription,
> >>>> etc. ?
> >>>>>>>>> Even
> >>>>>>>>>>>> if
> >>>>>>>>>>>>>>> we do
> >>>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>>>> implement it right now if the share group
> >>>>>>>> implementation
> >>>>>>>>>>>> may
> >>>>>>>>>>>>>>> still
> >>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>> incomplete, it may make sense to include it in
> >> the
> >>>>>>>> KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not
> MEMBER-EPOCH?
> >>>>>>> That
> >>>>>>>>> would
> >>>>>>>>>>>>>>> seem
> >>>>>>>>>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>>>>>>>>>>> consistent.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Lucas
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <
> >>>>>>>>>>>>>>> yangpoan@gmail.com>
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to kick off the discussion of
> >>>>>>> KIP-1099.
> >>>>>>>>> This
> >>>>>>>>>>>> KIP
> >>>>>>>>>>>>>>>>>>>>> enhances
> >>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups tools to include state
> >>>> which
> >>>>>>> is
> >>>>>>>>>>>>>>> introduced
> >>>>>>>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>>>>>>>>> KIP-848.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups command
> >>>> line
> >>>>>>>>> tool
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> support
> >>>>>>>>>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer group - Apache Kafka - Apache
> Software
> >>>>>>>>> Foundation
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> cwiki.apache.org
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> [image: favicon.ico]
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>
> >>
> >>
>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by David Jacot <dj...@confluent.io.INVALID>.
Thanks, LGTM.
On Mon, Nov 25, 2024 at 4:26 PM PoAn Yang <ya...@gmail.com> wrote:
> Hi David,
>
> Thanks for the feedback.
>
> Yes, I agree that the “UPGRADED” column name is better. I updated the KIP.
>
> Best,
> PoAn
>
> > On Nov 25, 2024, at 11:11 PM, PoAn Yang <ya...@gmail.com> wrote:
> >
> > Hi Andrew,
> >
> > Thanks for the feedback.
> >
> > AS4: Change TARGET-ASSIGNMENT-EPOCH to ASSIGNMENT-EPOCH for “—describe
> —state —verbose" in kafka-share-groups.sh.
> >
> > AS5: Remove TARGET-EPOCH and TARGET-ASSIGNMENT.
> > Change CURRENT-EPOCH to MEMBER-EPOCH and CURRENT-ASSIGNMENT to
> ASSIGNMENT for “—describe —member —verbose” in kafka-share-groups.sh.
> >
> > Thanks,
> > PoAn
> >
> >> On Nov 25, 2024, at 10:48 PM, Andrew Schofield <
> andrew_schofield_jira@outlook.com> wrote:
> >>
> >> Hi PoAn,
> >> I've taken another pass over the KIP and have some more comments around
> the share
> >> group support.
> >>
> >> AS4: Share groups do not have the concept of a target assignment. As a
> result,
> >> the verbose state TARGET-ASSIGNMENT-EPOCH is a bit misleading from my
> >> point of view. ASSIGNMENT-EPOCH is better.
> >>
> >> AS5: Similarly for the verbose members output, the TARGET-EPOCH and
> >> TARGET-ASSIGNMENT are not really appropriate. Personally, I would prefer
> >> just MEMBER-EPOCH and ASSIGNMENT, and remove the "target" fields.
> >>
> >> Thanks,
> >> Andrew
> >>
> >> ________________________________________
> >> From: PoAn Yang <ya...@gmail.com>
> >> Sent: 24 November 2024 15:08
> >> To: dev@kafka.apache.org <de...@kafka.apache.org>
> >> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command
> line tool to support new consumer group
> >>
> >> Hi Chia-Ping,
> >>
> >> Thanks for the suggestion.
> >>
> >> The “isClassic” field in ConsumerGroupDescribeResponse is a boolean
> value, so we can’t use it as the protocol string.
> >> I change the “PROTOCOL” column to “IS-CLASSIC” in
> kafka-consumer-groups.sh.
> >> It returns “true” if a member is in a classic group or it’s a classic
> member in a consumer group.
> >> In other cases, it returns “false”.
> >>
> >> I have a draft implementation in
> https://github.com/apache/kafka/pull/17872.
> >>
> >> Thanks,
> >> PoAn
> >>
> >>> On Nov 23, 2024, at 1:05 PM, Chia-Ping Tsai <ch...@gmail.com>
> wrote:
> >>>
> >>> hi PoAn
> >>>
> >>> `isClassic` is good to me :)
> >>>
> >>> Best,
> >>> Chia-Ping
> >>>
> >>> PoAn Yang <ya...@gmail.com> 於 2024年11月23日 週六 下午12:35寫道：
> >>>
> >>>> Hi Chia-Ping / David,
> >>>>
> >>>> Thanks for the great suggestion.
> >>>>
> >>>> How about using “isClassic”?
> >>>> The term like “isLegacy” may be equivocal in the future.
> >>>> For example, if we have a newer version of consumer / share consumer,
> the
> >>>> meaning will be different for different cases.
> >>>> If we use “isClassic”, for MemberDescription in
> ClassicGroupDescription,
> >>>> the value can be always “true”.
> >>>> In ConsumerGroupDescription, it’s “true” when it’s a classic member.
> >>>> In ShareGroupDescription, it’s always “false”.
> >>>>
> >>>> Thanks,
> >>>> PoAn
> >>>>
> >>>>> On Nov 23, 2024, at 1:02 AM, Chia-Ping Tsai <ch...@gmail.com>
> wrote:
> >>>>>
> >>>>> hi DJ
> >>>>>
> >>>>>> My goal was to have a way to tell the user that his (new) consumer
> group
> >>>>> may have non-upgraded members yet. I wonder if we could use a
> boolean to
> >>>>> signal this, e.g. isLegacy. Then, in the CLI we could have a column
> which
> >>>>> is only displayed when they are non upgraded members. What do you
> think?
> >>>>>
> >>>>> Yes, this is a good approach, but the downside is that the new
> isLegacy
> >>>>> field in MemberDescription would be irrelevant for classic and shared
> >>>>> groups. This would require thorough documentation to clarify its
> purpose.
> >>>>>
> >>>>> Additionally, we could use a boolean instead of a string for the new
> >>>> field
> >>>>> in the RPC to reduce the request size.
> >>>>> Best,
> >>>>> Chia-Ping
> >>>>>
> >>>>>
> >>>>>
> >>>>> David Jacot <dj...@confluent.io.invalid> 於 2024年11月23日 週六
> 上午12:41寫道：
> >>>>>
> >>>>>> Hi both,
> >>>>>>
> >>>>>> Ah, I understand what you mean now. You're saying that we have two
> >>>> protocol
> >>>>>> fields. One at the group level and one at the member level (the new
> >>>> one).
> >>>>>> The two fields are actually two different things. The former is the
> >>>> name of
> >>>>>> the embedded protocol used in the classic protocol whereas the
> latter is
> >>>>>> the name of the protocol/api used. This is indeed confusing.
> >>>>>>
> >>>>>> My goal was to have a way to tell the user that his (new) consumer
> group
> >>>>>> may have non-upgraded members yet. I wonder if we could use a
> boolean to
> >>>>>> signal this, e.g. isLegacy. Then, in the CLI we could have a column
> >>>> which
> >>>>>> is only displayed when they are non upgraded members. What do you
> think?
> >>>>>>
> >>>>>> If we cannot find a good way to represent this, we could drop it
> from
> >>>> this
> >>>>>> KIP too.
> >>>>>>
> >>>>>> Best,
> >>>>>> David
> >>>>>>
> >>>>>> On Fri, Nov 22, 2024 at 5:12 PM PoAn Yang <ya...@gmail.com>
> wrote:
> >>>>>>
> >>>>>>> Hi Chia-Ping / David,
> >>>>>>>
> >>>>>>> Thanks for the review and suggestions.
> >>>>>>>
> >>>>>>>>> I don't fully follow your comment so please excuse me if I say
> >>>>>> something
> >>>>>>>>> wrong.
> >>>>>>>
> >>>>>>> I think Chia-Ping wants to say: originally, the default protocol in
> >>>>>>> classic group is “consumer”.
> >>>>>>> After we have a new consumer group, the protocol of a classic
> member in
> >>>>>>> consumer group is “classic”.
> >>>>>>> The MemberDescription is used by both ClassicGroupDescription and
> >>>>>>> ConsumerGroupDescription.
> >>>>>>> If we add the protocol field to MemberDescription, the protocol of
> a
> >>>>>>> classic member in classic group is “consumer”, but it’s “classic”
> in
> >>>>>>> consumer group.
> >>>>>>> To avoid misleading word, we can add a new collection of member
> >>>> protocols
> >>>>>>> to ConsumerGroupDescription only.
> >>>>>>> The original goal is to show member protocol during consumer
> migration,
> >>>>>> so
> >>>>>>> we don’t need to add this information to ClassicGroupDescription.
> >>>>>>>
> >>>>>>> Please correct me if I misunderstood anything.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> PoAn
> >>>>>>>
> >>>>>>>> On Nov 22, 2024, at 7:00 PM, Chia-Ping Tsai <ch...@gmail.com>
> >>>>>> wrote:
> >>>>>>>>
> >>>>>>>> hi DJ
> >>>>>>>>
> >>>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and
> >>>>>> change
> >>>>>>>> return type of ModernGroup#protocolType?
> >>>>>>>>
> >>>>>>>> This is not my comment ... sorry for the incorrect quote :(
> >>>>>>>>
> >>>>>>>>> A member in a classic group should report "classic".
> >>>>>>>>
> >>>>>>>> I completely agree. However, in this case,
> >>>>>>> ClassicGroupDescription#protocol
> >>>>>>>> returns 'consumer', while
> ClassicGroupDescription#members[x].protocol
> >>>>>>>> returns 'classic'. This inconsistency feels a bit strange to me.
> >>>>>>>>
> >>>>>>>> Best,
> >>>>>>>> Chia-Ping
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> David Jacot <dj...@confluent.io.invalid> 於 2024年11月22日 週五
> 下午6:51寫道：
> >>>>>>>>
> >>>>>>>>> Hi Chia,
> >>>>>>>>>
> >>>>>>>>>>> LM4: I agree it’s better to use a type like GroupProtocol
> enum, but
> >>>>>> I
> >>>>>>>>> would like to align the format with kafka-groups.sh.
> >>>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and
> >>>>>> change
> >>>>>>>>> return type of ModernGroup#protocolType?
> >>>>>>>>>
> >>>>>>>>> My understanding is that `GroupProtocol` is used to drive the
> config
> >>>>>> of
> >>>>>>> the
> >>>>>>>>> `Consumer`. Hence adding `Share` seems wrong to me. Or are you
> >>>>>>> referring to
> >>>>>>>>> another GroupProtocol enum? Perhaps GroupType enum?
> >>>>>>>>>
> >>>>>>>>>> In fact, I'm wondering if we should even add 'protocol' to
> >>>>>>>>> MemberDescription,
> >>>>>>>>> since the field has different values between classic and consumer
> >>>>>>> groups—it
> >>>>>>>>> shows 'consumer' in classic groups, whereas it shows 'classic' in
> >>>>>>> consumer
> >>>>>>>>> groups. Yes, it seems like a typo, but it's true. This
> inconsistency
> >>>>>>> feels
> >>>>>>>>> confusing to me. For example, users call
> Admin#describeConsumerGroups
> >>>>>>> and
> >>>>>>>>> see that a member has the 'consumer' protocol when it's in a
> classic
> >>>>>>> group.
> >>>>>>>>> Then, if the group is upgraded to a consumer group, calling
> >>>>>>>>> Admin#describeConsumerGroups again shows the protocol as
> 'classic'
> >>>> for
> >>>>>>> the
> >>>>>>>>> same member.
> >>>>>>>>>
> >>>>>>>>> I don't fully follow your comment so please excuse me if I say
> >>>>>> something
> >>>>>>>>> wrong.
> >>>>>>>>>
> >>>>>>>>> I think that it should work as follows:
> >>>>>>>>> * A member in a classic group should report "classic".
> >>>>>>>>> * A member using the consumer protocol in a consumer group should
> >>>>>> report
> >>>>>>>>> "consumer".
> >>>>>>>>> * A member using the classic protocol in a consumer group should
> >>>>>> report
> >>>>>>>>> "classic".
> >>>>>>>>> * A member in a share group should report "share".
> >>>>>>>>>
> >>>>>>>>>> If all we want to do is distinguish classic members in the
> >>>>>>>>> ConsumerGroupDescription, then we shouldn't modify the common
> >>>>>>>>> MemberDescription structure. Instead, we could add a collection
> to
> >>>>>>>>> ConsumerGroupDescription to trace members using the classic
> protocol,
> >>>>>>> which
> >>>>>>>>> would allow us to enrich the output of command-line tools.
> >>>>>>>>>
> >>>>>>>>> Personally, I believe that using different collections per
> protocol
> >>>>>>> types
> >>>>>>>>> is not necessary. A field in the member is more than enough.
> >>>>>>>>>
> >>>>>>>>> Best,
> >>>>>>>>> DJ
> >>>>>>>>>
> >>>>>>>>> On Fri, Nov 22, 2024 at 12:01 PM Chia-Ping Tsai <
> chia7712@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> hi PoAn
> >>>>>>>>>>
> >>>>>>>>>>> LM4: I agree it’s better to use a type like GroupProtocol
> enum, but
> >>>>>> I
> >>>>>>>>>> would like to align the format with kafka-groups.sh.
> >>>>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol
> and
> >>>>>>> change
> >>>>>>>>>> return type of ModernGroup#protocolType?
> >>>>>>>>>>
> >>>>>>>>>> MemberDescription is also used by ClassicGroupDescription, and
> in
> >>>>>>> classic
> >>>>>>>>>> groups, the protocol can be an arbitrary string, so we can't
> use an
> >>>>>>> enum.
> >>>>>>>>>>
> >>>>>>>>>> In fact, I'm wondering if we should even add 'protocol' to
> >>>>>>>>>> MemberDescription,
> >>>>>>>>>> since the field has different values between classic and
> consumer
> >>>>>>>>> groups—it
> >>>>>>>>>> shows 'consumer' in classic groups, whereas it shows 'classic'
> in
> >>>>>>>>> consumer
> >>>>>>>>>> groups. Yes, it seems like a typo, but it's true. This
> inconsistency
> >>>>>>>>> feels
> >>>>>>>>>> confusing to me. For example, users call
> >>>> Admin#describeConsumerGroups
> >>>>>>> and
> >>>>>>>>>> see that a member has the 'consumer' protocol when it's in a
> classic
> >>>>>>>>> group.
> >>>>>>>>>> Then, if the group is upgraded to a consumer group, calling
> >>>>>>>>>> Admin#describeConsumerGroups again shows the protocol as
> 'classic'
> >>>>>> for
> >>>>>>>>> the
> >>>>>>>>>> same member.
> >>>>>>>>>>
> >>>>>>>>>> If all we want to do is distinguish classic members in the
> >>>>>>>>>> ConsumerGroupDescription, then we shouldn't modify the common
> >>>>>>>>>> MemberDescription structure. Instead, we could add a collection
> to
> >>>>>>>>>> ConsumerGroupDescription to trace members using the classic
> >>>> protocol,
> >>>>>>>>> which
> >>>>>>>>>> would allow us to enrich the output of command-line tools.
> >>>>>>>>>> Best,
> >>>>>>>>>> Chia-Ping
> >>>>>>>>>>
> >>>>>>>>>> PoAn Yang <ya...@gmail.com> 於 2024年11月22日 週五 下午3:46寫道：
> >>>>>>>>>>
> >>>>>>>>>>> Hi Lianet / Jeff,
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for the review.
> >>>>>>>>>>>
> >>>>>>>>>>> LM5: The kafka-share-groups.sh also uses “—describe” to show
> >>>> offsets
> >>>>>>>>>>> information.
> >>>>>>>>>>> Change to use “—describe —state —verbose” to show group level
> >>>>>>>>> information
> >>>>>>>>>>> in kafka-share-groups.sh.
> >>>>>>>>>>>
> >>>>>>>>>>> LM6: Update the example.
> >>>>>>>>>>>
> >>>>>>>>>>> LM7: Add a description to mention “—verbose” is a new option in
> >>>>>>>>>>> ShareGroupCommandOptions.
> >>>>>>>>>>>
> >>>>>>>>>>> JK2: Yes, add the statement about format change for ASSIGNMENT
> >>>>>> value.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>> PoAn
> >>>>>>>>>>>
> >>>>>>>>>>>> On Nov 22, 2024, at 7:35 AM, Jeff Kim <je...@apache.org>
> >>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>>>
> >>>>>>>>>>>> JK2: Can we state in the KIP that we will reformat the
> existing
> >>>>>>>>>>>> classic group's `--members --verbose` as well, specifically
> the
> >>>>>>>>>>>> "ASSIGNMENT" column to include the topic name?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>
> >>>>>>>>>>>> On 2024/11/21 14:33:37 "Lianet M." wrote:
> >>>>>>>>>>>>> Thanks for the updates PoAn!
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> LM6. Just a nit, under the --describe --state --verbose the
> >>>> header
> >>>>>>>>> is
> >>>>>>>>>>> fine
> >>>>>>>>>>>>> but the example is still missing the --state argument.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> LM7. Under the Proposed changes for kafka-share-groups.sh,
> should
> >>>>>> we
> >>>>>>>>>>>>> clearly mention that the KIP adds a new --verbose option? I
> think
> >>>>>>>>> it's
> >>>>>>>>>>>>> confusing because it's presented as if the option already
> exists
> >>>>>> and
> >>>>>>>>>>> we're
> >>>>>>>>>>>>> only changing the output (which is the case for
> >>>>>>>>> kafka-consumer-groups
> >>>>>>>>>>> but
> >>>>>>>>>>>>> not for kafka-share-groups)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> That's all on my side. Thanks!
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Lianet
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Wed, Nov 20, 2024 at 11:12 PM PoAn Yang <
> yangpoan@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi Lianet / Jeff,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks for review and suggestions.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> LM4: At group level, the protocol field is from
> >>>>>>>>>>> ModernGroup#protocolType.
> >>>>>>>>>>>>>> It’s a string.
> >>>>>>>>>>>>>> The value is defined in different places like
> >>>>>>>>>> ShareGroup#PROTOCOL_TYPE
> >>>>>>>>>>> and
> >>>>>>>>>>>>>> ConsumerProtocol#PROTOCOL_TYPE.
> >>>>>>>>>>>>>> In KIP-1043 <
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=305171038#KIP1043:Administrationofgroups-kafka-groups.sh
> >>>>>>>>>>>> ,
> >>>>>>>>>>>>>> the kafka-groups.sh also shows protocol like “classic” /
> >>>>>>>>> “consumer” /
> >>>>>>>>>>>>>> “share”.
> >>>>>>>>>>>>>> I agree it’s better to use a type like GroupProtocol enum,
> but I
> >>>>>>>>>> would
> >>>>>>>>>>>>>> like to align the format with kafka-groups.sh.
> >>>>>>>>>>>>>> How about we use another Jira to add “SHARE” to
> GroupProtocol
> >>>> and
> >>>>>>>>>>> change
> >>>>>>>>>>>>>> return type of ModernGroup#protocolType?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> LM5 / JK1: Sorry, I misunderstood LM3, so I deleted —state
> in
> >>>> the
> >>>>>>>>>>> title.
> >>>>>>>>>>>>>> You’re correct. I would like to use `—describe —state
> —verbose`
> >>>>>> to
> >>>>>>>>>> show
> >>>>>>>>>>>>>> group level information.
> >>>>>>>>>>>>>> For `—describe —verbose`, it will show output as same as
> >>>>>> `—describe
> >>>>>>>>>>>>>> —offsets —verbose`,
> >>>>>>>>>>>>>> because `—describe` also shows output as same as `—describe
> >>>>>>>>>> —offsets`.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> JK2: Yes, I only use consumer group in the sample output,
> but
> >>>> the
> >>>>>>>>>>> classic
> >>>>>>>>>>>>>> group will use the same format as well.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Nov 21, 2024, at 8:28 AM, Jeff Kim <
> jeffkbkim@apache.org>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks for the KIP. I have some questions:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> JK1: (Realized this is similar to LM5) On `--describe
> >>>> --verbose`
> >>>>>>>>>>>>>> proposed changes - doesn't `--describe $group` default to
> >>>>>> printing
> >>>>>>>>>> the
> >>>>>>>>>>>>>> offsets? Perhaps you're referring to the `--state` argument?
> >>>>>> Also,
> >>>>>>>>>>> would
> >>>>>>>>>>>>>> that mean the default `--describe $group --verbose` command
> >>>> would
> >>>>>>>>> not
> >>>>>>>>>>> print
> >>>>>>>>>>>>>> the added field to `--offsets verbose` (leader-epoch) or
> would
> >>>>>> it?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> JK2: Looking at ConsumerGroupCommand.java, the existing
> >>>>>>>>> "ASSIGNMENT"
> >>>>>>>>>>>>>> column under `--members --verbose` does group the topic
> >>>>>> partitions
> >>>>>>>>> by
> >>>>>>>>>>> topic
> >>>>>>>>>>>>>> but does not prefix the grouping with the topic name like
> you're
> >>>>>>>>>>> proposal:
> >>>>>>>>>>>>>> "my_topic:0,1;new_topic:0,1". Should we do apply the same
> format
> >>>>>>>>> for
> >>>>>>>>>>> the
> >>>>>>>>>>>>>> classic group as well?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 2024/11/20 21:32:41 "Lianet M." wrote:
> >>>>>>>>>>>>>>>> Hello PoAn, just of couple more minor comments:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> LM4. Regarding the new “protocol” field added to
> >>>>>>>>> MemberDescription.
> >>>>>>>>>>>>>> Should
> >>>>>>>>>>>>>>>> we consider reusing the existing GroupProtocol enum
> instead of
> >>>>>>>>>>> String?
> >>>>>>>>>>>>>>>> (It’s the one we use from the consumer side to refer to
> the
> >>>>>>>>>> protocol
> >>>>>>>>>>> in
> >>>>>>>>>>>>>>>> use, just missing Share I notice).
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> LM5. Regarding the change to the output for
> >>>>>>>>> kafka-consumer-groups,
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>> command shown does not include the —state option, but the
> >>>>>> output
> >>>>>>>>>>> shows
> >>>>>>>>>>>>>>>> state info (state, #members, epochs). I would guess that
> we
> >>>>>> want
> >>>>>>>>> to
> >>>>>>>>>>>>>> modify
> >>>>>>>>>>>>>>>> the output only when we describe a group with the —state
> >>>>>> —verbose
> >>>>>>>>>>>>>> option,
> >>>>>>>>>>>>>>>> is my understanding right? If my understanding is right
> we’re
> >>>>>>>>> just
> >>>>>>>>>>>>>> missing
> >>>>>>>>>>>>>>>> adding the —state in the example, and the KIP does not
> >>>>>> introduce
> >>>>>>>>>> any
> >>>>>>>>>>>>>>>> changes to the —describe —verbose option. (If not, it
> would
> >>>>>> mean
> >>>>>>>>> a
> >>>>>>>>>>>>>> bigger
> >>>>>>>>>>>>>>>> change to the output of —describe —verbose which I expect
> is
> >>>>>> not
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>> intention?)
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Lianet
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Wed, Nov 20, 2024 at 2:12 AM PoAn Yang <
> yangpoan@gmail.com
> >>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi Chia-Ping,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks for the review and suggestions.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> CI0: Thanks for the reminder. Update validVersions in
> >>>>>>>>>>>>>>>>> ConsumerGroupDescribeRequest to 0-1.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> CI1: Yes, I use `ConsumerGroupMember#useClassicProtocol`
> to
> >>>>>>>>> check
> >>>>>>>>>>>>>> whether
> >>>>>>>>>>>>>>>>> a member in consumer group uses “classic” or “consumer”
> >>>>>>>>> protocol.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> CI2: Yes, a member in share group always uses “share”
> >>>>>> protocol.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> CI3: Add a table to show meaning of “classic”,
> “consumer”,
> >>>> and
> >>>>>>>>>>> “share”
> >>>>>>>>>>>>>>>>> protocol.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> BTW, the vote thread is in
> >>>>>>>>>>>>>>>>>
> >>>>>>>>> https://lists.apache.org/thread/rb25tf75tzf4c7jqqldlo5jh9w8chsq6
> .
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Nov 20, 2024, at 11:46 AM, Chia-Ping Tsai <
> >>>>>>>>>> chia7712@apache.org>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> hi PoAn
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> CI0: We have to bump the version of
> >>>>>>>>> ConsumerGroupDescribeRequest
> >>>>>>>>>>> as
> >>>>>>>>>>>>>>>>> well, so server can distinguish the new and old behavior.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> CI1: the type of new filed is string, so I guess you
> plan to
> >>>>>>>>> use
> >>>>>>>>>>>>>>>>> `ConsumerGroupMember#useClassicProtocol` [0] flag to
> return
> >>>>>>>>> either
> >>>>>>>>>>>>>>>>> "classic" or "consumer", right?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> CI2: `MemberDescription` is used by
> `ShareGroupDescription`
> >>>>>>>>> too,
> >>>>>>>>>> so
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> new filed protocol in shared group is always "shared",
> right?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> CI3: Could you consider adding a table to show the
> value of
> >>>>>> the
> >>>>>>>>>>>>>> protocol
> >>>>>>>>>>>>>>>>> field in each case? Andrew has a beautiful table in
> KIP-1043
> >>>>>>>>> that
> >>>>>>>>>>>>>> lists all
> >>>>>>>>>>>>>>>>> possible protocol names.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> [0]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> https://github.com/apache/kafka/blob/441a6d0b790f4a17b454caeea7588a6b90fbd9db/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/consumer/ConsumerGroupMember.java#L454
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On 2024/11/19 15:45:16 PoAn Yang wrote:
> >>>>>>>>>>>>>>>>>>> Hi David,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks for the reminder.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> DJ4: Add related content to the KIP.
> >>>>>>>>>>>>>>>>>>> Bump validVersions in ConsumerGroupDescribeResponse to
> >>>>>> “0-1”.
> >>>>>>>>>>>>>>>>>>> Add a new field “Protocol” to
> >>>> ConsumerGroupDescribeResponse.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Nov 19, 2024, at 3:57 PM, David Jacot
> >>>>>>>>>>>>>> <dj...@confluent.io.INVALID>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> DJ4: Could you please also include the required
> changes to
> >>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> ConsumerGroupDescribed API in the public interface
> >>>> section?
> >>>>>>>>> We
> >>>>>>>>>>>>>>>>> basically
> >>>>>>>>>>>>>>>>>>>> need to bump the version, add the new field, etc.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>> David
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang <
> >>>>>>>>> yangpoan@gmail.com>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hi David,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> DJ4: Add a new field protocol to MemberDescription,
> >>>>>>>>>>>>>>>>>>>>> so the command line tool can show protocol
> information
> >>>>>> when
> >>>>>>>>>>> users
> >>>>>>>>>>>>>>>>> describe
> >>>>>>>>>>>>>>>>>>>>> members.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> If there is no further suggestion, I will start a
> vote
> >>>>>>>>> thread
> >>>>>>>>>>>>>> today.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Nov 15, 2024, at 11:50 PM, David Jacot
> >>>>>>>>>>>>>>>>> <dj...@confluent.io.INVALID>
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> DJ2: Using "-" sounds good to me.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> DJ3: That seems reasonable to me.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> DJ4: Why not add it right now? I don't want to
> change
> >>>> the
> >>>>>>>>>>> output
> >>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> tool too many times.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>> David
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang <
> >>>>>>>>>> yangpoan@gmail.com>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Hi David / Andrew,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks for review. Thanks Andrew for picking up
> >>>>>>>>>>>>>>>>> kafka-share-groups.sh
> >>>>>>>>>>>>>>>>>>>>>>> implementation.
> >>>>>>>>>>>>>>>>>>>>>>> I will handle kafka-consumer-groups.sh.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> DJ3: After discussing with @Chia-Ping Tsai, we
> think
> >>>>>> that
> >>>>>>>>>>> using
> >>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>> format
> >>>>>>>>>>>>>>>>>>>>>>> is more clear.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> The new format will be like
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Using colon(:) to concat topic name and partition
> IDs.
> >>>>>>>>>>>>>>>>>>>>>>> Using comma(,) to concat partition IDs within same
> >>>> topic
> >>>>>>>>>> name.
> >>>>>>>>>>>>>>>>>>>>>>> Using semicolon(;) to concat topic strings.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> AS3: Fix it with kafka-share-groups.sh. Thanks.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> If there is no further suggestion, I will start a
> vote
> >>>>>>>>>> thread
> >>>>>>>>>>>>>> next
> >>>>>>>>>>>>>>>>>>>>> Monday.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield <
> >>>>>>>>>>>>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>>>>>>>>>>>>>>> DJ2: I was just going to comment that "-" would
> be a
> >>>>>> more
> >>>>>>>>>>>>>>>>> appropriate
> >>>>>>>>>>>>>>>>>>>>>>> missing value, but
> >>>>>>>>>>>>>>>>>>>>>>>> you got there first.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> AS3: The examples for kafka-share-groups.sh
> include
> >>>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups.sh in the
> >>>>>>>>>>>>>>>>>>>>>>>> command line.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> If this is accepted in time, I'm happy to pick up
> the
> >>>>>>>>>>>>>>>>> implementation of
> >>>>>>>>>>>>>>>>>>>>>>> the share groups
> >>>>>>>>>>>>>>>>>>>>>>>> part of this if it helps.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>> Andrew
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>>>>>>>>>>>> From: Frank Yang <ya...@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>>>> Sent: 14 November 2024 10:48
> >>>>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
> >>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
> >>>>>>>>>> kafka-consumer-groups
> >>>>>>>>>>>>>>>>> command
> >>>>>>>>>>>>>>>>>>>>>>> line tool to support new consumer group
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Hi David,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the review and suggestion! I would
> like to
> >>>>>> get
> >>>>>>>>>>> this
> >>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>> AK
> >>>>>>>>>>>>>>>>>>>>> 4.0
> >>>>>>>>>>>>>>>>>>>>>>> as well. I will do my best.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> DJ1: Update KIP to put GROUP-EPOCH and
> >>>>>>>>>>> TARGET-ASSIGNMENT-EPOCH
> >>>>>>>>>>>>>>>>> before
> >>>>>>>>>>>>>>>>>>>>>>> #MEMBERS.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> DJ2: I prefer to follow current missing column
> value
> >>>>>> “-“.
> >>>>>>>>>>>>>>>>> (reference <
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92
> >>>>>>>>>>>>>>>>>>>>>>>> )
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> DJ3: Update KIP to use CURRENT-EPOCH
> >>>> CURRENT-ASSIGNMENT
> >>>>>>>>>>>>>>>>> TARGET-EPOCH
> >>>>>>>>>>>>>>>>>>>>>>> TARGET-ASSIGNMENT.
> >>>>>>>>>>>>>>>>>>>>>>>> Remove GROUP-EPOCH.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> For assignment value, it follows current output
> >>>>>>>>> (reference
> >>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418
> >>>>>>>>>>>>>>>>>>>>>> ).
> >>>>>>>>>>>>>>>>>>>>>>> I think the form of `topicid-partitionid` is more
> >>>> clear.
> >>>>>>>>>>>>>>>>>>>>>>>> If we would like to use this form, I will update
> both
> >>>>>>>>>> output
> >>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups.sh and kafka-share-groups.sh.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> DJ4: It looks like DescribeGroupsResponseData
> only has
> >>>>>>>>>>> protocol
> >>>>>>>>>>>>>>>>> type at
> >>>>>>>>>>>>>>>>>>>>>>> group level.
> >>>>>>>>>>>>>>>>>>>>>>>> Both DescribeGroupsResponseData and
> >>>>>>>>>>>>>>>>> ConsumerGroupDescribeResponseData
> >>>>>>>>>>>>>>>>>>>>>>> don’t have protocol at member level.
> >>>>>>>>>>>>>>>>>>>>>>>> Could we use a followup to add it?
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> DJ5: Update KIP to put LEADER-EPOCH before
> >>>>>>>>> CURRENT-OFFSET.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot
> >>>>>>>>>>>>>>>>> <djacot@confluent.io.INVALID
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP! I have a few minor
> >>>>>>>>>> comments/suggestions:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> DJ1: In the output of `--describe --verbose`, I
> would
> >>>>>>>>>>> suggest
> >>>>>>>>>>>>>>>>> putting
> >>>>>>>>>>>>>>>>>>>>>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH`
> before
> >>>>>>>>>>> `#MEMBERS`.
> >>>>>>>>>>>>>>>>>>>>>>>>> DJ2: Continuing on the above, I assume that we
> will
> >>>>>>>>> print
> >>>>>>>>>>> out
> >>>>>>>>>>>>>> N/A
> >>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> fields not supported by classic groups. Is this
> >>>>>> correct?
> >>>>>>>>>>>>>>>>>>>>>>>>> DJ3: In the output of `--describe --members
> >>>>>> --verbose`,
> >>>>>>>>> I
> >>>>>>>>>>>>>> wonder
> >>>>>>>>>>>>>>>>> if we
> >>>>>>>>>>>>>>>>>>>>>>>>> should use the following order and terms:
> >>>>>> CURRENT-EPOCH
> >>>>>>>>>>>>>>>>>>>>>>> CURRENT-ASSIGNMENT
> >>>>>>>>>>>>>>>>>>>>>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove
> the
> >>>>>>>>>>> GROUP-EPOCH
> >>>>>>>>>>>>>>>>>>>>> because
> >>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>> is already in the group description. The value
> `(0)`
> >>>>>>>>> used
> >>>>>>>>>>> for
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> assignment is incorrect. Here I suppose that we
> will
> >>>>>>>>> print
> >>>>>>>>>>> out
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> list
> >>>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>> partitions in the form of `topicid-partitionid`.
> >>>>>>>>>>>>>>>>>>>>>>>>> DJ4: Continuing on the above, I wonder if we
> should
> >>>>>> also
> >>>>>>>>>> add
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> protocol
> >>>>>>>>>>>>>>>>>>>>>>>>> used `classic` or `consumer`. For context, it is
> >>>>>>>>> possible
> >>>>>>>>>> to
> >>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>>>>>> `classic`
> >>>>>>>>>>>>>>>>>>>>>>>>> members and `consumer` members in a `consumer`
> group
> >>>>>>>>>> during
> >>>>>>>>>>> an
> >>>>>>>>>>>>>>>>> online
> >>>>>>>>>>>>>>>>>>>>>>>>> upgrade from the classic protocol to the consumer
> >>>>>>>>>> protocol.
> >>>>>>>>>>>>>> Having
> >>>>>>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>>>>> information may be handy for administrators.
> What do
> >>>>>> you
> >>>>>>>>>>> think?
> >>>>>>>>>>>>>>>>>>>>>>>>> DJ5: In the output of `--describe --offsets
> >>>>>> --verbose`,
> >>>>>>>>> I
> >>>>>>>>>>> would
> >>>>>>>>>>>>>>>>>>>>> suggest
> >>>>>>>>>>>>>>>>>>>>>>>>> putting `LEADER-EPOCH` closer to
> `CURRENT-OFFSET`.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> It would be great if we could get this one in AK
> 4.0
> >>>>>> as
> >>>>>>>>> it
> >>>>>>>>>>>>>>>>> changes the
> >>>>>>>>>>>>>>>>>>>>>>>>> output of the command.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>> DJ
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <
> >>>>>>>>>>> yangpoan@gmail.com>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi Sean / Andrew / Lianet,
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for all your review and suggestions.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> AS1, LM1, LM4: Change to add KIP-848 information
> >>>> when
> >>>>>>>>>> users
> >>>>>>>>>>>>>> give
> >>>>>>>>>>>>>>>>>>>>>>> —verbose
> >>>>>>>>>>>>>>>>>>>>>>>>>> option.
> >>>>>>>>>>>>>>>>>>>>>>>>>> —describe —verbose: shows group epoch and target
> >>>>>>>>>> assignment
> >>>>>>>>>>>>>>>>> epoch.
> >>>>>>>>>>>>>>>>>>>>>>>>>> —describe —members —verbose: shows above
> >>>> information,
> >>>>>>>>>>> member
> >>>>>>>>>>>>>>>>> epoch,
> >>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>> target assignment.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> AS2: Change to use MEMBER-EPOCH to align with
> >>>> KIP-848
> >>>>>>>>>>>>>> definition.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> LM2: For classic group, it doesn’t have epoch,
> so I
> >>>>>> use
> >>>>>>>>>>>>>> Optional
> >>>>>>>>>>>>>>>>>>>>>>> fields in
> >>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupDescription.
> >>>>>>>>>>>>>>>>>>>>>>>>>> For share group, it relies on KIP-848. It must
> have
> >>>>>>>>>> epoch,
> >>>>>>>>>>> so
> >>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>> use
> >>>>>>>>>>>>>>>>>>>>> int
> >>>>>>>>>>>>>>>>>>>>>>>>>> fields in ShareGroupDescription.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> LM3: Remove —state change. We can get group
> level
> >>>>>>>>>>> information
> >>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>>>>>> —describe
> >>>>>>>>>>>>>>>>>>>>>>>>>> —verbose.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> SQ1: Add LEADER-EPOCH when users give —describe
> >>>>>>>>> —offsets
> >>>>>>>>>>>>>>>>> —verbose.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks.
> >>>>>>>>>>>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <
> >>>>>>>>>> lianetmr@gmail.com
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Frank, thanks for the KIP! A few
> comments:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> LM1. I strongly agree with Andrew's suggestion
> of
> >>>>>>>>> moving
> >>>>>>>>>>> this
> >>>>>>>>>>>>>>>>> into a
> >>>>>>>>>>>>>>>>>>>>>>>>>>> --verbose option. I expect someone would only
> >>>>>> attempt
> >>>>>>>>> to
> >>>>>>>>>>> make
> >>>>>>>>>>>>>>>>> sense
> >>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>> epochs while debugging an issue, not while
> trying
> >>>> to
> >>>>>>>>>> view
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> group
> >>>>>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>>>>>> member's info. (Also member-epoch makes more
> sense
> >>>>>> to
> >>>>>>>>>> me,
> >>>>>>>>>>>>>> even
> >>>>>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>>>>>>> if we
> >>>>>>>>>>>>>>>>>>>>>>>>>>> end up agreeing in a --verbose). Related to
> both
> >>>>>>>>> issues,
> >>>>>>>>>>> my
> >>>>>>>>>>>>>>>>> take is
> >>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>> whoever is close to the protocol will expect
> >>>>>>>>>> member-epoch,
> >>>>>>>>>>>>>>>>> whoever
> >>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>>>> will probably won't even need to see the
> epochs at
> >>>>>>>>> all.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> LM2. Why are the epochs defined as Optional
> (don't
> >>>>>> we
> >>>>>>>>>>> expect
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> always
> >>>>>>>>>>>>>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>>>>>>>>>> them? with 0 initially, defined on the broker
> side
> >>>>>> for
> >>>>>>>>>> the
> >>>>>>>>>>>>>> group
> >>>>>>>>>>>>>>>>>>>>> ones,
> >>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>> client side for the member)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> LM3. Why is the KIP including the –state
> option in
> >>>>>> the
> >>>>>>>>>>>>>> proposed
> >>>>>>>>>>>>>>>>>>>>>>> changes?
> >>>>>>>>>>>>>>>>>>>>>>>>>> (
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
> >>>>>>>>>>>>>>>>>>>>>>>>>>> ).
> >>>>>>>>>>>>>>>>>>>>>>>>>>> I get that the output would change in that
> example,
> >>>>>>>>> but
> >>>>>>>>>>> it’s
> >>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>> because
> >>>>>>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>> any change to the –state option itself. It's
> >>>> because
> >>>>>>>>> of
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> change
> >>>>>>>>>>>>>>>>>>>>>>>>>> proposed
> >>>>>>>>>>>>>>>>>>>>>>>>>>> to the –described (required with the --state),
> and
> >>>>>> the
> >>>>>>>>>>>>>> changes
> >>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>> --describe are already explained above (seems
> >>>>>>>>> confusing,
> >>>>>>>>>>> got
> >>>>>>>>>>>>>> me
> >>>>>>>>>>>>>>>>>>>>>>> looking
> >>>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>> a change to the state filter which seemed
> >>>>>> unrelated).
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> LM4. Group epoch and target assignment epoch
> are
> >>>>>>>>>>> conceptually
> >>>>>>>>>>>>>>>>> at the
> >>>>>>>>>>>>>>>>>>>>>>>>>> group
> >>>>>>>>>>>>>>>>>>>>>>>>>>> level. vs member epoch that lands at a member
> >>>> level.
> >>>>>>>>> So
> >>>>>>>>>>>>>> wonder
> >>>>>>>>>>>>>>>>> if we
> >>>>>>>>>>>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>>>>>>>>> show them accordingly (ex. using the --verbose
> >>>>>> option)
> >>>>>>>>>>>>>>>>>>>>>>>>>>> –describe –verbose => shows group epoch and
> target
> >>>>>>>>>>> assignment
> >>>>>>>>>>>>>>>>> epoch
> >>>>>>>>>>>>>>>>>>>>>>>>>>> –describe –members –verbose => shows member
> >>>>>>>>>> epoch
> >>>>>>>>>>>>>>>>> (along
> >>>>>>>>>>>>>>>>>>>>>>>>>>> with group epoch and target assignment epoch)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Lianet
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew
> Schofield <
> >>>>>>>>>>>>>>>>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I have a few comments.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> AS1: It seems to me that these new additions
> are
> >>>>>> most
> >>>>>>>>>>>>>> useful to
> >>>>>>>>>>>>>>>>>>>>>>> people
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> trying to understand
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> the progress of rebalancing in quite some
> detail.
> >>>>>> The
> >>>>>>>>>>>>>>>>> information
> >>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> really not understandable
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> for most users who do not have deep knowledge
> of
> >>>>>>>>>>>>>> KIP-848/932.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> As a result, I suggest for
> kafka-share-groups.sh
> >>>>>> that
> >>>>>>>>>> you
> >>>>>>>>>>>>>> add a
> >>>>>>>>>>>>>>>>>>>>>>>>>> --members
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> --verbose option
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> and only include the new information in the
> output
> >>>>>>>>> for
> >>>>>>>>>>> that
> >>>>>>>>>>>>>>>>> option,
> >>>>>>>>>>>>>>>>>>>>>>>>>> rather
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> than changing the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> non-verbose --members output.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I also make a similar suggestion for
> >>>>>>>>>>>>>> kafka-consumer-groups.sh
> >>>>>>>>>>>>>>>>>>>>>>> --members
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> and only add the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> new information for the --verbose output.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> AS2: I strongly suggest that you use
> MEMBER-EPOCH
> >>>>>>>>>>> instead of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> CONSUMER-EPOCH.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it's more confusing not following the
> >>>>>>>>>>> terminology of
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> KIPs.
> >>>>>>>>>>>>>>>>>>>>>>>>>> For
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> one thing,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> we already have "member" in the admin client
> such
> >>>>>> as
> >>>>>>>>>>>>>>>>>>>>>>> MemberDescription.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's not a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> new term.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Andrew
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> From: PoAn Yang <pa...@apache.org>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Sent: 25 October 2024 13:55
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <
> dev@kafka.apache.org>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
> >>>>>>>>>>>>>> kafka-consumer-groups
> >>>>>>>>>>>>>>>>>>>>> command
> >>>>>>>>>>>>>>>>>>>>>>>>>> line
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> tool to support new consumer group
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Lucas,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the review!
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Yes, I add related change for
> >>>>>>>>> kafka-share-groups.sh
> >>>>>>>>>> to
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> KIP.
> >>>>>>>>>>>>>>>>>>>>>>> Could
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> you take a look? Thanks for the suggestion.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use
> >>>>>>>>>>> MEMBER-EPOCH
> >>>>>>>>>>>>>>>>> here,
> >>>>>>>>>>>>>>>>>>>>>>> users
> >>>>>>>>>>>>>>>>>>>>>>>>>> may
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> confuse what is different between CONSUMER and
> >>>>>>>>> MEMBER.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Frank,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks for the KIP!
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) For consistency, should we do the same for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription,
> >>>>>> etc. ?
> >>>>>>>>>>> Even
> >>>>>>>>>>>>>> if
> >>>>>>>>>>>>>>>>> we do
> >>>>>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> implement it right now if the share group
> >>>>>>>>>> implementation
> >>>>>>>>>>>>>> may
> >>>>>>>>>>>>>>>>> still
> >>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> incomplete, it may make sense to include it
> in
> >>>> the
> >>>>>>>>>> KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not
> MEMBER-EPOCH?
> >>>>>>>>> That
> >>>>>>>>>>> would
> >>>>>>>>>>>>>>>>> seem
> >>>>>>>>>>>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> consistent.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lucas
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <
> >>>>>>>>>>>>>>>>> yangpoan@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to kick off the discussion of
> >>>>>>>>> KIP-1099.
> >>>>>>>>>>> This
> >>>>>>>>>>>>>> KIP
> >>>>>>>>>>>>>>>>>>>>>>> enhances
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups tools to include state
> >>>>>> which
> >>>>>>>>> is
> >>>>>>>>>>>>>>>>> introduced
> >>>>>>>>>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP-848.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups
> command
> >>>>>> line
> >>>>>>>>>>> tool
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> support
> >>>>>>>>>>>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer group - Apache Kafka - Apache
> Software
> >>>>>>>>>>> Foundation
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cwiki.apache.org
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [image: favicon.ico]
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>>
> >
>
>
>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by PoAn Yang <ya...@gmail.com>.
Hi David,
Thanks for the feedback.
Yes, I agree that the “UPGRADED” column name is better. I updated the KIP.
Best,
PoAn
> On Nov 25, 2024, at 11:11 PM, PoAn Yang <ya...@gmail.com> wrote:
>
> Hi Andrew,
>
> Thanks for the feedback.
>
> AS4: Change TARGET-ASSIGNMENT-EPOCH to ASSIGNMENT-EPOCH for “—describe —state —verbose" in kafka-share-groups.sh.
>
> AS5: Remove TARGET-EPOCH and TARGET-ASSIGNMENT.
> Change CURRENT-EPOCH to MEMBER-EPOCH and CURRENT-ASSIGNMENT to ASSIGNMENT for “—describe —member —verbose” in kafka-share-groups.sh.
>
> Thanks,
> PoAn
>
>> On Nov 25, 2024, at 10:48 PM, Andrew Schofield <an...@outlook.com> wrote:
>>
>> Hi PoAn,
>> I've taken another pass over the KIP and have some more comments around the share
>> group support.
>>
>> AS4: Share groups do not have the concept of a target assignment. As a result,
>> the verbose state TARGET-ASSIGNMENT-EPOCH is a bit misleading from my
>> point of view. ASSIGNMENT-EPOCH is better.
>>
>> AS5: Similarly for the verbose members output, the TARGET-EPOCH and
>> TARGET-ASSIGNMENT are not really appropriate. Personally, I would prefer
>> just MEMBER-EPOCH and ASSIGNMENT, and remove the "target" fields.
>>
>> Thanks,
>> Andrew
>>
>> ________________________________________
>> From: PoAn Yang <ya...@gmail.com>
>> Sent: 24 November 2024 15:08
>> To: dev@kafka.apache.org <de...@kafka.apache.org>
>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
>>
>> Hi Chia-Ping,
>>
>> Thanks for the suggestion.
>>
>> The “isClassic” field in ConsumerGroupDescribeResponse is a boolean value, so we can’t use it as the protocol string.
>> I change the “PROTOCOL” column to “IS-CLASSIC” in kafka-consumer-groups.sh.
>> It returns “true” if a member is in a classic group or it’s a classic member in a consumer group.
>> In other cases, it returns “false”.
>>
>> I have a draft implementation in https://github.com/apache/kafka/pull/17872.
>>
>> Thanks,
>> PoAn
>>
>>> On Nov 23, 2024, at 1:05 PM, Chia-Ping Tsai <ch...@gmail.com> wrote:
>>>
>>> hi PoAn
>>>
>>> `isClassic` is good to me :)
>>>
>>> Best,
>>> Chia-Ping
>>>
>>> PoAn Yang <ya...@gmail.com> 於 2024年11月23日 週六 下午12:35寫道：
>>>
>>>> Hi Chia-Ping / David,
>>>>
>>>> Thanks for the great suggestion.
>>>>
>>>> How about using “isClassic”?
>>>> The term like “isLegacy” may be equivocal in the future.
>>>> For example, if we have a newer version of consumer / share consumer, the
>>>> meaning will be different for different cases.
>>>> If we use “isClassic”, for MemberDescription in ClassicGroupDescription,
>>>> the value can be always “true”.
>>>> In ConsumerGroupDescription, it’s “true” when it’s a classic member.
>>>> In ShareGroupDescription, it’s always “false”.
>>>>
>>>> Thanks,
>>>> PoAn
>>>>
>>>>> On Nov 23, 2024, at 1:02 AM, Chia-Ping Tsai <ch...@gmail.com> wrote:
>>>>>
>>>>> hi DJ
>>>>>
>>>>>> My goal was to have a way to tell the user that his (new) consumer group
>>>>> may have non-upgraded members yet. I wonder if we could use a boolean to
>>>>> signal this, e.g. isLegacy. Then, in the CLI we could have a column which
>>>>> is only displayed when they are non upgraded members. What do you think?
>>>>>
>>>>> Yes, this is a good approach, but the downside is that the new isLegacy
>>>>> field in MemberDescription would be irrelevant for classic and shared
>>>>> groups. This would require thorough documentation to clarify its purpose.
>>>>>
>>>>> Additionally, we could use a boolean instead of a string for the new
>>>> field
>>>>> in the RPC to reduce the request size.
>>>>> Best,
>>>>> Chia-Ping
>>>>>
>>>>>
>>>>>
>>>>> David Jacot <dj...@confluent.io.invalid> 於 2024年11月23日 週六 上午12:41寫道：
>>>>>
>>>>>> Hi both,
>>>>>>
>>>>>> Ah, I understand what you mean now. You're saying that we have two
>>>> protocol
>>>>>> fields. One at the group level and one at the member level (the new
>>>> one).
>>>>>> The two fields are actually two different things. The former is the
>>>> name of
>>>>>> the embedded protocol used in the classic protocol whereas the latter is
>>>>>> the name of the protocol/api used. This is indeed confusing.
>>>>>>
>>>>>> My goal was to have a way to tell the user that his (new) consumer group
>>>>>> may have non-upgraded members yet. I wonder if we could use a boolean to
>>>>>> signal this, e.g. isLegacy. Then, in the CLI we could have a column
>>>> which
>>>>>> is only displayed when they are non upgraded members. What do you think?
>>>>>>
>>>>>> If we cannot find a good way to represent this, we could drop it from
>>>> this
>>>>>> KIP too.
>>>>>>
>>>>>> Best,
>>>>>> David
>>>>>>
>>>>>> On Fri, Nov 22, 2024 at 5:12 PM PoAn Yang <ya...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Chia-Ping / David,
>>>>>>>
>>>>>>> Thanks for the review and suggestions.
>>>>>>>
>>>>>>>>> I don't fully follow your comment so please excuse me if I say
>>>>>> something
>>>>>>>>> wrong.
>>>>>>>
>>>>>>> I think Chia-Ping wants to say: originally, the default protocol in
>>>>>>> classic group is “consumer”.
>>>>>>> After we have a new consumer group, the protocol of a classic member in
>>>>>>> consumer group is “classic”.
>>>>>>> The MemberDescription is used by both ClassicGroupDescription and
>>>>>>> ConsumerGroupDescription.
>>>>>>> If we add the protocol field to MemberDescription, the protocol of a
>>>>>>> classic member in classic group is “consumer”, but it’s “classic” in
>>>>>>> consumer group.
>>>>>>> To avoid misleading word, we can add a new collection of member
>>>> protocols
>>>>>>> to ConsumerGroupDescription only.
>>>>>>> The original goal is to show member protocol during consumer migration,
>>>>>> so
>>>>>>> we don’t need to add this information to ClassicGroupDescription.
>>>>>>>
>>>>>>> Please correct me if I misunderstood anything.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> PoAn
>>>>>>>
>>>>>>>> On Nov 22, 2024, at 7:00 PM, Chia-Ping Tsai <ch...@gmail.com>
>>>>>> wrote:
>>>>>>>>
>>>>>>>> hi DJ
>>>>>>>>
>>>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and
>>>>>> change
>>>>>>>> return type of ModernGroup#protocolType?
>>>>>>>>
>>>>>>>> This is not my comment ... sorry for the incorrect quote :(
>>>>>>>>
>>>>>>>>> A member in a classic group should report "classic".
>>>>>>>>
>>>>>>>> I completely agree. However, in this case,
>>>>>>> ClassicGroupDescription#protocol
>>>>>>>> returns 'consumer', while ClassicGroupDescription#members[x].protocol
>>>>>>>> returns 'classic'. This inconsistency feels a bit strange to me.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Chia-Ping
>>>>>>>>
>>>>>>>>
>>>>>>>> David Jacot <dj...@confluent.io.invalid> 於 2024年11月22日 週五 下午6:51寫道：
>>>>>>>>
>>>>>>>>> Hi Chia,
>>>>>>>>>
>>>>>>>>>>> LM4: I agree it’s better to use a type like GroupProtocol enum, but
>>>>>> I
>>>>>>>>> would like to align the format with kafka-groups.sh.
>>>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and
>>>>>> change
>>>>>>>>> return type of ModernGroup#protocolType?
>>>>>>>>>
>>>>>>>>> My understanding is that `GroupProtocol` is used to drive the config
>>>>>> of
>>>>>>> the
>>>>>>>>> `Consumer`. Hence adding `Share` seems wrong to me. Or are you
>>>>>>> referring to
>>>>>>>>> another GroupProtocol enum? Perhaps GroupType enum?
>>>>>>>>>
>>>>>>>>>> In fact, I'm wondering if we should even add 'protocol' to
>>>>>>>>> MemberDescription,
>>>>>>>>> since the field has different values between classic and consumer
>>>>>>> groups—it
>>>>>>>>> shows 'consumer' in classic groups, whereas it shows 'classic' in
>>>>>>> consumer
>>>>>>>>> groups. Yes, it seems like a typo, but it's true. This inconsistency
>>>>>>> feels
>>>>>>>>> confusing to me. For example, users call Admin#describeConsumerGroups
>>>>>>> and
>>>>>>>>> see that a member has the 'consumer' protocol when it's in a classic
>>>>>>> group.
>>>>>>>>> Then, if the group is upgraded to a consumer group, calling
>>>>>>>>> Admin#describeConsumerGroups again shows the protocol as 'classic'
>>>> for
>>>>>>> the
>>>>>>>>> same member.
>>>>>>>>>
>>>>>>>>> I don't fully follow your comment so please excuse me if I say
>>>>>> something
>>>>>>>>> wrong.
>>>>>>>>>
>>>>>>>>> I think that it should work as follows:
>>>>>>>>> * A member in a classic group should report "classic".
>>>>>>>>> * A member using the consumer protocol in a consumer group should
>>>>>> report
>>>>>>>>> "consumer".
>>>>>>>>> * A member using the classic protocol in a consumer group should
>>>>>> report
>>>>>>>>> "classic".
>>>>>>>>> * A member in a share group should report "share".
>>>>>>>>>
>>>>>>>>>> If all we want to do is distinguish classic members in the
>>>>>>>>> ConsumerGroupDescription, then we shouldn't modify the common
>>>>>>>>> MemberDescription structure. Instead, we could add a collection to
>>>>>>>>> ConsumerGroupDescription to trace members using the classic protocol,
>>>>>>> which
>>>>>>>>> would allow us to enrich the output of command-line tools.
>>>>>>>>>
>>>>>>>>> Personally, I believe that using different collections per protocol
>>>>>>> types
>>>>>>>>> is not necessary. A field in the member is more than enough.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> DJ
>>>>>>>>>
>>>>>>>>> On Fri, Nov 22, 2024 at 12:01 PM Chia-Ping Tsai <ch...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> hi PoAn
>>>>>>>>>>
>>>>>>>>>>> LM4: I agree it’s better to use a type like GroupProtocol enum, but
>>>>>> I
>>>>>>>>>> would like to align the format with kafka-groups.sh.
>>>>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and
>>>>>>> change
>>>>>>>>>> return type of ModernGroup#protocolType?
>>>>>>>>>>
>>>>>>>>>> MemberDescription is also used by ClassicGroupDescription, and in
>>>>>>> classic
>>>>>>>>>> groups, the protocol can be an arbitrary string, so we can't use an
>>>>>>> enum.
>>>>>>>>>>
>>>>>>>>>> In fact, I'm wondering if we should even add 'protocol' to
>>>>>>>>>> MemberDescription,
>>>>>>>>>> since the field has different values between classic and consumer
>>>>>>>>> groups—it
>>>>>>>>>> shows 'consumer' in classic groups, whereas it shows 'classic' in
>>>>>>>>> consumer
>>>>>>>>>> groups. Yes, it seems like a typo, but it's true. This inconsistency
>>>>>>>>> feels
>>>>>>>>>> confusing to me. For example, users call
>>>> Admin#describeConsumerGroups
>>>>>>> and
>>>>>>>>>> see that a member has the 'consumer' protocol when it's in a classic
>>>>>>>>> group.
>>>>>>>>>> Then, if the group is upgraded to a consumer group, calling
>>>>>>>>>> Admin#describeConsumerGroups again shows the protocol as 'classic'
>>>>>> for
>>>>>>>>> the
>>>>>>>>>> same member.
>>>>>>>>>>
>>>>>>>>>> If all we want to do is distinguish classic members in the
>>>>>>>>>> ConsumerGroupDescription, then we shouldn't modify the common
>>>>>>>>>> MemberDescription structure. Instead, we could add a collection to
>>>>>>>>>> ConsumerGroupDescription to trace members using the classic
>>>> protocol,
>>>>>>>>> which
>>>>>>>>>> would allow us to enrich the output of command-line tools.
>>>>>>>>>> Best,
>>>>>>>>>> Chia-Ping
>>>>>>>>>>
>>>>>>>>>> PoAn Yang <ya...@gmail.com> 於 2024年11月22日 週五 下午3:46寫道：
>>>>>>>>>>
>>>>>>>>>>> Hi Lianet / Jeff,
>>>>>>>>>>>
>>>>>>>>>>> Thanks for the review.
>>>>>>>>>>>
>>>>>>>>>>> LM5: The kafka-share-groups.sh also uses “—describe” to show
>>>> offsets
>>>>>>>>>>> information.
>>>>>>>>>>> Change to use “—describe —state —verbose” to show group level
>>>>>>>>> information
>>>>>>>>>>> in kafka-share-groups.sh.
>>>>>>>>>>>
>>>>>>>>>>> LM6: Update the example.
>>>>>>>>>>>
>>>>>>>>>>> LM7: Add a description to mention “—verbose” is a new option in
>>>>>>>>>>> ShareGroupCommandOptions.
>>>>>>>>>>>
>>>>>>>>>>> JK2: Yes, add the statement about format change for ASSIGNMENT
>>>>>> value.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> PoAn
>>>>>>>>>>>
>>>>>>>>>>>> On Nov 22, 2024, at 7:35 AM, Jeff Kim <je...@apache.org>
>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>
>>>>>>>>>>>> JK2: Can we state in the KIP that we will reformat the existing
>>>>>>>>>>>> classic group's `--members --verbose` as well, specifically the
>>>>>>>>>>>> "ASSIGNMENT" column to include the topic name?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>
>>>>>>>>>>>> On 2024/11/21 14:33:37 "Lianet M." wrote:
>>>>>>>>>>>>> Thanks for the updates PoAn!
>>>>>>>>>>>>>
>>>>>>>>>>>>> LM6. Just a nit, under the --describe --state --verbose the
>>>> header
>>>>>>>>> is
>>>>>>>>>>> fine
>>>>>>>>>>>>> but the example is still missing the --state argument.
>>>>>>>>>>>>>
>>>>>>>>>>>>> LM7. Under the Proposed changes for kafka-share-groups.sh, should
>>>>>> we
>>>>>>>>>>>>> clearly mention that the KIP adds a new --verbose option? I think
>>>>>>>>> it's
>>>>>>>>>>>>> confusing because it's presented as if the option already exists
>>>>>> and
>>>>>>>>>>> we're
>>>>>>>>>>>>> only changing the output (which is the case for
>>>>>>>>> kafka-consumer-groups
>>>>>>>>>>> but
>>>>>>>>>>>>> not for kafka-share-groups)
>>>>>>>>>>>>>
>>>>>>>>>>>>> That's all on my side. Thanks!
>>>>>>>>>>>>>
>>>>>>>>>>>>> Lianet
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Nov 20, 2024 at 11:12 PM PoAn Yang <ya...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Lianet / Jeff,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for review and suggestions.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> LM4: At group level, the protocol field is from
>>>>>>>>>>> ModernGroup#protocolType.
>>>>>>>>>>>>>> It’s a string.
>>>>>>>>>>>>>> The value is defined in different places like
>>>>>>>>>> ShareGroup#PROTOCOL_TYPE
>>>>>>>>>>> and
>>>>>>>>>>>>>> ConsumerProtocol#PROTOCOL_TYPE.
>>>>>>>>>>>>>> In KIP-1043 <
>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=305171038#KIP1043:Administrationofgroups-kafka-groups.sh
>>>>>>>>>>>> ,
>>>>>>>>>>>>>> the kafka-groups.sh also shows protocol like “classic” /
>>>>>>>>> “consumer” /
>>>>>>>>>>>>>> “share”.
>>>>>>>>>>>>>> I agree it’s better to use a type like GroupProtocol enum, but I
>>>>>>>>>> would
>>>>>>>>>>>>>> like to align the format with kafka-groups.sh.
>>>>>>>>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol
>>>> and
>>>>>>>>>>> change
>>>>>>>>>>>>>> return type of ModernGroup#protocolType?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> LM5 / JK1: Sorry, I misunderstood LM3, so I deleted —state in
>>>> the
>>>>>>>>>>> title.
>>>>>>>>>>>>>> You’re correct. I would like to use `—describe —state —verbose`
>>>>>> to
>>>>>>>>>> show
>>>>>>>>>>>>>> group level information.
>>>>>>>>>>>>>> For `—describe —verbose`, it will show output as same as
>>>>>> `—describe
>>>>>>>>>>>>>> —offsets —verbose`,
>>>>>>>>>>>>>> because `—describe` also shows output as same as `—describe
>>>>>>>>>> —offsets`.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> JK2: Yes, I only use consumer group in the sample output, but
>>>> the
>>>>>>>>>>> classic
>>>>>>>>>>>>>> group will use the same format as well.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Nov 21, 2024, at 8:28 AM, Jeff Kim <je...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for the KIP. I have some questions:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> JK1: (Realized this is similar to LM5) On `--describe
>>>> --verbose`
>>>>>>>>>>>>>> proposed changes - doesn't `--describe $group` default to
>>>>>> printing
>>>>>>>>>> the
>>>>>>>>>>>>>> offsets? Perhaps you're referring to the `--state` argument?
>>>>>> Also,
>>>>>>>>>>> would
>>>>>>>>>>>>>> that mean the default `--describe $group --verbose` command
>>>> would
>>>>>>>>> not
>>>>>>>>>>> print
>>>>>>>>>>>>>> the added field to `--offsets verbose` (leader-epoch) or would
>>>>>> it?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> JK2: Looking at ConsumerGroupCommand.java, the existing
>>>>>>>>> "ASSIGNMENT"
>>>>>>>>>>>>>> column under `--members --verbose` does group the topic
>>>>>> partitions
>>>>>>>>> by
>>>>>>>>>>> topic
>>>>>>>>>>>>>> but does not prefix the grouping with the topic name like you're
>>>>>>>>>>> proposal:
>>>>>>>>>>>>>> "my_topic:0,1;new_topic:0,1". Should we do apply the same format
>>>>>>>>> for
>>>>>>>>>>> the
>>>>>>>>>>>>>> classic group as well?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 2024/11/20 21:32:41 "Lianet M." wrote:
>>>>>>>>>>>>>>>> Hello PoAn, just of couple more minor comments:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> LM4. Regarding the new “protocol” field added to
>>>>>>>>> MemberDescription.
>>>>>>>>>>>>>> Should
>>>>>>>>>>>>>>>> we consider reusing the existing GroupProtocol enum instead of
>>>>>>>>>>> String?
>>>>>>>>>>>>>>>> (It’s the one we use from the consumer side to refer to the
>>>>>>>>>> protocol
>>>>>>>>>>> in
>>>>>>>>>>>>>>>> use, just missing Share I notice).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> LM5. Regarding the change to the output for
>>>>>>>>> kafka-consumer-groups,
>>>>>>>>>>> the
>>>>>>>>>>>>>>>> command shown does not include the —state option, but the
>>>>>> output
>>>>>>>>>>> shows
>>>>>>>>>>>>>>>> state info (state, #members, epochs). I would guess that we
>>>>>> want
>>>>>>>>> to
>>>>>>>>>>>>>> modify
>>>>>>>>>>>>>>>> the output only when we describe a group with the —state
>>>>>> —verbose
>>>>>>>>>>>>>> option,
>>>>>>>>>>>>>>>> is my understanding right? If my understanding is right we’re
>>>>>>>>> just
>>>>>>>>>>>>>> missing
>>>>>>>>>>>>>>>> adding the —state in the example, and the KIP does not
>>>>>> introduce
>>>>>>>>>> any
>>>>>>>>>>>>>>>> changes to the —describe —verbose option. (If not, it would
>>>>>> mean
>>>>>>>>> a
>>>>>>>>>>>>>> bigger
>>>>>>>>>>>>>>>> change to the output of —describe —verbose which I expect is
>>>>>> not
>>>>>>>>>> the
>>>>>>>>>>>>>>>> intention?)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Lianet
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Nov 20, 2024 at 2:12 AM PoAn Yang <yangpoan@gmail.com
>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Chia-Ping,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for the review and suggestions.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> CI0: Thanks for the reminder. Update validVersions in
>>>>>>>>>>>>>>>>> ConsumerGroupDescribeRequest to 0-1.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> CI1: Yes, I use `ConsumerGroupMember#useClassicProtocol` to
>>>>>>>>> check
>>>>>>>>>>>>>> whether
>>>>>>>>>>>>>>>>> a member in consumer group uses “classic” or “consumer”
>>>>>>>>> protocol.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> CI2: Yes, a member in share group always uses “share”
>>>>>> protocol.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> CI3: Add a table to show meaning of “classic”, “consumer”,
>>>> and
>>>>>>>>>>> “share”
>>>>>>>>>>>>>>>>> protocol.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> BTW, the vote thread is in
>>>>>>>>>>>>>>>>>
>>>>>>>>> https://lists.apache.org/thread/rb25tf75tzf4c7jqqldlo5jh9w8chsq6.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Nov 20, 2024, at 11:46 AM, Chia-Ping Tsai <
>>>>>>>>>> chia7712@apache.org>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> hi PoAn
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> CI0: We have to bump the version of
>>>>>>>>> ConsumerGroupDescribeRequest
>>>>>>>>>>> as
>>>>>>>>>>>>>>>>> well, so server can distinguish the new and old behavior.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> CI1: the type of new filed is string, so I guess you plan to
>>>>>>>>> use
>>>>>>>>>>>>>>>>> `ConsumerGroupMember#useClassicProtocol` [0] flag to return
>>>>>>>>> either
>>>>>>>>>>>>>>>>> "classic" or "consumer", right?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> CI2: `MemberDescription` is used by `ShareGroupDescription`
>>>>>>>>> too,
>>>>>>>>>> so
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> new filed protocol in shared group is always "shared", right?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> CI3: Could you consider adding a table to show the value of
>>>>>> the
>>>>>>>>>>>>>> protocol
>>>>>>>>>>>>>>>>> field in each case? Andrew has a beautiful table in KIP-1043
>>>>>>>>> that
>>>>>>>>>>>>>> lists all
>>>>>>>>>>>>>>>>> possible protocol names.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [0]
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>> https://github.com/apache/kafka/blob/441a6d0b790f4a17b454caeea7588a6b90fbd9db/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/consumer/ConsumerGroupMember.java#L454
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 2024/11/19 15:45:16 PoAn Yang wrote:
>>>>>>>>>>>>>>>>>>> Hi David,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks for the reminder.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> DJ4: Add related content to the KIP.
>>>>>>>>>>>>>>>>>>> Bump validVersions in ConsumerGroupDescribeResponse to
>>>>>> “0-1”.
>>>>>>>>>>>>>>>>>>> Add a new field “Protocol” to
>>>> ConsumerGroupDescribeResponse.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Nov 19, 2024, at 3:57 PM, David Jacot
>>>>>>>>>>>>>> <dj...@confluent.io.INVALID>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> DJ4: Could you please also include the required changes to
>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> ConsumerGroupDescribed API in the public interface
>>>> section?
>>>>>>>>> We
>>>>>>>>>>>>>>>>> basically
>>>>>>>>>>>>>>>>>>>> need to bump the version, add the new field, etc.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> David
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang <
>>>>>>>>> yangpoan@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi David,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> DJ4: Add a new field protocol to MemberDescription,
>>>>>>>>>>>>>>>>>>>>> so the command line tool can show protocol information
>>>>>> when
>>>>>>>>>>> users
>>>>>>>>>>>>>>>>> describe
>>>>>>>>>>>>>>>>>>>>> members.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> If there is no further suggestion, I will start a vote
>>>>>>>>> thread
>>>>>>>>>>>>>> today.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Nov 15, 2024, at 11:50 PM, David Jacot
>>>>>>>>>>>>>>>>> <dj...@confluent.io.INVALID>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> DJ2: Using "-" sounds good to me.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> DJ3: That seems reasonable to me.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> DJ4: Why not add it right now? I don't want to change
>>>> the
>>>>>>>>>>> output
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> tool too many times.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>> David
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang <
>>>>>>>>>> yangpoan@gmail.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi David / Andrew,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks for review. Thanks Andrew for picking up
>>>>>>>>>>>>>>>>> kafka-share-groups.sh
>>>>>>>>>>>>>>>>>>>>>>> implementation.
>>>>>>>>>>>>>>>>>>>>>>> I will handle kafka-consumer-groups.sh.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> DJ3: After discussing with @Chia-Ping Tsai, we think
>>>>>> that
>>>>>>>>>>> using
>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>> format
>>>>>>>>>>>>>>>>>>>>>>> is more clear.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> The new format will be like
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>> <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Using colon(:) to concat topic name and partition IDs.
>>>>>>>>>>>>>>>>>>>>>>> Using comma(,) to concat partition IDs within same
>>>> topic
>>>>>>>>>> name.
>>>>>>>>>>>>>>>>>>>>>>> Using semicolon(;) to concat topic strings.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> AS3: Fix it with kafka-share-groups.sh. Thanks.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> If there is no further suggestion, I will start a vote
>>>>>>>>>> thread
>>>>>>>>>>>>>> next
>>>>>>>>>>>>>>>>>>>>> Monday.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield <
>>>>>>>>>>>>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>>>>>>>>>>>> DJ2: I was just going to comment that "-" would be a
>>>>>> more
>>>>>>>>>>>>>>>>> appropriate
>>>>>>>>>>>>>>>>>>>>>>> missing value, but
>>>>>>>>>>>>>>>>>>>>>>>> you got there first.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> AS3: The examples for kafka-share-groups.sh include
>>>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups.sh in the
>>>>>>>>>>>>>>>>>>>>>>>> command line.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> If this is accepted in time, I'm happy to pick up the
>>>>>>>>>>>>>>>>> implementation of
>>>>>>>>>>>>>>>>>>>>>>> the share groups
>>>>>>>>>>>>>>>>>>>>>>>> part of this if it helps.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> Andrew
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>>>>>>>>>> From: Frank Yang <ya...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>> Sent: 14 November 2024 10:48
>>>>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
>>>>>>>>>> kafka-consumer-groups
>>>>>>>>>>>>>>>>> command
>>>>>>>>>>>>>>>>>>>>>>> line tool to support new consumer group
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi David,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the review and suggestion! I would like to
>>>>>> get
>>>>>>>>>>> this
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> AK
>>>>>>>>>>>>>>>>>>>>> 4.0
>>>>>>>>>>>>>>>>>>>>>>> as well. I will do my best.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> DJ1: Update KIP to put GROUP-EPOCH and
>>>>>>>>>>> TARGET-ASSIGNMENT-EPOCH
>>>>>>>>>>>>>>>>> before
>>>>>>>>>>>>>>>>>>>>>>> #MEMBERS.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> DJ2: I prefer to follow current missing column value
>>>>>> “-“.
>>>>>>>>>>>>>>>>> (reference <
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92
>>>>>>>>>>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> DJ3: Update KIP to use CURRENT-EPOCH
>>>> CURRENT-ASSIGNMENT
>>>>>>>>>>>>>>>>> TARGET-EPOCH
>>>>>>>>>>>>>>>>>>>>>>> TARGET-ASSIGNMENT.
>>>>>>>>>>>>>>>>>>>>>>>> Remove GROUP-EPOCH.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> For assignment value, it follows current output
>>>>>>>>> (reference
>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418
>>>>>>>>>>>>>>>>>>>>>> ).
>>>>>>>>>>>>>>>>>>>>>>> I think the form of `topicid-partitionid` is more
>>>> clear.
>>>>>>>>>>>>>>>>>>>>>>>> If we would like to use this form, I will update both
>>>>>>>>>> output
>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups.sh and kafka-share-groups.sh.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> DJ4: It looks like DescribeGroupsResponseData only has
>>>>>>>>>>> protocol
>>>>>>>>>>>>>>>>> type at
>>>>>>>>>>>>>>>>>>>>>>> group level.
>>>>>>>>>>>>>>>>>>>>>>>> Both DescribeGroupsResponseData and
>>>>>>>>>>>>>>>>> ConsumerGroupDescribeResponseData
>>>>>>>>>>>>>>>>>>>>>>> don’t have protocol at member level.
>>>>>>>>>>>>>>>>>>>>>>>> Could we use a followup to add it?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> DJ5: Update KIP to put LEADER-EPOCH before
>>>>>>>>> CURRENT-OFFSET.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot
>>>>>>>>>>>>>>>>> <djacot@confluent.io.INVALID
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP! I have a few minor
>>>>>>>>>> comments/suggestions:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> DJ1: In the output of `--describe --verbose`, I would
>>>>>>>>>>> suggest
>>>>>>>>>>>>>>>>> putting
>>>>>>>>>>>>>>>>>>>>>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before
>>>>>>>>>>> `#MEMBERS`.
>>>>>>>>>>>>>>>>>>>>>>>>> DJ2: Continuing on the above, I assume that we will
>>>>>>>>> print
>>>>>>>>>>> out
>>>>>>>>>>>>>> N/A
>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> fields not supported by classic groups. Is this
>>>>>> correct?
>>>>>>>>>>>>>>>>>>>>>>>>> DJ3: In the output of `--describe --members
>>>>>> --verbose`,
>>>>>>>>> I
>>>>>>>>>>>>>> wonder
>>>>>>>>>>>>>>>>> if we
>>>>>>>>>>>>>>>>>>>>>>>>> should use the following order and terms:
>>>>>> CURRENT-EPOCH
>>>>>>>>>>>>>>>>>>>>>>> CURRENT-ASSIGNMENT
>>>>>>>>>>>>>>>>>>>>>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the
>>>>>>>>>>> GROUP-EPOCH
>>>>>>>>>>>>>>>>>>>>> because
>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>> is already in the group description. The value `(0)`
>>>>>>>>> used
>>>>>>>>>>> for
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> assignment is incorrect. Here I suppose that we will
>>>>>>>>> print
>>>>>>>>>>> out
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> list
>>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>> partitions in the form of `topicid-partitionid`.
>>>>>>>>>>>>>>>>>>>>>>>>> DJ4: Continuing on the above, I wonder if we should
>>>>>> also
>>>>>>>>>> add
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> protocol
>>>>>>>>>>>>>>>>>>>>>>>>> used `classic` or `consumer`. For context, it is
>>>>>>>>> possible
>>>>>>>>>> to
>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>>> `classic`
>>>>>>>>>>>>>>>>>>>>>>>>> members and `consumer` members in a `consumer` group
>>>>>>>>>> during
>>>>>>>>>>> an
>>>>>>>>>>>>>>>>> online
>>>>>>>>>>>>>>>>>>>>>>>>> upgrade from the classic protocol to the consumer
>>>>>>>>>> protocol.
>>>>>>>>>>>>>> Having
>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>> information may be handy for administrators. What do
>>>>>> you
>>>>>>>>>>> think?
>>>>>>>>>>>>>>>>>>>>>>>>> DJ5: In the output of `--describe --offsets
>>>>>> --verbose`,
>>>>>>>>> I
>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>>>>> suggest
>>>>>>>>>>>>>>>>>>>>>>>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> It would be great if we could get this one in AK 4.0
>>>>>> as
>>>>>>>>> it
>>>>>>>>>>>>>>>>> changes the
>>>>>>>>>>>>>>>>>>>>>>>>> output of the command.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>> DJ
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <
>>>>>>>>>>> yangpoan@gmail.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Sean / Andrew / Lianet,
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for all your review and suggestions.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> AS1, LM1, LM4: Change to add KIP-848 information
>>>> when
>>>>>>>>>> users
>>>>>>>>>>>>>> give
>>>>>>>>>>>>>>>>>>>>>>> —verbose
>>>>>>>>>>>>>>>>>>>>>>>>>> option.
>>>>>>>>>>>>>>>>>>>>>>>>>> —describe —verbose: shows group epoch and target
>>>>>>>>>> assignment
>>>>>>>>>>>>>>>>> epoch.
>>>>>>>>>>>>>>>>>>>>>>>>>> —describe —members —verbose: shows above
>>>> information,
>>>>>>>>>>> member
>>>>>>>>>>>>>>>>> epoch,
>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>> target assignment.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> AS2: Change to use MEMBER-EPOCH to align with
>>>> KIP-848
>>>>>>>>>>>>>> definition.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> LM2: For classic group, it doesn’t have epoch, so I
>>>>>> use
>>>>>>>>>>>>>> Optional
>>>>>>>>>>>>>>>>>>>>>>> fields in
>>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupDescription.
>>>>>>>>>>>>>>>>>>>>>>>>>> For share group, it relies on KIP-848. It must have
>>>>>>>>>> epoch,
>>>>>>>>>>> so
>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>>>>> int
>>>>>>>>>>>>>>>>>>>>>>>>>> fields in ShareGroupDescription.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> LM3: Remove —state change. We can get group level
>>>>>>>>>>> information
>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>> —describe
>>>>>>>>>>>>>>>>>>>>>>>>>> —verbose.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> SQ1: Add LEADER-EPOCH when users give —describe
>>>>>>>>> —offsets
>>>>>>>>>>>>>>>>> —verbose.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <
>>>>>>>>>> lianetmr@gmail.com
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Frank, thanks for the KIP! A few comments:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> LM1. I strongly agree with Andrew's suggestion of
>>>>>>>>> moving
>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> into a
>>>>>>>>>>>>>>>>>>>>>>>>>>> --verbose option. I expect someone would only
>>>>>> attempt
>>>>>>>>> to
>>>>>>>>>>> make
>>>>>>>>>>>>>>>>> sense
>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> epochs while debugging an issue, not while trying
>>>> to
>>>>>>>>>> view
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> group
>>>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>>>>>> member's info. (Also member-epoch makes more sense
>>>>>> to
>>>>>>>>>> me,
>>>>>>>>>>>>>> even
>>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>>>>> if we
>>>>>>>>>>>>>>>>>>>>>>>>>>> end up agreeing in a --verbose). Related to both
>>>>>>>>> issues,
>>>>>>>>>>> my
>>>>>>>>>>>>>>>>> take is
>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>> whoever is close to the protocol will expect
>>>>>>>>>> member-epoch,
>>>>>>>>>>>>>>>>> whoever
>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>>> will probably won't even need to see the epochs at
>>>>>>>>> all.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> LM2. Why are the epochs defined as Optional (don't
>>>>>> we
>>>>>>>>>>> expect
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> always
>>>>>>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>>>>>>> them? with 0 initially, defined on the broker side
>>>>>> for
>>>>>>>>>> the
>>>>>>>>>>>>>> group
>>>>>>>>>>>>>>>>>>>>> ones,
>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>> client side for the member)
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> LM3. Why is the KIP including the –state option in
>>>>>> the
>>>>>>>>>>>>>> proposed
>>>>>>>>>>>>>>>>>>>>>>> changes?
>>>>>>>>>>>>>>>>>>>>>>>>>> (
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
>>>>>>>>>>>>>>>>>>>>>>>>>>> ).
>>>>>>>>>>>>>>>>>>>>>>>>>>> I get that the output would change in that example,
>>>>>>>>> but
>>>>>>>>>>> it’s
>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>> because
>>>>>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>>>> any change to the –state option itself. It's
>>>> because
>>>>>>>>> of
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>>>>>>>>>>>> proposed
>>>>>>>>>>>>>>>>>>>>>>>>>>> to the –described (required with the --state), and
>>>>>> the
>>>>>>>>>>>>>> changes
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>> --describe are already explained above (seems
>>>>>>>>> confusing,
>>>>>>>>>>> got
>>>>>>>>>>>>>> me
>>>>>>>>>>>>>>>>>>>>>>> looking
>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>> a change to the state filter which seemed
>>>>>> unrelated).
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> LM4. Group epoch and target assignment epoch are
>>>>>>>>>>> conceptually
>>>>>>>>>>>>>>>>> at the
>>>>>>>>>>>>>>>>>>>>>>>>>> group
>>>>>>>>>>>>>>>>>>>>>>>>>>> level. vs member epoch that lands at a member
>>>> level.
>>>>>>>>> So
>>>>>>>>>>>>>> wonder
>>>>>>>>>>>>>>>>> if we
>>>>>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>>>> show them accordingly (ex. using the --verbose
>>>>>> option)
>>>>>>>>>>>>>>>>>>>>>>>>>>> –describe –verbose => shows group epoch and target
>>>>>>>>>>> assignment
>>>>>>>>>>>>>>>>> epoch
>>>>>>>>>>>>>>>>>>>>>>>>>>> –describe –members –verbose => shows member
>>>>>>>>>> epoch
>>>>>>>>>>>>>>>>> (along
>>>>>>>>>>>>>>>>>>>>>>>>>>> with group epoch and target assignment epoch)
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Lianet
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield <
>>>>>>>>>>>>>>>>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I have a few comments.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> AS1: It seems to me that these new additions are
>>>>>> most
>>>>>>>>>>>>>> useful to
>>>>>>>>>>>>>>>>>>>>>>> people
>>>>>>>>>>>>>>>>>>>>>>>>>>>> trying to understand
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the progress of rebalancing in quite some detail.
>>>>>> The
>>>>>>>>>>>>>>>>> information
>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>> really not understandable
>>>>>>>>>>>>>>>>>>>>>>>>>>>> for most users who do not have deep knowledge of
>>>>>>>>>>>>>> KIP-848/932.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> As a result, I suggest for kafka-share-groups.sh
>>>>>> that
>>>>>>>>>> you
>>>>>>>>>>>>>> add a
>>>>>>>>>>>>>>>>>>>>>>>>>> --members
>>>>>>>>>>>>>>>>>>>>>>>>>>>> --verbose option
>>>>>>>>>>>>>>>>>>>>>>>>>>>> and only include the new information in the output
>>>>>>>>> for
>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> option,
>>>>>>>>>>>>>>>>>>>>>>>>>> rather
>>>>>>>>>>>>>>>>>>>>>>>>>>>> than changing the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> non-verbose --members output.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I also make a similar suggestion for
>>>>>>>>>>>>>> kafka-consumer-groups.sh
>>>>>>>>>>>>>>>>>>>>>>> --members
>>>>>>>>>>>>>>>>>>>>>>>>>>>> and only add the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> new information for the --verbose output.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> AS2: I strongly suggest that you use MEMBER-EPOCH
>>>>>>>>>>> instead of
>>>>>>>>>>>>>>>>>>>>>>>>>>>> CONSUMER-EPOCH.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it's more confusing not following the
>>>>>>>>>>> terminology of
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> KIPs.
>>>>>>>>>>>>>>>>>>>>>>>>>> For
>>>>>>>>>>>>>>>>>>>>>>>>>>>> one thing,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> we already have "member" in the admin client such
>>>>>> as
>>>>>>>>>>>>>>>>>>>>>>> MemberDescription.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's not a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> new term.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Andrew
>>>>>>>>>>>>>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>>>>>>>>>>>>>> From: PoAn Yang <pa...@apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sent: 25 October 2024 13:55
>>>>>>>>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
>>>>>>>>>>>>>> kafka-consumer-groups
>>>>>>>>>>>>>>>>>>>>> command
>>>>>>>>>>>>>>>>>>>>>>>>>> line
>>>>>>>>>>>>>>>>>>>>>>>>>>>> tool to support new consumer group
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Lucas,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the review!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Yes, I add related change for
>>>>>>>>> kafka-share-groups.sh
>>>>>>>>>> to
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> KIP.
>>>>>>>>>>>>>>>>>>>>>>> Could
>>>>>>>>>>>>>>>>>>>>>>>>>>>> you take a look? Thanks for the suggestion.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use
>>>>>>>>>>> MEMBER-EPOCH
>>>>>>>>>>>>>>>>> here,
>>>>>>>>>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>>>>>>>>>>>>>> confuse what is different between CONSUMER and
>>>>>>>>> MEMBER.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Frank,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks for the KIP!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) For consistency, should we do the same for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription,
>>>>>> etc. ?
>>>>>>>>>>> Even
>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>> we do
>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implement it right now if the share group
>>>>>>>>>> implementation
>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>>> still
>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> incomplete, it may make sense to include it in
>>>> the
>>>>>>>>>> KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH?
>>>>>>>>> That
>>>>>>>>>>> would
>>>>>>>>>>>>>>>>> seem
>>>>>>>>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consistent.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lucas
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <
>>>>>>>>>>>>>>>>> yangpoan@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to kick off the discussion of
>>>>>>>>> KIP-1099.
>>>>>>>>>>> This
>>>>>>>>>>>>>> KIP
>>>>>>>>>>>>>>>>>>>>>>> enhances
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups tools to include state
>>>>>> which
>>>>>>>>> is
>>>>>>>>>>>>>>>>> introduced
>>>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP-848.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups command
>>>>>> line
>>>>>>>>>>> tool
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> support
>>>>>>>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer group - Apache Kafka - Apache Software
>>>>>>>>>>> Foundation
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cwiki.apache.org
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [image: favicon.ico]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>>
>>>>
>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by PoAn Yang <ya...@gmail.com>.
Hi Andrew,
Thanks for the feedback.
AS4: Change TARGET-ASSIGNMENT-EPOCH to ASSIGNMENT-EPOCH for “—describe —state —verbose" in kafka-share-groups.sh.
AS5: Remove TARGET-EPOCH and TARGET-ASSIGNMENT.
Change CURRENT-EPOCH to MEMBER-EPOCH and CURRENT-ASSIGNMENT to ASSIGNMENT for “—describe —member —verbose” in kafka-share-groups.sh.
Thanks,
PoAn
> On Nov 25, 2024, at 10:48 PM, Andrew Schofield <an...@outlook.com> wrote:
>
> Hi PoAn,
> I've taken another pass over the KIP and have some more comments around the share
> group support.
>
> AS4: Share groups do not have the concept of a target assignment. As a result,
> the verbose state TARGET-ASSIGNMENT-EPOCH is a bit misleading from my
> point of view. ASSIGNMENT-EPOCH is better.
>
> AS5: Similarly for the verbose members output, the TARGET-EPOCH and
> TARGET-ASSIGNMENT are not really appropriate. Personally, I would prefer
> just MEMBER-EPOCH and ASSIGNMENT, and remove the "target" fields.
>
> Thanks,
> Andrew
>
> ________________________________________
> From: PoAn Yang <ya...@gmail.com>
> Sent: 24 November 2024 15:08
> To: dev@kafka.apache.org <de...@kafka.apache.org>
> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
>
> Hi Chia-Ping,
>
> Thanks for the suggestion.
>
> The “isClassic” field in ConsumerGroupDescribeResponse is a boolean value, so we can’t use it as the protocol string.
> I change the “PROTOCOL” column to “IS-CLASSIC” in kafka-consumer-groups.sh.
> It returns “true” if a member is in a classic group or it’s a classic member in a consumer group.
> In other cases, it returns “false”.
>
> I have a draft implementation in https://github.com/apache/kafka/pull/17872.
>
> Thanks,
> PoAn
>
>> On Nov 23, 2024, at 1:05 PM, Chia-Ping Tsai <ch...@gmail.com> wrote:
>>
>> hi PoAn
>>
>> `isClassic` is good to me :)
>>
>> Best,
>> Chia-Ping
>>
>> PoAn Yang <ya...@gmail.com> 於 2024年11月23日 週六 下午12:35寫道：
>>
>>> Hi Chia-Ping / David,
>>>
>>> Thanks for the great suggestion.
>>>
>>> How about using “isClassic”?
>>> The term like “isLegacy” may be equivocal in the future.
>>> For example, if we have a newer version of consumer / share consumer, the
>>> meaning will be different for different cases.
>>> If we use “isClassic”, for MemberDescription in ClassicGroupDescription,
>>> the value can be always “true”.
>>> In ConsumerGroupDescription, it’s “true” when it’s a classic member.
>>> In ShareGroupDescription, it’s always “false”.
>>>
>>> Thanks,
>>> PoAn
>>>
>>>> On Nov 23, 2024, at 1:02 AM, Chia-Ping Tsai <ch...@gmail.com> wrote:
>>>>
>>>> hi DJ
>>>>
>>>>> My goal was to have a way to tell the user that his (new) consumer group
>>>> may have non-upgraded members yet. I wonder if we could use a boolean to
>>>> signal this, e.g. isLegacy. Then, in the CLI we could have a column which
>>>> is only displayed when they are non upgraded members. What do you think?
>>>>
>>>> Yes, this is a good approach, but the downside is that the new isLegacy
>>>> field in MemberDescription would be irrelevant for classic and shared
>>>> groups. This would require thorough documentation to clarify its purpose.
>>>>
>>>> Additionally, we could use a boolean instead of a string for the new
>>> field
>>>> in the RPC to reduce the request size.
>>>> Best,
>>>> Chia-Ping
>>>>
>>>>
>>>>
>>>> David Jacot <dj...@confluent.io.invalid> 於 2024年11月23日 週六 上午12:41寫道：
>>>>
>>>>> Hi both,
>>>>>
>>>>> Ah, I understand what you mean now. You're saying that we have two
>>> protocol
>>>>> fields. One at the group level and one at the member level (the new
>>> one).
>>>>> The two fields are actually two different things. The former is the
>>> name of
>>>>> the embedded protocol used in the classic protocol whereas the latter is
>>>>> the name of the protocol/api used. This is indeed confusing.
>>>>>
>>>>> My goal was to have a way to tell the user that his (new) consumer group
>>>>> may have non-upgraded members yet. I wonder if we could use a boolean to
>>>>> signal this, e.g. isLegacy. Then, in the CLI we could have a column
>>> which
>>>>> is only displayed when they are non upgraded members. What do you think?
>>>>>
>>>>> If we cannot find a good way to represent this, we could drop it from
>>> this
>>>>> KIP too.
>>>>>
>>>>> Best,
>>>>> David
>>>>>
>>>>> On Fri, Nov 22, 2024 at 5:12 PM PoAn Yang <ya...@gmail.com> wrote:
>>>>>
>>>>>> Hi Chia-Ping / David,
>>>>>>
>>>>>> Thanks for the review and suggestions.
>>>>>>
>>>>>>>> I don't fully follow your comment so please excuse me if I say
>>>>> something
>>>>>>>> wrong.
>>>>>>
>>>>>> I think Chia-Ping wants to say: originally, the default protocol in
>>>>>> classic group is “consumer”.
>>>>>> After we have a new consumer group, the protocol of a classic member in
>>>>>> consumer group is “classic”.
>>>>>> The MemberDescription is used by both ClassicGroupDescription and
>>>>>> ConsumerGroupDescription.
>>>>>> If we add the protocol field to MemberDescription, the protocol of a
>>>>>> classic member in classic group is “consumer”, but it’s “classic” in
>>>>>> consumer group.
>>>>>> To avoid misleading word, we can add a new collection of member
>>> protocols
>>>>>> to ConsumerGroupDescription only.
>>>>>> The original goal is to show member protocol during consumer migration,
>>>>> so
>>>>>> we don’t need to add this information to ClassicGroupDescription.
>>>>>>
>>>>>> Please correct me if I misunderstood anything.
>>>>>>
>>>>>> Thanks,
>>>>>> PoAn
>>>>>>
>>>>>>> On Nov 22, 2024, at 7:00 PM, Chia-Ping Tsai <ch...@gmail.com>
>>>>> wrote:
>>>>>>>
>>>>>>> hi DJ
>>>>>>>
>>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and
>>>>> change
>>>>>>> return type of ModernGroup#protocolType?
>>>>>>>
>>>>>>> This is not my comment ... sorry for the incorrect quote :(
>>>>>>>
>>>>>>>> A member in a classic group should report "classic".
>>>>>>>
>>>>>>> I completely agree. However, in this case,
>>>>>> ClassicGroupDescription#protocol
>>>>>>> returns 'consumer', while ClassicGroupDescription#members[x].protocol
>>>>>>> returns 'classic'. This inconsistency feels a bit strange to me.
>>>>>>>
>>>>>>> Best,
>>>>>>> Chia-Ping
>>>>>>>
>>>>>>>
>>>>>>> David Jacot <dj...@confluent.io.invalid> 於 2024年11月22日 週五 下午6:51寫道：
>>>>>>>
>>>>>>>> Hi Chia,
>>>>>>>>
>>>>>>>>>> LM4: I agree it’s better to use a type like GroupProtocol enum, but
>>>>> I
>>>>>>>> would like to align the format with kafka-groups.sh.
>>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and
>>>>> change
>>>>>>>> return type of ModernGroup#protocolType?
>>>>>>>>
>>>>>>>> My understanding is that `GroupProtocol` is used to drive the config
>>>>> of
>>>>>> the
>>>>>>>> `Consumer`. Hence adding `Share` seems wrong to me. Or are you
>>>>>> referring to
>>>>>>>> another GroupProtocol enum? Perhaps GroupType enum?
>>>>>>>>
>>>>>>>>> In fact, I'm wondering if we should even add 'protocol' to
>>>>>>>> MemberDescription,
>>>>>>>> since the field has different values between classic and consumer
>>>>>> groups—it
>>>>>>>> shows 'consumer' in classic groups, whereas it shows 'classic' in
>>>>>> consumer
>>>>>>>> groups. Yes, it seems like a typo, but it's true. This inconsistency
>>>>>> feels
>>>>>>>> confusing to me. For example, users call Admin#describeConsumerGroups
>>>>>> and
>>>>>>>> see that a member has the 'consumer' protocol when it's in a classic
>>>>>> group.
>>>>>>>> Then, if the group is upgraded to a consumer group, calling
>>>>>>>> Admin#describeConsumerGroups again shows the protocol as 'classic'
>>> for
>>>>>> the
>>>>>>>> same member.
>>>>>>>>
>>>>>>>> I don't fully follow your comment so please excuse me if I say
>>>>> something
>>>>>>>> wrong.
>>>>>>>>
>>>>>>>> I think that it should work as follows:
>>>>>>>> * A member in a classic group should report "classic".
>>>>>>>> * A member using the consumer protocol in a consumer group should
>>>>> report
>>>>>>>> "consumer".
>>>>>>>> * A member using the classic protocol in a consumer group should
>>>>> report
>>>>>>>> "classic".
>>>>>>>> * A member in a share group should report "share".
>>>>>>>>
>>>>>>>>> If all we want to do is distinguish classic members in the
>>>>>>>> ConsumerGroupDescription, then we shouldn't modify the common
>>>>>>>> MemberDescription structure. Instead, we could add a collection to
>>>>>>>> ConsumerGroupDescription to trace members using the classic protocol,
>>>>>> which
>>>>>>>> would allow us to enrich the output of command-line tools.
>>>>>>>>
>>>>>>>> Personally, I believe that using different collections per protocol
>>>>>> types
>>>>>>>> is not necessary. A field in the member is more than enough.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> DJ
>>>>>>>>
>>>>>>>> On Fri, Nov 22, 2024 at 12:01 PM Chia-Ping Tsai <ch...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> hi PoAn
>>>>>>>>>
>>>>>>>>>> LM4: I agree it’s better to use a type like GroupProtocol enum, but
>>>>> I
>>>>>>>>> would like to align the format with kafka-groups.sh.
>>>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and
>>>>>> change
>>>>>>>>> return type of ModernGroup#protocolType?
>>>>>>>>>
>>>>>>>>> MemberDescription is also used by ClassicGroupDescription, and in
>>>>>> classic
>>>>>>>>> groups, the protocol can be an arbitrary string, so we can't use an
>>>>>> enum.
>>>>>>>>>
>>>>>>>>> In fact, I'm wondering if we should even add 'protocol' to
>>>>>>>>> MemberDescription,
>>>>>>>>> since the field has different values between classic and consumer
>>>>>>>> groups—it
>>>>>>>>> shows 'consumer' in classic groups, whereas it shows 'classic' in
>>>>>>>> consumer
>>>>>>>>> groups. Yes, it seems like a typo, but it's true. This inconsistency
>>>>>>>> feels
>>>>>>>>> confusing to me. For example, users call
>>> Admin#describeConsumerGroups
>>>>>> and
>>>>>>>>> see that a member has the 'consumer' protocol when it's in a classic
>>>>>>>> group.
>>>>>>>>> Then, if the group is upgraded to a consumer group, calling
>>>>>>>>> Admin#describeConsumerGroups again shows the protocol as 'classic'
>>>>> for
>>>>>>>> the
>>>>>>>>> same member.
>>>>>>>>>
>>>>>>>>> If all we want to do is distinguish classic members in the
>>>>>>>>> ConsumerGroupDescription, then we shouldn't modify the common
>>>>>>>>> MemberDescription structure. Instead, we could add a collection to
>>>>>>>>> ConsumerGroupDescription to trace members using the classic
>>> protocol,
>>>>>>>> which
>>>>>>>>> would allow us to enrich the output of command-line tools.
>>>>>>>>> Best,
>>>>>>>>> Chia-Ping
>>>>>>>>>
>>>>>>>>> PoAn Yang <ya...@gmail.com> 於 2024年11月22日 週五 下午3:46寫道：
>>>>>>>>>
>>>>>>>>>> Hi Lianet / Jeff,
>>>>>>>>>>
>>>>>>>>>> Thanks for the review.
>>>>>>>>>>
>>>>>>>>>> LM5: The kafka-share-groups.sh also uses “—describe” to show
>>> offsets
>>>>>>>>>> information.
>>>>>>>>>> Change to use “—describe —state —verbose” to show group level
>>>>>>>> information
>>>>>>>>>> in kafka-share-groups.sh.
>>>>>>>>>>
>>>>>>>>>> LM6: Update the example.
>>>>>>>>>>
>>>>>>>>>> LM7: Add a description to mention “—verbose” is a new option in
>>>>>>>>>> ShareGroupCommandOptions.
>>>>>>>>>>
>>>>>>>>>> JK2: Yes, add the statement about format change for ASSIGNMENT
>>>>> value.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> PoAn
>>>>>>>>>>
>>>>>>>>>>> On Nov 22, 2024, at 7:35 AM, Jeff Kim <je...@apache.org>
>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>
>>>>>>>>>>> JK2: Can we state in the KIP that we will reformat the existing
>>>>>>>>>>> classic group's `--members --verbose` as well, specifically the
>>>>>>>>>>> "ASSIGNMENT" column to include the topic name?
>>>>>>>>>>>
>>>>>>>>>>> Thanks!
>>>>>>>>>>>
>>>>>>>>>>> On 2024/11/21 14:33:37 "Lianet M." wrote:
>>>>>>>>>>>> Thanks for the updates PoAn!
>>>>>>>>>>>>
>>>>>>>>>>>> LM6. Just a nit, under the --describe --state --verbose the
>>> header
>>>>>>>> is
>>>>>>>>>> fine
>>>>>>>>>>>> but the example is still missing the --state argument.
>>>>>>>>>>>>
>>>>>>>>>>>> LM7. Under the Proposed changes for kafka-share-groups.sh, should
>>>>> we
>>>>>>>>>>>> clearly mention that the KIP adds a new --verbose option? I think
>>>>>>>> it's
>>>>>>>>>>>> confusing because it's presented as if the option already exists
>>>>> and
>>>>>>>>>> we're
>>>>>>>>>>>> only changing the output (which is the case for
>>>>>>>> kafka-consumer-groups
>>>>>>>>>> but
>>>>>>>>>>>> not for kafka-share-groups)
>>>>>>>>>>>>
>>>>>>>>>>>> That's all on my side. Thanks!
>>>>>>>>>>>>
>>>>>>>>>>>> Lianet
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Nov 20, 2024 at 11:12 PM PoAn Yang <ya...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Lianet / Jeff,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for review and suggestions.
>>>>>>>>>>>>>
>>>>>>>>>>>>> LM4: At group level, the protocol field is from
>>>>>>>>>> ModernGroup#protocolType.
>>>>>>>>>>>>> It’s a string.
>>>>>>>>>>>>> The value is defined in different places like
>>>>>>>>> ShareGroup#PROTOCOL_TYPE
>>>>>>>>>> and
>>>>>>>>>>>>> ConsumerProtocol#PROTOCOL_TYPE.
>>>>>>>>>>>>> In KIP-1043 <
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=305171038#KIP1043:Administrationofgroups-kafka-groups.sh
>>>>>>>>>>> ,
>>>>>>>>>>>>> the kafka-groups.sh also shows protocol like “classic” /
>>>>>>>> “consumer” /
>>>>>>>>>>>>> “share”.
>>>>>>>>>>>>> I agree it’s better to use a type like GroupProtocol enum, but I
>>>>>>>>> would
>>>>>>>>>>>>> like to align the format with kafka-groups.sh.
>>>>>>>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol
>>> and
>>>>>>>>>> change
>>>>>>>>>>>>> return type of ModernGroup#protocolType?
>>>>>>>>>>>>>
>>>>>>>>>>>>> LM5 / JK1: Sorry, I misunderstood LM3, so I deleted —state in
>>> the
>>>>>>>>>> title.
>>>>>>>>>>>>> You’re correct. I would like to use `—describe —state —verbose`
>>>>> to
>>>>>>>>> show
>>>>>>>>>>>>> group level information.
>>>>>>>>>>>>> For `—describe —verbose`, it will show output as same as
>>>>> `—describe
>>>>>>>>>>>>> —offsets —verbose`,
>>>>>>>>>>>>> because `—describe` also shows output as same as `—describe
>>>>>>>>> —offsets`.
>>>>>>>>>>>>>
>>>>>>>>>>>>> JK2: Yes, I only use consumer group in the sample output, but
>>> the
>>>>>>>>>> classic
>>>>>>>>>>>>> group will use the same format as well.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Nov 21, 2024, at 8:28 AM, Jeff Kim <je...@apache.org>
>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for the KIP. I have some questions:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> JK1: (Realized this is similar to LM5) On `--describe
>>> --verbose`
>>>>>>>>>>>>> proposed changes - doesn't `--describe $group` default to
>>>>> printing
>>>>>>>>> the
>>>>>>>>>>>>> offsets? Perhaps you're referring to the `--state` argument?
>>>>> Also,
>>>>>>>>>> would
>>>>>>>>>>>>> that mean the default `--describe $group --verbose` command
>>> would
>>>>>>>> not
>>>>>>>>>> print
>>>>>>>>>>>>> the added field to `--offsets verbose` (leader-epoch) or would
>>>>> it?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> JK2: Looking at ConsumerGroupCommand.java, the existing
>>>>>>>> "ASSIGNMENT"
>>>>>>>>>>>>> column under `--members --verbose` does group the topic
>>>>> partitions
>>>>>>>> by
>>>>>>>>>> topic
>>>>>>>>>>>>> but does not prefix the grouping with the topic name like you're
>>>>>>>>>> proposal:
>>>>>>>>>>>>> "my_topic:0,1;new_topic:0,1". Should we do apply the same format
>>>>>>>> for
>>>>>>>>>> the
>>>>>>>>>>>>> classic group as well?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 2024/11/20 21:32:41 "Lianet M." wrote:
>>>>>>>>>>>>>>> Hello PoAn, just of couple more minor comments:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> LM4. Regarding the new “protocol” field added to
>>>>>>>> MemberDescription.
>>>>>>>>>>>>> Should
>>>>>>>>>>>>>>> we consider reusing the existing GroupProtocol enum instead of
>>>>>>>>>> String?
>>>>>>>>>>>>>>> (It’s the one we use from the consumer side to refer to the
>>>>>>>>> protocol
>>>>>>>>>> in
>>>>>>>>>>>>>>> use, just missing Share I notice).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> LM5. Regarding the change to the output for
>>>>>>>> kafka-consumer-groups,
>>>>>>>>>> the
>>>>>>>>>>>>>>> command shown does not include the —state option, but the
>>>>> output
>>>>>>>>>> shows
>>>>>>>>>>>>>>> state info (state, #members, epochs). I would guess that we
>>>>> want
>>>>>>>> to
>>>>>>>>>>>>> modify
>>>>>>>>>>>>>>> the output only when we describe a group with the —state
>>>>> —verbose
>>>>>>>>>>>>> option,
>>>>>>>>>>>>>>> is my understanding right? If my understanding is right we’re
>>>>>>>> just
>>>>>>>>>>>>> missing
>>>>>>>>>>>>>>> adding the —state in the example, and the KIP does not
>>>>> introduce
>>>>>>>>> any
>>>>>>>>>>>>>>> changes to the —describe —verbose option. (If not, it would
>>>>> mean
>>>>>>>> a
>>>>>>>>>>>>> bigger
>>>>>>>>>>>>>>> change to the output of —describe —verbose which I expect is
>>>>> not
>>>>>>>>> the
>>>>>>>>>>>>>>> intention?)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Lianet
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Nov 20, 2024 at 2:12 AM PoAn Yang <yangpoan@gmail.com
>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Chia-Ping,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks for the review and suggestions.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> CI0: Thanks for the reminder. Update validVersions in
>>>>>>>>>>>>>>>> ConsumerGroupDescribeRequest to 0-1.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> CI1: Yes, I use `ConsumerGroupMember#useClassicProtocol` to
>>>>>>>> check
>>>>>>>>>>>>> whether
>>>>>>>>>>>>>>>> a member in consumer group uses “classic” or “consumer”
>>>>>>>> protocol.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> CI2: Yes, a member in share group always uses “share”
>>>>> protocol.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> CI3: Add a table to show meaning of “classic”, “consumer”,
>>> and
>>>>>>>>>> “share”
>>>>>>>>>>>>>>>> protocol.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> BTW, the vote thread is in
>>>>>>>>>>>>>>>>
>>>>>>>> https://lists.apache.org/thread/rb25tf75tzf4c7jqqldlo5jh9w8chsq6.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Nov 20, 2024, at 11:46 AM, Chia-Ping Tsai <
>>>>>>>>> chia7712@apache.org>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> hi PoAn
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> CI0: We have to bump the version of
>>>>>>>> ConsumerGroupDescribeRequest
>>>>>>>>>> as
>>>>>>>>>>>>>>>> well, so server can distinguish the new and old behavior.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> CI1: the type of new filed is string, so I guess you plan to
>>>>>>>> use
>>>>>>>>>>>>>>>> `ConsumerGroupMember#useClassicProtocol` [0] flag to return
>>>>>>>> either
>>>>>>>>>>>>>>>> "classic" or "consumer", right?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> CI2: `MemberDescription` is used by `ShareGroupDescription`
>>>>>>>> too,
>>>>>>>>> so
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> new filed protocol in shared group is always "shared", right?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> CI3: Could you consider adding a table to show the value of
>>>>> the
>>>>>>>>>>>>> protocol
>>>>>>>>>>>>>>>> field in each case? Andrew has a beautiful table in KIP-1043
>>>>>>>> that
>>>>>>>>>>>>> lists all
>>>>>>>>>>>>>>>> possible protocol names.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [0]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>> https://github.com/apache/kafka/blob/441a6d0b790f4a17b454caeea7588a6b90fbd9db/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/consumer/ConsumerGroupMember.java#L454
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 2024/11/19 15:45:16 PoAn Yang wrote:
>>>>>>>>>>>>>>>>>> Hi David,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks for the reminder.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> DJ4: Add related content to the KIP.
>>>>>>>>>>>>>>>>>> Bump validVersions in ConsumerGroupDescribeResponse to
>>>>> “0-1”.
>>>>>>>>>>>>>>>>>> Add a new field “Protocol” to
>>> ConsumerGroupDescribeResponse.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Nov 19, 2024, at 3:57 PM, David Jacot
>>>>>>>>>>>>> <dj...@confluent.io.INVALID>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> DJ4: Could you please also include the required changes to
>>>>>>>> the
>>>>>>>>>>>>>>>>>>> ConsumerGroupDescribed API in the public interface
>>> section?
>>>>>>>> We
>>>>>>>>>>>>>>>> basically
>>>>>>>>>>>>>>>>>>> need to bump the version, add the new field, etc.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> David
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang <
>>>>>>>> yangpoan@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi David,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> DJ4: Add a new field protocol to MemberDescription,
>>>>>>>>>>>>>>>>>>>> so the command line tool can show protocol information
>>>>> when
>>>>>>>>>> users
>>>>>>>>>>>>>>>> describe
>>>>>>>>>>>>>>>>>>>> members.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> If there is no further suggestion, I will start a vote
>>>>>>>> thread
>>>>>>>>>>>>> today.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Nov 15, 2024, at 11:50 PM, David Jacot
>>>>>>>>>>>>>>>> <dj...@confluent.io.INVALID>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> DJ2: Using "-" sounds good to me.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> DJ3: That seems reasonable to me.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> DJ4: Why not add it right now? I don't want to change
>>> the
>>>>>>>>>> output
>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> tool too many times.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>> David
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang <
>>>>>>>>> yangpoan@gmail.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi David / Andrew,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks for review. Thanks Andrew for picking up
>>>>>>>>>>>>>>>> kafka-share-groups.sh
>>>>>>>>>>>>>>>>>>>>>> implementation.
>>>>>>>>>>>>>>>>>>>>>> I will handle kafka-consumer-groups.sh.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> DJ3: After discussing with @Chia-Ping Tsai, we think
>>>>> that
>>>>>>>>>> using
>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>> format
>>>>>>>>>>>>>>>>>>>>>> is more clear.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> The new format will be like
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>> <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Using colon(:) to concat topic name and partition IDs.
>>>>>>>>>>>>>>>>>>>>>> Using comma(,) to concat partition IDs within same
>>> topic
>>>>>>>>> name.
>>>>>>>>>>>>>>>>>>>>>> Using semicolon(;) to concat topic strings.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> AS3: Fix it with kafka-share-groups.sh. Thanks.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> If there is no further suggestion, I will start a vote
>>>>>>>>> thread
>>>>>>>>>>>>> next
>>>>>>>>>>>>>>>>>>>> Monday.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield <
>>>>>>>>>>>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>>>>>>>>>>> DJ2: I was just going to comment that "-" would be a
>>>>> more
>>>>>>>>>>>>>>>> appropriate
>>>>>>>>>>>>>>>>>>>>>> missing value, but
>>>>>>>>>>>>>>>>>>>>>>> you got there first.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> AS3: The examples for kafka-share-groups.sh include
>>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups.sh in the
>>>>>>>>>>>>>>>>>>>>>>> command line.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> If this is accepted in time, I'm happy to pick up the
>>>>>>>>>>>>>>>> implementation of
>>>>>>>>>>>>>>>>>>>>>> the share groups
>>>>>>>>>>>>>>>>>>>>>>> part of this if it helps.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> Andrew
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>>>>>>>>> From: Frank Yang <ya...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>> Sent: 14 November 2024 10:48
>>>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
>>>>>>>>> kafka-consumer-groups
>>>>>>>>>>>>>>>> command
>>>>>>>>>>>>>>>>>>>>>> line tool to support new consumer group
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi David,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks for the review and suggestion! I would like to
>>>>> get
>>>>>>>>>> this
>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>> AK
>>>>>>>>>>>>>>>>>>>> 4.0
>>>>>>>>>>>>>>>>>>>>>> as well. I will do my best.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> DJ1: Update KIP to put GROUP-EPOCH and
>>>>>>>>>> TARGET-ASSIGNMENT-EPOCH
>>>>>>>>>>>>>>>> before
>>>>>>>>>>>>>>>>>>>>>> #MEMBERS.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> DJ2: I prefer to follow current missing column value
>>>>> “-“.
>>>>>>>>>>>>>>>> (reference <
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92
>>>>>>>>>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> DJ3: Update KIP to use CURRENT-EPOCH
>>> CURRENT-ASSIGNMENT
>>>>>>>>>>>>>>>> TARGET-EPOCH
>>>>>>>>>>>>>>>>>>>>>> TARGET-ASSIGNMENT.
>>>>>>>>>>>>>>>>>>>>>>> Remove GROUP-EPOCH.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> For assignment value, it follows current output
>>>>>>>> (reference
>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418
>>>>>>>>>>>>>>>>>>>>> ).
>>>>>>>>>>>>>>>>>>>>>> I think the form of `topicid-partitionid` is more
>>> clear.
>>>>>>>>>>>>>>>>>>>>>>> If we would like to use this form, I will update both
>>>>>>>>> output
>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups.sh and kafka-share-groups.sh.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> DJ4: It looks like DescribeGroupsResponseData only has
>>>>>>>>>> protocol
>>>>>>>>>>>>>>>> type at
>>>>>>>>>>>>>>>>>>>>>> group level.
>>>>>>>>>>>>>>>>>>>>>>> Both DescribeGroupsResponseData and
>>>>>>>>>>>>>>>> ConsumerGroupDescribeResponseData
>>>>>>>>>>>>>>>>>>>>>> don’t have protocol at member level.
>>>>>>>>>>>>>>>>>>>>>>> Could we use a followup to add it?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> DJ5: Update KIP to put LEADER-EPOCH before
>>>>>>>> CURRENT-OFFSET.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot
>>>>>>>>>>>>>>>> <djacot@confluent.io.INVALID
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP! I have a few minor
>>>>>>>>> comments/suggestions:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> DJ1: In the output of `--describe --verbose`, I would
>>>>>>>>>> suggest
>>>>>>>>>>>>>>>> putting
>>>>>>>>>>>>>>>>>>>>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before
>>>>>>>>>> `#MEMBERS`.
>>>>>>>>>>>>>>>>>>>>>>>> DJ2: Continuing on the above, I assume that we will
>>>>>>>> print
>>>>>>>>>> out
>>>>>>>>>>>>> N/A
>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> fields not supported by classic groups. Is this
>>>>> correct?
>>>>>>>>>>>>>>>>>>>>>>>> DJ3: In the output of `--describe --members
>>>>> --verbose`,
>>>>>>>> I
>>>>>>>>>>>>> wonder
>>>>>>>>>>>>>>>> if we
>>>>>>>>>>>>>>>>>>>>>>>> should use the following order and terms:
>>>>> CURRENT-EPOCH
>>>>>>>>>>>>>>>>>>>>>> CURRENT-ASSIGNMENT
>>>>>>>>>>>>>>>>>>>>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the
>>>>>>>>>> GROUP-EPOCH
>>>>>>>>>>>>>>>>>>>> because
>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>> is already in the group description. The value `(0)`
>>>>>>>> used
>>>>>>>>>> for
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> assignment is incorrect. Here I suppose that we will
>>>>>>>> print
>>>>>>>>>> out
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> list
>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>> partitions in the form of `topicid-partitionid`.
>>>>>>>>>>>>>>>>>>>>>>>> DJ4: Continuing on the above, I wonder if we should
>>>>> also
>>>>>>>>> add
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> protocol
>>>>>>>>>>>>>>>>>>>>>>>> used `classic` or `consumer`. For context, it is
>>>>>>>> possible
>>>>>>>>> to
>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>> `classic`
>>>>>>>>>>>>>>>>>>>>>>>> members and `consumer` members in a `consumer` group
>>>>>>>>> during
>>>>>>>>>> an
>>>>>>>>>>>>>>>> online
>>>>>>>>>>>>>>>>>>>>>>>> upgrade from the classic protocol to the consumer
>>>>>>>>> protocol.
>>>>>>>>>>>>> Having
>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>> information may be handy for administrators. What do
>>>>> you
>>>>>>>>>> think?
>>>>>>>>>>>>>>>>>>>>>>>> DJ5: In the output of `--describe --offsets
>>>>> --verbose`,
>>>>>>>> I
>>>>>>>>>> would
>>>>>>>>>>>>>>>>>>>> suggest
>>>>>>>>>>>>>>>>>>>>>>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> It would be great if we could get this one in AK 4.0
>>>>> as
>>>>>>>> it
>>>>>>>>>>>>>>>> changes the
>>>>>>>>>>>>>>>>>>>>>>>> output of the command.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> DJ
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <
>>>>>>>>>> yangpoan@gmail.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Hi Sean / Andrew / Lianet,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for all your review and suggestions.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> AS1, LM1, LM4: Change to add KIP-848 information
>>> when
>>>>>>>>> users
>>>>>>>>>>>>> give
>>>>>>>>>>>>>>>>>>>>>> —verbose
>>>>>>>>>>>>>>>>>>>>>>>>> option.
>>>>>>>>>>>>>>>>>>>>>>>>> —describe —verbose: shows group epoch and target
>>>>>>>>> assignment
>>>>>>>>>>>>>>>> epoch.
>>>>>>>>>>>>>>>>>>>>>>>>> —describe —members —verbose: shows above
>>> information,
>>>>>>>>>> member
>>>>>>>>>>>>>>>> epoch,
>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>> target assignment.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> AS2: Change to use MEMBER-EPOCH to align with
>>> KIP-848
>>>>>>>>>>>>> definition.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> LM2: For classic group, it doesn’t have epoch, so I
>>>>> use
>>>>>>>>>>>>> Optional
>>>>>>>>>>>>>>>>>>>>>> fields in
>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupDescription.
>>>>>>>>>>>>>>>>>>>>>>>>> For share group, it relies on KIP-848. It must have
>>>>>>>>> epoch,
>>>>>>>>>> so
>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>>>> int
>>>>>>>>>>>>>>>>>>>>>>>>> fields in ShareGroupDescription.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> LM3: Remove —state change. We can get group level
>>>>>>>>>> information
>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>> —describe
>>>>>>>>>>>>>>>>>>>>>>>>> —verbose.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> SQ1: Add LEADER-EPOCH when users give —describe
>>>>>>>> —offsets
>>>>>>>>>>>>>>>> —verbose.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <
>>>>>>>>> lianetmr@gmail.com
>>>>>>>>>>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Frank, thanks for the KIP! A few comments:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> LM1. I strongly agree with Andrew's suggestion of
>>>>>>>> moving
>>>>>>>>>> this
>>>>>>>>>>>>>>>> into a
>>>>>>>>>>>>>>>>>>>>>>>>>> --verbose option. I expect someone would only
>>>>> attempt
>>>>>>>> to
>>>>>>>>>> make
>>>>>>>>>>>>>>>> sense
>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> epochs while debugging an issue, not while trying
>>> to
>>>>>>>>> view
>>>>>>>>>> the
>>>>>>>>>>>>>>>> group
>>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>>>>> member's info. (Also member-epoch makes more sense
>>>>> to
>>>>>>>>> me,
>>>>>>>>>>>>> even
>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>>>> if we
>>>>>>>>>>>>>>>>>>>>>>>>>> end up agreeing in a --verbose). Related to both
>>>>>>>> issues,
>>>>>>>>>> my
>>>>>>>>>>>>>>>> take is
>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>> whoever is close to the protocol will expect
>>>>>>>>> member-epoch,
>>>>>>>>>>>>>>>> whoever
>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>> will probably won't even need to see the epochs at
>>>>>>>> all.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> LM2. Why are the epochs defined as Optional (don't
>>>>> we
>>>>>>>>>> expect
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> always
>>>>>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>>>>>> them? with 0 initially, defined on the broker side
>>>>> for
>>>>>>>>> the
>>>>>>>>>>>>> group
>>>>>>>>>>>>>>>>>>>> ones,
>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>> client side for the member)
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> LM3. Why is the KIP including the –state option in
>>>>> the
>>>>>>>>>>>>> proposed
>>>>>>>>>>>>>>>>>>>>>> changes?
>>>>>>>>>>>>>>>>>>>>>>>>> (
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
>>>>>>>>>>>>>>>>>>>>>>>>>> ).
>>>>>>>>>>>>>>>>>>>>>>>>>> I get that the output would change in that example,
>>>>>>>> but
>>>>>>>>>> it’s
>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>> because
>>>>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>>> any change to the –state option itself. It's
>>> because
>>>>>>>> of
>>>>>>>>>> the
>>>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>>>>>>>>>>> proposed
>>>>>>>>>>>>>>>>>>>>>>>>>> to the –described (required with the --state), and
>>>>> the
>>>>>>>>>>>>> changes
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>> --describe are already explained above (seems
>>>>>>>> confusing,
>>>>>>>>>> got
>>>>>>>>>>>>> me
>>>>>>>>>>>>>>>>>>>>>> looking
>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>> a change to the state filter which seemed
>>>>> unrelated).
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> LM4. Group epoch and target assignment epoch are
>>>>>>>>>> conceptually
>>>>>>>>>>>>>>>> at the
>>>>>>>>>>>>>>>>>>>>>>>>> group
>>>>>>>>>>>>>>>>>>>>>>>>>> level. vs member epoch that lands at a member
>>> level.
>>>>>>>> So
>>>>>>>>>>>>> wonder
>>>>>>>>>>>>>>>> if we
>>>>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>>> show them accordingly (ex. using the --verbose
>>>>> option)
>>>>>>>>>>>>>>>>>>>>>>>>>> –describe –verbose => shows group epoch and target
>>>>>>>>>> assignment
>>>>>>>>>>>>>>>> epoch
>>>>>>>>>>>>>>>>>>>>>>>>>> –describe –members –verbose => shows member
>>>>>>>>> epoch
>>>>>>>>>>>>>>>> (along
>>>>>>>>>>>>>>>>>>>>>>>>>> with group epoch and target assignment epoch)
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Lianet
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield <
>>>>>>>>>>>>>>>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I have a few comments.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> AS1: It seems to me that these new additions are
>>>>> most
>>>>>>>>>>>>> useful to
>>>>>>>>>>>>>>>>>>>>>> people
>>>>>>>>>>>>>>>>>>>>>>>>>>> trying to understand
>>>>>>>>>>>>>>>>>>>>>>>>>>> the progress of rebalancing in quite some detail.
>>>>> The
>>>>>>>>>>>>>>>> information
>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>> really not understandable
>>>>>>>>>>>>>>>>>>>>>>>>>>> for most users who do not have deep knowledge of
>>>>>>>>>>>>> KIP-848/932.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> As a result, I suggest for kafka-share-groups.sh
>>>>> that
>>>>>>>>> you
>>>>>>>>>>>>> add a
>>>>>>>>>>>>>>>>>>>>>>>>> --members
>>>>>>>>>>>>>>>>>>>>>>>>>>> --verbose option
>>>>>>>>>>>>>>>>>>>>>>>>>>> and only include the new information in the output
>>>>>>>> for
>>>>>>>>>> that
>>>>>>>>>>>>>>>> option,
>>>>>>>>>>>>>>>>>>>>>>>>> rather
>>>>>>>>>>>>>>>>>>>>>>>>>>> than changing the
>>>>>>>>>>>>>>>>>>>>>>>>>>> non-verbose --members output.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I also make a similar suggestion for
>>>>>>>>>>>>> kafka-consumer-groups.sh
>>>>>>>>>>>>>>>>>>>>>> --members
>>>>>>>>>>>>>>>>>>>>>>>>>>> and only add the
>>>>>>>>>>>>>>>>>>>>>>>>>>> new information for the --verbose output.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> AS2: I strongly suggest that you use MEMBER-EPOCH
>>>>>>>>>> instead of
>>>>>>>>>>>>>>>>>>>>>>>>>>> CONSUMER-EPOCH.
>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it's more confusing not following the
>>>>>>>>>> terminology of
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> KIPs.
>>>>>>>>>>>>>>>>>>>>>>>>> For
>>>>>>>>>>>>>>>>>>>>>>>>>>> one thing,
>>>>>>>>>>>>>>>>>>>>>>>>>>> we already have "member" in the admin client such
>>>>> as
>>>>>>>>>>>>>>>>>>>>>> MemberDescription.
>>>>>>>>>>>>>>>>>>>>>>>>>>> It's not a
>>>>>>>>>>>>>>>>>>>>>>>>>>> new term.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>> Andrew
>>>>>>>>>>>>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>>>>>>>>>>>>> From: PoAn Yang <pa...@apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Sent: 25 October 2024 13:55
>>>>>>>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
>>>>>>>>>>>>> kafka-consumer-groups
>>>>>>>>>>>>>>>>>>>> command
>>>>>>>>>>>>>>>>>>>>>>>>> line
>>>>>>>>>>>>>>>>>>>>>>>>>>> tool to support new consumer group
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Lucas,
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the review!
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Yes, I add related change for
>>>>>>>> kafka-share-groups.sh
>>>>>>>>> to
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> KIP.
>>>>>>>>>>>>>>>>>>>>>> Could
>>>>>>>>>>>>>>>>>>>>>>>>>>> you take a look? Thanks for the suggestion.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use
>>>>>>>>>> MEMBER-EPOCH
>>>>>>>>>>>>>>>> here,
>>>>>>>>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>>>>>>>>>>>>> confuse what is different between CONSUMER and
>>>>>>>> MEMBER.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Frank,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks for the KIP!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) For consistency, should we do the same for
>>>>>>>>>>>>>>>>>>>>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription,
>>>>> etc. ?
>>>>>>>>>> Even
>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>> we do
>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>>>> implement it right now if the share group
>>>>>>>>> implementation
>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>> still
>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>> incomplete, it may make sense to include it in
>>> the
>>>>>>>>> KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH?
>>>>>>>> That
>>>>>>>>>> would
>>>>>>>>>>>>>>>> seem
>>>>>>>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>>>>>>>>>> consistent.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lucas
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <
>>>>>>>>>>>>>>>> yangpoan@gmail.com>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to kick off the discussion of
>>>>>>>> KIP-1099.
>>>>>>>>>> This
>>>>>>>>>>>>> KIP
>>>>>>>>>>>>>>>>>>>>>> enhances
>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups tools to include state
>>>>> which
>>>>>>>> is
>>>>>>>>>>>>>>>> introduced
>>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP-848.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups command
>>>>> line
>>>>>>>>>> tool
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> support
>>>>>>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer group - Apache Kafka - Apache Software
>>>>>>>>>> Foundation
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cwiki.apache.org
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [image: favicon.ico]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
>>>
>>>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by Andrew Schofield <an...@outlook.com>.
Hi PoAn,
I've taken another pass over the KIP and have some more comments around the share
group support.
AS4: Share groups do not have the concept of a target assignment. As a result,
the verbose state TARGET-ASSIGNMENT-EPOCH is a bit misleading from my
point of view. ASSIGNMENT-EPOCH is better.
AS5: Similarly for the verbose members output, the TARGET-EPOCH and
TARGET-ASSIGNMENT are not really appropriate. Personally, I would prefer
just MEMBER-EPOCH and ASSIGNMENT, and remove the "target" fields.
Thanks,
Andrew
________________________________________
From: PoAn Yang <ya...@gmail.com>
Sent: 24 November 2024 15:08
To: dev@kafka.apache.org <de...@kafka.apache.org>
Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Hi Chia-Ping,
Thanks for the suggestion.
The “isClassic” field in ConsumerGroupDescribeResponse is a boolean value, so we can’t use it as the protocol string.
I change the “PROTOCOL” column to “IS-CLASSIC” in kafka-consumer-groups.sh.
It returns “true” if a member is in a classic group or it’s a classic member in a consumer group.
In other cases, it returns “false”.
I have a draft implementation in https://github.com/apache/kafka/pull/17872.
Thanks,
PoAn
> On Nov 23, 2024, at 1:05 PM, Chia-Ping Tsai <ch...@gmail.com> wrote:
>
> hi PoAn
>
> `isClassic` is good to me :)
>
> Best,
> Chia-Ping
>
> PoAn Yang <ya...@gmail.com> 於 2024年11月23日 週六 下午12:35寫道：
>
>> Hi Chia-Ping / David,
>>
>> Thanks for the great suggestion.
>>
>> How about using “isClassic”?
>> The term like “isLegacy” may be equivocal in the future.
>> For example, if we have a newer version of consumer / share consumer, the
>> meaning will be different for different cases.
>> If we use “isClassic”, for MemberDescription in ClassicGroupDescription,
>> the value can be always “true”.
>> In ConsumerGroupDescription, it’s “true” when it’s a classic member.
>> In ShareGroupDescription, it’s always “false”.
>>
>> Thanks,
>> PoAn
>>
>>> On Nov 23, 2024, at 1:02 AM, Chia-Ping Tsai <ch...@gmail.com> wrote:
>>>
>>> hi DJ
>>>
>>>> My goal was to have a way to tell the user that his (new) consumer group
>>> may have non-upgraded members yet. I wonder if we could use a boolean to
>>> signal this, e.g. isLegacy. Then, in the CLI we could have a column which
>>> is only displayed when they are non upgraded members. What do you think?
>>>
>>> Yes, this is a good approach, but the downside is that the new isLegacy
>>> field in MemberDescription would be irrelevant for classic and shared
>>> groups. This would require thorough documentation to clarify its purpose.
>>>
>>> Additionally, we could use a boolean instead of a string for the new
>> field
>>> in the RPC to reduce the request size.
>>> Best,
>>> Chia-Ping
>>>
>>>
>>>
>>> David Jacot <dj...@confluent.io.invalid> 於 2024年11月23日 週六 上午12:41寫道：
>>>
>>>> Hi both,
>>>>
>>>> Ah, I understand what you mean now. You're saying that we have two
>> protocol
>>>> fields. One at the group level and one at the member level (the new
>> one).
>>>> The two fields are actually two different things. The former is the
>> name of
>>>> the embedded protocol used in the classic protocol whereas the latter is
>>>> the name of the protocol/api used. This is indeed confusing.
>>>>
>>>> My goal was to have a way to tell the user that his (new) consumer group
>>>> may have non-upgraded members yet. I wonder if we could use a boolean to
>>>> signal this, e.g. isLegacy. Then, in the CLI we could have a column
>> which
>>>> is only displayed when they are non upgraded members. What do you think?
>>>>
>>>> If we cannot find a good way to represent this, we could drop it from
>> this
>>>> KIP too.
>>>>
>>>> Best,
>>>> David
>>>>
>>>> On Fri, Nov 22, 2024 at 5:12 PM PoAn Yang <ya...@gmail.com> wrote:
>>>>
>>>>> Hi Chia-Ping / David,
>>>>>
>>>>> Thanks for the review and suggestions.
>>>>>
>>>>>>> I don't fully follow your comment so please excuse me if I say
>>>> something
>>>>>>> wrong.
>>>>>
>>>>> I think Chia-Ping wants to say: originally, the default protocol in
>>>>> classic group is “consumer”.
>>>>> After we have a new consumer group, the protocol of a classic member in
>>>>> consumer group is “classic”.
>>>>> The MemberDescription is used by both ClassicGroupDescription and
>>>>> ConsumerGroupDescription.
>>>>> If we add the protocol field to MemberDescription, the protocol of a
>>>>> classic member in classic group is “consumer”, but it’s “classic” in
>>>>> consumer group.
>>>>> To avoid misleading word, we can add a new collection of member
>> protocols
>>>>> to ConsumerGroupDescription only.
>>>>> The original goal is to show member protocol during consumer migration,
>>>> so
>>>>> we don’t need to add this information to ClassicGroupDescription.
>>>>>
>>>>> Please correct me if I misunderstood anything.
>>>>>
>>>>> Thanks,
>>>>> PoAn
>>>>>
>>>>>> On Nov 22, 2024, at 7:00 PM, Chia-Ping Tsai <ch...@gmail.com>
>>>> wrote:
>>>>>>
>>>>>> hi DJ
>>>>>>
>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and
>>>> change
>>>>>> return type of ModernGroup#protocolType?
>>>>>>
>>>>>> This is not my comment ... sorry for the incorrect quote :(
>>>>>>
>>>>>>> A member in a classic group should report "classic".
>>>>>>
>>>>>> I completely agree. However, in this case,
>>>>> ClassicGroupDescription#protocol
>>>>>> returns 'consumer', while ClassicGroupDescription#members[x].protocol
>>>>>> returns 'classic'. This inconsistency feels a bit strange to me.
>>>>>>
>>>>>> Best,
>>>>>> Chia-Ping
>>>>>>
>>>>>>
>>>>>> David Jacot <dj...@confluent.io.invalid> 於 2024年11月22日 週五 下午6:51寫道：
>>>>>>
>>>>>>> Hi Chia,
>>>>>>>
>>>>>>>>> LM4: I agree it’s better to use a type like GroupProtocol enum, but
>>>> I
>>>>>>> would like to align the format with kafka-groups.sh.
>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and
>>>> change
>>>>>>> return type of ModernGroup#protocolType?
>>>>>>>
>>>>>>> My understanding is that `GroupProtocol` is used to drive the config
>>>> of
>>>>> the
>>>>>>> `Consumer`. Hence adding `Share` seems wrong to me. Or are you
>>>>> referring to
>>>>>>> another GroupProtocol enum? Perhaps GroupType enum?
>>>>>>>
>>>>>>>> In fact, I'm wondering if we should even add 'protocol' to
>>>>>>> MemberDescription,
>>>>>>> since the field has different values between classic and consumer
>>>>> groups—it
>>>>>>> shows 'consumer' in classic groups, whereas it shows 'classic' in
>>>>> consumer
>>>>>>> groups. Yes, it seems like a typo, but it's true. This inconsistency
>>>>> feels
>>>>>>> confusing to me. For example, users call Admin#describeConsumerGroups
>>>>> and
>>>>>>> see that a member has the 'consumer' protocol when it's in a classic
>>>>> group.
>>>>>>> Then, if the group is upgraded to a consumer group, calling
>>>>>>> Admin#describeConsumerGroups again shows the protocol as 'classic'
>> for
>>>>> the
>>>>>>> same member.
>>>>>>>
>>>>>>> I don't fully follow your comment so please excuse me if I say
>>>> something
>>>>>>> wrong.
>>>>>>>
>>>>>>> I think that it should work as follows:
>>>>>>> * A member in a classic group should report "classic".
>>>>>>> * A member using the consumer protocol in a consumer group should
>>>> report
>>>>>>> "consumer".
>>>>>>> * A member using the classic protocol in a consumer group should
>>>> report
>>>>>>> "classic".
>>>>>>> * A member in a share group should report "share".
>>>>>>>
>>>>>>>> If all we want to do is distinguish classic members in the
>>>>>>> ConsumerGroupDescription, then we shouldn't modify the common
>>>>>>> MemberDescription structure. Instead, we could add a collection to
>>>>>>> ConsumerGroupDescription to trace members using the classic protocol,
>>>>> which
>>>>>>> would allow us to enrich the output of command-line tools.
>>>>>>>
>>>>>>> Personally, I believe that using different collections per protocol
>>>>> types
>>>>>>> is not necessary. A field in the member is more than enough.
>>>>>>>
>>>>>>> Best,
>>>>>>> DJ
>>>>>>>
>>>>>>> On Fri, Nov 22, 2024 at 12:01 PM Chia-Ping Tsai <ch...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> hi PoAn
>>>>>>>>
>>>>>>>>> LM4: I agree it’s better to use a type like GroupProtocol enum, but
>>>> I
>>>>>>>> would like to align the format with kafka-groups.sh.
>>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and
>>>>> change
>>>>>>>> return type of ModernGroup#protocolType?
>>>>>>>>
>>>>>>>> MemberDescription is also used by ClassicGroupDescription, and in
>>>>> classic
>>>>>>>> groups, the protocol can be an arbitrary string, so we can't use an
>>>>> enum.
>>>>>>>>
>>>>>>>> In fact, I'm wondering if we should even add 'protocol' to
>>>>>>>> MemberDescription,
>>>>>>>> since the field has different values between classic and consumer
>>>>>>> groups—it
>>>>>>>> shows 'consumer' in classic groups, whereas it shows 'classic' in
>>>>>>> consumer
>>>>>>>> groups. Yes, it seems like a typo, but it's true. This inconsistency
>>>>>>> feels
>>>>>>>> confusing to me. For example, users call
>> Admin#describeConsumerGroups
>>>>> and
>>>>>>>> see that a member has the 'consumer' protocol when it's in a classic
>>>>>>> group.
>>>>>>>> Then, if the group is upgraded to a consumer group, calling
>>>>>>>> Admin#describeConsumerGroups again shows the protocol as 'classic'
>>>> for
>>>>>>> the
>>>>>>>> same member.
>>>>>>>>
>>>>>>>> If all we want to do is distinguish classic members in the
>>>>>>>> ConsumerGroupDescription, then we shouldn't modify the common
>>>>>>>> MemberDescription structure. Instead, we could add a collection to
>>>>>>>> ConsumerGroupDescription to trace members using the classic
>> protocol,
>>>>>>> which
>>>>>>>> would allow us to enrich the output of command-line tools.
>>>>>>>> Best,
>>>>>>>> Chia-Ping
>>>>>>>>
>>>>>>>> PoAn Yang <ya...@gmail.com> 於 2024年11月22日 週五 下午3:46寫道：
>>>>>>>>
>>>>>>>>> Hi Lianet / Jeff,
>>>>>>>>>
>>>>>>>>> Thanks for the review.
>>>>>>>>>
>>>>>>>>> LM5: The kafka-share-groups.sh also uses “—describe” to show
>> offsets
>>>>>>>>> information.
>>>>>>>>> Change to use “—describe —state —verbose” to show group level
>>>>>>> information
>>>>>>>>> in kafka-share-groups.sh.
>>>>>>>>>
>>>>>>>>> LM6: Update the example.
>>>>>>>>>
>>>>>>>>> LM7: Add a description to mention “—verbose” is a new option in
>>>>>>>>> ShareGroupCommandOptions.
>>>>>>>>>
>>>>>>>>> JK2: Yes, add the statement about format change for ASSIGNMENT
>>>> value.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> PoAn
>>>>>>>>>
>>>>>>>>>> On Nov 22, 2024, at 7:35 AM, Jeff Kim <je...@apache.org>
>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi PoAn,
>>>>>>>>>>
>>>>>>>>>> JK2: Can we state in the KIP that we will reformat the existing
>>>>>>>>>> classic group's `--members --verbose` as well, specifically the
>>>>>>>>>> "ASSIGNMENT" column to include the topic name?
>>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>>
>>>>>>>>>> On 2024/11/21 14:33:37 "Lianet M." wrote:
>>>>>>>>>>> Thanks for the updates PoAn!
>>>>>>>>>>>
>>>>>>>>>>> LM6. Just a nit, under the --describe --state --verbose the
>> header
>>>>>>> is
>>>>>>>>> fine
>>>>>>>>>>> but the example is still missing the --state argument.
>>>>>>>>>>>
>>>>>>>>>>> LM7. Under the Proposed changes for kafka-share-groups.sh, should
>>>> we
>>>>>>>>>>> clearly mention that the KIP adds a new --verbose option? I think
>>>>>>> it's
>>>>>>>>>>> confusing because it's presented as if the option already exists
>>>> and
>>>>>>>>> we're
>>>>>>>>>>> only changing the output (which is the case for
>>>>>>> kafka-consumer-groups
>>>>>>>>> but
>>>>>>>>>>> not for kafka-share-groups)
>>>>>>>>>>>
>>>>>>>>>>> That's all on my side. Thanks!
>>>>>>>>>>>
>>>>>>>>>>> Lianet
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Nov 20, 2024 at 11:12 PM PoAn Yang <ya...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Lianet / Jeff,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for review and suggestions.
>>>>>>>>>>>>
>>>>>>>>>>>> LM4: At group level, the protocol field is from
>>>>>>>>> ModernGroup#protocolType.
>>>>>>>>>>>> It’s a string.
>>>>>>>>>>>> The value is defined in different places like
>>>>>>>> ShareGroup#PROTOCOL_TYPE
>>>>>>>>> and
>>>>>>>>>>>> ConsumerProtocol#PROTOCOL_TYPE.
>>>>>>>>>>>> In KIP-1043 <
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=305171038#KIP1043:Administrationofgroups-kafka-groups.sh
>>>>>>>>>> ,
>>>>>>>>>>>> the kafka-groups.sh also shows protocol like “classic” /
>>>>>>> “consumer” /
>>>>>>>>>>>> “share”.
>>>>>>>>>>>> I agree it’s better to use a type like GroupProtocol enum, but I
>>>>>>>> would
>>>>>>>>>>>> like to align the format with kafka-groups.sh.
>>>>>>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol
>> and
>>>>>>>>> change
>>>>>>>>>>>> return type of ModernGroup#protocolType?
>>>>>>>>>>>>
>>>>>>>>>>>> LM5 / JK1: Sorry, I misunderstood LM3, so I deleted —state in
>> the
>>>>>>>>> title.
>>>>>>>>>>>> You’re correct. I would like to use `—describe —state —verbose`
>>>> to
>>>>>>>> show
>>>>>>>>>>>> group level information.
>>>>>>>>>>>> For `—describe —verbose`, it will show output as same as
>>>> `—describe
>>>>>>>>>>>> —offsets —verbose`,
>>>>>>>>>>>> because `—describe` also shows output as same as `—describe
>>>>>>>> —offsets`.
>>>>>>>>>>>>
>>>>>>>>>>>> JK2: Yes, I only use consumer group in the sample output, but
>> the
>>>>>>>>> classic
>>>>>>>>>>>> group will use the same format as well.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> PoAn
>>>>>>>>>>>>
>>>>>>>>>>>>> On Nov 21, 2024, at 8:28 AM, Jeff Kim <je...@apache.org>
>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the KIP. I have some questions:
>>>>>>>>>>>>>
>>>>>>>>>>>>> JK1: (Realized this is similar to LM5) On `--describe
>> --verbose`
>>>>>>>>>>>> proposed changes - doesn't `--describe $group` default to
>>>> printing
>>>>>>>> the
>>>>>>>>>>>> offsets? Perhaps you're referring to the `--state` argument?
>>>> Also,
>>>>>>>>> would
>>>>>>>>>>>> that mean the default `--describe $group --verbose` command
>> would
>>>>>>> not
>>>>>>>>> print
>>>>>>>>>>>> the added field to `--offsets verbose` (leader-epoch) or would
>>>> it?
>>>>>>>>>>>>>
>>>>>>>>>>>>> JK2: Looking at ConsumerGroupCommand.java, the existing
>>>>>>> "ASSIGNMENT"
>>>>>>>>>>>> column under `--members --verbose` does group the topic
>>>> partitions
>>>>>>> by
>>>>>>>>> topic
>>>>>>>>>>>> but does not prefix the grouping with the topic name like you're
>>>>>>>>> proposal:
>>>>>>>>>>>> "my_topic:0,1;new_topic:0,1". Should we do apply the same format
>>>>>>> for
>>>>>>>>> the
>>>>>>>>>>>> classic group as well?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 2024/11/20 21:32:41 "Lianet M." wrote:
>>>>>>>>>>>>>> Hello PoAn, just of couple more minor comments:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> LM4. Regarding the new “protocol” field added to
>>>>>>> MemberDescription.
>>>>>>>>>>>> Should
>>>>>>>>>>>>>> we consider reusing the existing GroupProtocol enum instead of
>>>>>>>>> String?
>>>>>>>>>>>>>> (It’s the one we use from the consumer side to refer to the
>>>>>>>> protocol
>>>>>>>>> in
>>>>>>>>>>>>>> use, just missing Share I notice).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> LM5. Regarding the change to the output for
>>>>>>> kafka-consumer-groups,
>>>>>>>>> the
>>>>>>>>>>>>>> command shown does not include the —state option, but the
>>>> output
>>>>>>>>> shows
>>>>>>>>>>>>>> state info (state, #members, epochs). I would guess that we
>>>> want
>>>>>>> to
>>>>>>>>>>>> modify
>>>>>>>>>>>>>> the output only when we describe a group with the —state
>>>> —verbose
>>>>>>>>>>>> option,
>>>>>>>>>>>>>> is my understanding right? If my understanding is right we’re
>>>>>>> just
>>>>>>>>>>>> missing
>>>>>>>>>>>>>> adding the —state in the example, and the KIP does not
>>>> introduce
>>>>>>>> any
>>>>>>>>>>>>>> changes to the —describe —verbose option. (If not, it would
>>>> mean
>>>>>>> a
>>>>>>>>>>>> bigger
>>>>>>>>>>>>>> change to the output of —describe —verbose which I expect is
>>>> not
>>>>>>>> the
>>>>>>>>>>>>>> intention?)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Lianet
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Nov 20, 2024 at 2:12 AM PoAn Yang <yangpoan@gmail.com
>>>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Chia-Ping,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for the review and suggestions.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> CI0: Thanks for the reminder. Update validVersions in
>>>>>>>>>>>>>>> ConsumerGroupDescribeRequest to 0-1.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> CI1: Yes, I use `ConsumerGroupMember#useClassicProtocol` to
>>>>>>> check
>>>>>>>>>>>> whether
>>>>>>>>>>>>>>> a member in consumer group uses “classic” or “consumer”
>>>>>>> protocol.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> CI2: Yes, a member in share group always uses “share”
>>>> protocol.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> CI3: Add a table to show meaning of “classic”, “consumer”,
>> and
>>>>>>>>> “share”
>>>>>>>>>>>>>>> protocol.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> BTW, the vote thread is in
>>>>>>>>>>>>>>>
>>>>>>> https://lists.apache.org/thread/rb25tf75tzf4c7jqqldlo5jh9w8chsq6.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Nov 20, 2024, at 11:46 AM, Chia-Ping Tsai <
>>>>>>>> chia7712@apache.org>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> hi PoAn
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> CI0: We have to bump the version of
>>>>>>> ConsumerGroupDescribeRequest
>>>>>>>>> as
>>>>>>>>>>>>>>> well, so server can distinguish the new and old behavior.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> CI1: the type of new filed is string, so I guess you plan to
>>>>>>> use
>>>>>>>>>>>>>>> `ConsumerGroupMember#useClassicProtocol` [0] flag to return
>>>>>>> either
>>>>>>>>>>>>>>> "classic" or "consumer", right?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> CI2: `MemberDescription` is used by `ShareGroupDescription`
>>>>>>> too,
>>>>>>>> so
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> new filed protocol in shared group is always "shared", right?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> CI3: Could you consider adding a table to show the value of
>>>> the
>>>>>>>>>>>> protocol
>>>>>>>>>>>>>>> field in each case? Andrew has a beautiful table in KIP-1043
>>>>>>> that
>>>>>>>>>>>> lists all
>>>>>>>>>>>>>>> possible protocol names.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [0]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>> https://github.com/apache/kafka/blob/441a6d0b790f4a17b454caeea7588a6b90fbd9db/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/consumer/ConsumerGroupMember.java#L454
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 2024/11/19 15:45:16 PoAn Yang wrote:
>>>>>>>>>>>>>>>>> Hi David,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for the reminder.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> DJ4: Add related content to the KIP.
>>>>>>>>>>>>>>>>> Bump validVersions in ConsumerGroupDescribeResponse to
>>>> “0-1”.
>>>>>>>>>>>>>>>>> Add a new field “Protocol” to
>> ConsumerGroupDescribeResponse.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Nov 19, 2024, at 3:57 PM, David Jacot
>>>>>>>>>>>> <dj...@confluent.io.INVALID>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> DJ4: Could you please also include the required changes to
>>>>>>> the
>>>>>>>>>>>>>>>>>> ConsumerGroupDescribed API in the public interface
>> section?
>>>>>>> We
>>>>>>>>>>>>>>> basically
>>>>>>>>>>>>>>>>>> need to bump the version, add the new field, etc.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> David
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang <
>>>>>>> yangpoan@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi David,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> DJ4: Add a new field protocol to MemberDescription,
>>>>>>>>>>>>>>>>>>> so the command line tool can show protocol information
>>>> when
>>>>>>>>> users
>>>>>>>>>>>>>>> describe
>>>>>>>>>>>>>>>>>>> members.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> If there is no further suggestion, I will start a vote
>>>>>>> thread
>>>>>>>>>>>> today.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Nov 15, 2024, at 11:50 PM, David Jacot
>>>>>>>>>>>>>>> <dj...@confluent.io.INVALID>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> DJ2: Using "-" sounds good to me.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> DJ3: That seems reasonable to me.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> DJ4: Why not add it right now? I don't want to change
>> the
>>>>>>>>> output
>>>>>>>>>>>> of
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> tool too many times.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>> David
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang <
>>>>>>>> yangpoan@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi David / Andrew,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks for review. Thanks Andrew for picking up
>>>>>>>>>>>>>>> kafka-share-groups.sh
>>>>>>>>>>>>>>>>>>>>> implementation.
>>>>>>>>>>>>>>>>>>>>> I will handle kafka-consumer-groups.sh.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> DJ3: After discussing with @Chia-Ping Tsai, we think
>>>> that
>>>>>>>>> using
>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>> format
>>>>>>>>>>>>>>>>>>>>> is more clear.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> The new format will be like
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>> <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Using colon(:) to concat topic name and partition IDs.
>>>>>>>>>>>>>>>>>>>>> Using comma(,) to concat partition IDs within same
>> topic
>>>>>>>> name.
>>>>>>>>>>>>>>>>>>>>> Using semicolon(;) to concat topic strings.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> AS3: Fix it with kafka-share-groups.sh. Thanks.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> If there is no further suggestion, I will start a vote
>>>>>>>> thread
>>>>>>>>>>>> next
>>>>>>>>>>>>>>>>>>> Monday.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield <
>>>>>>>>>>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>>>>>>>>>> DJ2: I was just going to comment that "-" would be a
>>>> more
>>>>>>>>>>>>>>> appropriate
>>>>>>>>>>>>>>>>>>>>> missing value, but
>>>>>>>>>>>>>>>>>>>>>> you got there first.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> AS3: The examples for kafka-share-groups.sh include
>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups.sh in the
>>>>>>>>>>>>>>>>>>>>>> command line.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> If this is accepted in time, I'm happy to pick up the
>>>>>>>>>>>>>>> implementation of
>>>>>>>>>>>>>>>>>>>>> the share groups
>>>>>>>>>>>>>>>>>>>>>> part of this if it helps.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> Andrew
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>>>>>>>> From: Frank Yang <ya...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>> Sent: 14 November 2024 10:48
>>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
>>>>>>>> kafka-consumer-groups
>>>>>>>>>>>>>>> command
>>>>>>>>>>>>>>>>>>>>> line tool to support new consumer group
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi David,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks for the review and suggestion! I would like to
>>>> get
>>>>>>>>> this
>>>>>>>>>>>> in
>>>>>>>>>>>>>>> AK
>>>>>>>>>>>>>>>>>>> 4.0
>>>>>>>>>>>>>>>>>>>>> as well. I will do my best.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> DJ1: Update KIP to put GROUP-EPOCH and
>>>>>>>>> TARGET-ASSIGNMENT-EPOCH
>>>>>>>>>>>>>>> before
>>>>>>>>>>>>>>>>>>>>> #MEMBERS.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> DJ2: I prefer to follow current missing column value
>>>> “-“.
>>>>>>>>>>>>>>> (reference <
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92
>>>>>>>>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> DJ3: Update KIP to use CURRENT-EPOCH
>> CURRENT-ASSIGNMENT
>>>>>>>>>>>>>>> TARGET-EPOCH
>>>>>>>>>>>>>>>>>>>>> TARGET-ASSIGNMENT.
>>>>>>>>>>>>>>>>>>>>>> Remove GROUP-EPOCH.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> For assignment value, it follows current output
>>>>>>> (reference
>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418
>>>>>>>>>>>>>>>>>>>> ).
>>>>>>>>>>>>>>>>>>>>> I think the form of `topicid-partitionid` is more
>> clear.
>>>>>>>>>>>>>>>>>>>>>> If we would like to use this form, I will update both
>>>>>>>> output
>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups.sh and kafka-share-groups.sh.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> DJ4: It looks like DescribeGroupsResponseData only has
>>>>>>>>> protocol
>>>>>>>>>>>>>>> type at
>>>>>>>>>>>>>>>>>>>>> group level.
>>>>>>>>>>>>>>>>>>>>>> Both DescribeGroupsResponseData and
>>>>>>>>>>>>>>> ConsumerGroupDescribeResponseData
>>>>>>>>>>>>>>>>>>>>> don’t have protocol at member level.
>>>>>>>>>>>>>>>>>>>>>> Could we use a followup to add it?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> DJ5: Update KIP to put LEADER-EPOCH before
>>>>>>> CURRENT-OFFSET.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot
>>>>>>>>>>>>>>> <djacot@confluent.io.INVALID
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP! I have a few minor
>>>>>>>> comments/suggestions:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> DJ1: In the output of `--describe --verbose`, I would
>>>>>>>>> suggest
>>>>>>>>>>>>>>> putting
>>>>>>>>>>>>>>>>>>>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before
>>>>>>>>> `#MEMBERS`.
>>>>>>>>>>>>>>>>>>>>>>> DJ2: Continuing on the above, I assume that we will
>>>>>>> print
>>>>>>>>> out
>>>>>>>>>>>> N/A
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> fields not supported by classic groups. Is this
>>>> correct?
>>>>>>>>>>>>>>>>>>>>>>> DJ3: In the output of `--describe --members
>>>> --verbose`,
>>>>>>> I
>>>>>>>>>>>> wonder
>>>>>>>>>>>>>>> if we
>>>>>>>>>>>>>>>>>>>>>>> should use the following order and terms:
>>>> CURRENT-EPOCH
>>>>>>>>>>>>>>>>>>>>> CURRENT-ASSIGNMENT
>>>>>>>>>>>>>>>>>>>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the
>>>>>>>>> GROUP-EPOCH
>>>>>>>>>>>>>>>>>>> because
>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>> is already in the group description. The value `(0)`
>>>>>>> used
>>>>>>>>> for
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> assignment is incorrect. Here I suppose that we will
>>>>>>> print
>>>>>>>>> out
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> list
>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>> partitions in the form of `topicid-partitionid`.
>>>>>>>>>>>>>>>>>>>>>>> DJ4: Continuing on the above, I wonder if we should
>>>> also
>>>>>>>> add
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> protocol
>>>>>>>>>>>>>>>>>>>>>>> used `classic` or `consumer`. For context, it is
>>>>>>> possible
>>>>>>>> to
>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>> `classic`
>>>>>>>>>>>>>>>>>>>>>>> members and `consumer` members in a `consumer` group
>>>>>>>> during
>>>>>>>>> an
>>>>>>>>>>>>>>> online
>>>>>>>>>>>>>>>>>>>>>>> upgrade from the classic protocol to the consumer
>>>>>>>> protocol.
>>>>>>>>>>>> Having
>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>> information may be handy for administrators. What do
>>>> you
>>>>>>>>> think?
>>>>>>>>>>>>>>>>>>>>>>> DJ5: In the output of `--describe --offsets
>>>> --verbose`,
>>>>>>> I
>>>>>>>>> would
>>>>>>>>>>>>>>>>>>> suggest
>>>>>>>>>>>>>>>>>>>>>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> It would be great if we could get this one in AK 4.0
>>>> as
>>>>>>> it
>>>>>>>>>>>>>>> changes the
>>>>>>>>>>>>>>>>>>>>>>> output of the command.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> DJ
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <
>>>>>>>>> yangpoan@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi Sean / Andrew / Lianet,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks for all your review and suggestions.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> AS1, LM1, LM4: Change to add KIP-848 information
>> when
>>>>>>>> users
>>>>>>>>>>>> give
>>>>>>>>>>>>>>>>>>>>> —verbose
>>>>>>>>>>>>>>>>>>>>>>>> option.
>>>>>>>>>>>>>>>>>>>>>>>> —describe —verbose: shows group epoch and target
>>>>>>>> assignment
>>>>>>>>>>>>>>> epoch.
>>>>>>>>>>>>>>>>>>>>>>>> —describe —members —verbose: shows above
>> information,
>>>>>>>>> member
>>>>>>>>>>>>>>> epoch,
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>> target assignment.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> AS2: Change to use MEMBER-EPOCH to align with
>> KIP-848
>>>>>>>>>>>> definition.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> LM2: For classic group, it doesn’t have epoch, so I
>>>> use
>>>>>>>>>>>> Optional
>>>>>>>>>>>>>>>>>>>>> fields in
>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupDescription.
>>>>>>>>>>>>>>>>>>>>>>>> For share group, it relies on KIP-848. It must have
>>>>>>>> epoch,
>>>>>>>>> so
>>>>>>>>>>>> I
>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>>> int
>>>>>>>>>>>>>>>>>>>>>>>> fields in ShareGroupDescription.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> LM3: Remove —state change. We can get group level
>>>>>>>>> information
>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>> —describe
>>>>>>>>>>>>>>>>>>>>>>>> —verbose.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> SQ1: Add LEADER-EPOCH when users give —describe
>>>>>>> —offsets
>>>>>>>>>>>>>>> —verbose.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <
>>>>>>>> lianetmr@gmail.com
>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Hello Frank, thanks for the KIP! A few comments:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> LM1. I strongly agree with Andrew's suggestion of
>>>>>>> moving
>>>>>>>>> this
>>>>>>>>>>>>>>> into a
>>>>>>>>>>>>>>>>>>>>>>>>> --verbose option. I expect someone would only
>>>> attempt
>>>>>>> to
>>>>>>>>> make
>>>>>>>>>>>>>>> sense
>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> epochs while debugging an issue, not while trying
>> to
>>>>>>>> view
>>>>>>>>> the
>>>>>>>>>>>>>>> group
>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>>>> member's info. (Also member-epoch makes more sense
>>>> to
>>>>>>>> me,
>>>>>>>>>>>> even
>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>>> if we
>>>>>>>>>>>>>>>>>>>>>>>>> end up agreeing in a --verbose). Related to both
>>>>>>> issues,
>>>>>>>>> my
>>>>>>>>>>>>>>> take is
>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>> whoever is close to the protocol will expect
>>>>>>>> member-epoch,
>>>>>>>>>>>>>>> whoever
>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>> will probably won't even need to see the epochs at
>>>>>>> all.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> LM2. Why are the epochs defined as Optional (don't
>>>> we
>>>>>>>>> expect
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> always
>>>>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>>>>> them? with 0 initially, defined on the broker side
>>>> for
>>>>>>>> the
>>>>>>>>>>>> group
>>>>>>>>>>>>>>>>>>> ones,
>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>> client side for the member)
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> LM3. Why is the KIP including the –state option in
>>>> the
>>>>>>>>>>>> proposed
>>>>>>>>>>>>>>>>>>>>> changes?
>>>>>>>>>>>>>>>>>>>>>>>> (
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
>>>>>>>>>>>>>>>>>>>>>>>>> ).
>>>>>>>>>>>>>>>>>>>>>>>>> I get that the output would change in that example,
>>>>>>> but
>>>>>>>>> it’s
>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>> because
>>>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>> any change to the –state option itself. It's
>> because
>>>>>>> of
>>>>>>>>> the
>>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>>>>>>>>>> proposed
>>>>>>>>>>>>>>>>>>>>>>>>> to the –described (required with the --state), and
>>>> the
>>>>>>>>>>>> changes
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> --describe are already explained above (seems
>>>>>>> confusing,
>>>>>>>>> got
>>>>>>>>>>>> me
>>>>>>>>>>>>>>>>>>>>> looking
>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>> a change to the state filter which seemed
>>>> unrelated).
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> LM4. Group epoch and target assignment epoch are
>>>>>>>>> conceptually
>>>>>>>>>>>>>>> at the
>>>>>>>>>>>>>>>>>>>>>>>> group
>>>>>>>>>>>>>>>>>>>>>>>>> level. vs member epoch that lands at a member
>> level.
>>>>>>> So
>>>>>>>>>>>> wonder
>>>>>>>>>>>>>>> if we
>>>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>> show them accordingly (ex. using the --verbose
>>>> option)
>>>>>>>>>>>>>>>>>>>>>>>>> –describe –verbose => shows group epoch and target
>>>>>>>>> assignment
>>>>>>>>>>>>>>> epoch
>>>>>>>>>>>>>>>>>>>>>>>>> –describe –members –verbose => shows member
>>>>>>>> epoch
>>>>>>>>>>>>>>> (along
>>>>>>>>>>>>>>>>>>>>>>>>> with group epoch and target assignment epoch)
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Lianet
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield <
>>>>>>>>>>>>>>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I have a few comments.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> AS1: It seems to me that these new additions are
>>>> most
>>>>>>>>>>>> useful to
>>>>>>>>>>>>>>>>>>>>> people
>>>>>>>>>>>>>>>>>>>>>>>>>> trying to understand
>>>>>>>>>>>>>>>>>>>>>>>>>> the progress of rebalancing in quite some detail.
>>>> The
>>>>>>>>>>>>>>> information
>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>> really not understandable
>>>>>>>>>>>>>>>>>>>>>>>>>> for most users who do not have deep knowledge of
>>>>>>>>>>>> KIP-848/932.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> As a result, I suggest for kafka-share-groups.sh
>>>> that
>>>>>>>> you
>>>>>>>>>>>> add a
>>>>>>>>>>>>>>>>>>>>>>>> --members
>>>>>>>>>>>>>>>>>>>>>>>>>> --verbose option
>>>>>>>>>>>>>>>>>>>>>>>>>> and only include the new information in the output
>>>>>>> for
>>>>>>>>> that
>>>>>>>>>>>>>>> option,
>>>>>>>>>>>>>>>>>>>>>>>> rather
>>>>>>>>>>>>>>>>>>>>>>>>>> than changing the
>>>>>>>>>>>>>>>>>>>>>>>>>> non-verbose --members output.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I also make a similar suggestion for
>>>>>>>>>>>> kafka-consumer-groups.sh
>>>>>>>>>>>>>>>>>>>>> --members
>>>>>>>>>>>>>>>>>>>>>>>>>> and only add the
>>>>>>>>>>>>>>>>>>>>>>>>>> new information for the --verbose output.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> AS2: I strongly suggest that you use MEMBER-EPOCH
>>>>>>>>> instead of
>>>>>>>>>>>>>>>>>>>>>>>>>> CONSUMER-EPOCH.
>>>>>>>>>>>>>>>>>>>>>>>>>> I think it's more confusing not following the
>>>>>>>>> terminology of
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> KIPs.
>>>>>>>>>>>>>>>>>>>>>>>> For
>>>>>>>>>>>>>>>>>>>>>>>>>> one thing,
>>>>>>>>>>>>>>>>>>>>>>>>>> we already have "member" in the admin client such
>>>> as
>>>>>>>>>>>>>>>>>>>>> MemberDescription.
>>>>>>>>>>>>>>>>>>>>>>>>>> It's not a
>>>>>>>>>>>>>>>>>>>>>>>>>> new term.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> Andrew
>>>>>>>>>>>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>>>>>>>>>>>> From: PoAn Yang <pa...@apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>> Sent: 25 October 2024 13:55
>>>>>>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
>>>>>>>>>>>> kafka-consumer-groups
>>>>>>>>>>>>>>>>>>> command
>>>>>>>>>>>>>>>>>>>>>>>> line
>>>>>>>>>>>>>>>>>>>>>>>>>> tool to support new consumer group
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Lucas,
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the review!
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Yes, I add related change for
>>>>>>> kafka-share-groups.sh
>>>>>>>> to
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> KIP.
>>>>>>>>>>>>>>>>>>>>> Could
>>>>>>>>>>>>>>>>>>>>>>>>>> you take a look? Thanks for the suggestion.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use
>>>>>>>>> MEMBER-EPOCH
>>>>>>>>>>>>>>> here,
>>>>>>>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>>>>>>>>>>>> confuse what is different between CONSUMER and
>>>>>>> MEMBER.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Frank,
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks for the KIP!
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) For consistency, should we do the same for
>>>>>>>>>>>>>>>>>>>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription,
>>>> etc. ?
>>>>>>>>> Even
>>>>>>>>>>>> if
>>>>>>>>>>>>>>> we do
>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>>> implement it right now if the share group
>>>>>>>> implementation
>>>>>>>>>>>> may
>>>>>>>>>>>>>>> still
>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>> incomplete, it may make sense to include it in
>> the
>>>>>>>> KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH?
>>>>>>> That
>>>>>>>>> would
>>>>>>>>>>>>>>> seem
>>>>>>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>>>>>>>>> consistent.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>> Lucas
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <
>>>>>>>>>>>>>>> yangpoan@gmail.com>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to kick off the discussion of
>>>>>>> KIP-1099.
>>>>>>>>> This
>>>>>>>>>>>> KIP
>>>>>>>>>>>>>>>>>>>>> enhances
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups tools to include state
>>>> which
>>>>>>> is
>>>>>>>>>>>>>>> introduced
>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>>>> KIP-848.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups command
>>>> line
>>>>>>>>> tool
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> support
>>>>>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer group - Apache Kafka - Apache Software
>>>>>>>>> Foundation
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> cwiki.apache.org
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> [image: favicon.ico]
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>
>>
>>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by PoAn Yang <ya...@gmail.com>.
Hi Chia-Ping,
Thanks for the suggestion.
The “isClassic” field in ConsumerGroupDescribeResponse is a boolean value, so we can’t use it as the protocol string.
I change the “PROTOCOL” column to “IS-CLASSIC” in kafka-consumer-groups.sh.
It returns “true” if a member is in a classic group or it’s a classic member in a consumer group.
In other cases, it returns “false”.
I have a draft implementation in https://github.com/apache/kafka/pull/17872.
Thanks,
PoAn
> On Nov 23, 2024, at 1:05 PM, Chia-Ping Tsai <ch...@gmail.com> wrote:
>
> hi PoAn
>
> `isClassic` is good to me :)
>
> Best,
> Chia-Ping
>
> PoAn Yang <ya...@gmail.com> 於 2024年11月23日 週六 下午12:35寫道：
>
>> Hi Chia-Ping / David,
>>
>> Thanks for the great suggestion.
>>
>> How about using “isClassic”?
>> The term like “isLegacy” may be equivocal in the future.
>> For example, if we have a newer version of consumer / share consumer, the
>> meaning will be different for different cases.
>> If we use “isClassic”, for MemberDescription in ClassicGroupDescription,
>> the value can be always “true”.
>> In ConsumerGroupDescription, it’s “true” when it’s a classic member.
>> In ShareGroupDescription, it’s always “false”.
>>
>> Thanks,
>> PoAn
>>
>>> On Nov 23, 2024, at 1:02 AM, Chia-Ping Tsai <ch...@gmail.com> wrote:
>>>
>>> hi DJ
>>>
>>>> My goal was to have a way to tell the user that his (new) consumer group
>>> may have non-upgraded members yet. I wonder if we could use a boolean to
>>> signal this, e.g. isLegacy. Then, in the CLI we could have a column which
>>> is only displayed when they are non upgraded members. What do you think?
>>>
>>> Yes, this is a good approach, but the downside is that the new isLegacy
>>> field in MemberDescription would be irrelevant for classic and shared
>>> groups. This would require thorough documentation to clarify its purpose.
>>>
>>> Additionally, we could use a boolean instead of a string for the new
>> field
>>> in the RPC to reduce the request size.
>>> Best,
>>> Chia-Ping
>>>
>>>
>>>
>>> David Jacot <dj...@confluent.io.invalid> 於 2024年11月23日 週六 上午12:41寫道：
>>>
>>>> Hi both,
>>>>
>>>> Ah, I understand what you mean now. You're saying that we have two
>> protocol
>>>> fields. One at the group level and one at the member level (the new
>> one).
>>>> The two fields are actually two different things. The former is the
>> name of
>>>> the embedded protocol used in the classic protocol whereas the latter is
>>>> the name of the protocol/api used. This is indeed confusing.
>>>>
>>>> My goal was to have a way to tell the user that his (new) consumer group
>>>> may have non-upgraded members yet. I wonder if we could use a boolean to
>>>> signal this, e.g. isLegacy. Then, in the CLI we could have a column
>> which
>>>> is only displayed when they are non upgraded members. What do you think?
>>>>
>>>> If we cannot find a good way to represent this, we could drop it from
>> this
>>>> KIP too.
>>>>
>>>> Best,
>>>> David
>>>>
>>>> On Fri, Nov 22, 2024 at 5:12 PM PoAn Yang <ya...@gmail.com> wrote:
>>>>
>>>>> Hi Chia-Ping / David,
>>>>>
>>>>> Thanks for the review and suggestions.
>>>>>
>>>>>>> I don't fully follow your comment so please excuse me if I say
>>>> something
>>>>>>> wrong.
>>>>>
>>>>> I think Chia-Ping wants to say: originally, the default protocol in
>>>>> classic group is “consumer”.
>>>>> After we have a new consumer group, the protocol of a classic member in
>>>>> consumer group is “classic”.
>>>>> The MemberDescription is used by both ClassicGroupDescription and
>>>>> ConsumerGroupDescription.
>>>>> If we add the protocol field to MemberDescription, the protocol of a
>>>>> classic member in classic group is “consumer”, but it’s “classic” in
>>>>> consumer group.
>>>>> To avoid misleading word, we can add a new collection of member
>> protocols
>>>>> to ConsumerGroupDescription only.
>>>>> The original goal is to show member protocol during consumer migration,
>>>> so
>>>>> we don’t need to add this information to ClassicGroupDescription.
>>>>>
>>>>> Please correct me if I misunderstood anything.
>>>>>
>>>>> Thanks,
>>>>> PoAn
>>>>>
>>>>>> On Nov 22, 2024, at 7:00 PM, Chia-Ping Tsai <ch...@gmail.com>
>>>> wrote:
>>>>>>
>>>>>> hi DJ
>>>>>>
>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and
>>>> change
>>>>>> return type of ModernGroup#protocolType?
>>>>>>
>>>>>> This is not my comment ... sorry for the incorrect quote :(
>>>>>>
>>>>>>> A member in a classic group should report "classic".
>>>>>>
>>>>>> I completely agree. However, in this case,
>>>>> ClassicGroupDescription#protocol
>>>>>> returns 'consumer', while ClassicGroupDescription#members[x].protocol
>>>>>> returns 'classic'. This inconsistency feels a bit strange to me.
>>>>>>
>>>>>> Best,
>>>>>> Chia-Ping
>>>>>>
>>>>>>
>>>>>> David Jacot <dj...@confluent.io.invalid> 於 2024年11月22日 週五 下午6:51寫道：
>>>>>>
>>>>>>> Hi Chia,
>>>>>>>
>>>>>>>>> LM4: I agree it’s better to use a type like GroupProtocol enum, but
>>>> I
>>>>>>> would like to align the format with kafka-groups.sh.
>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and
>>>> change
>>>>>>> return type of ModernGroup#protocolType?
>>>>>>>
>>>>>>> My understanding is that `GroupProtocol` is used to drive the config
>>>> of
>>>>> the
>>>>>>> `Consumer`. Hence adding `Share` seems wrong to me. Or are you
>>>>> referring to
>>>>>>> another GroupProtocol enum? Perhaps GroupType enum?
>>>>>>>
>>>>>>>> In fact, I'm wondering if we should even add 'protocol' to
>>>>>>> MemberDescription,
>>>>>>> since the field has different values between classic and consumer
>>>>> groups—it
>>>>>>> shows 'consumer' in classic groups, whereas it shows 'classic' in
>>>>> consumer
>>>>>>> groups. Yes, it seems like a typo, but it's true. This inconsistency
>>>>> feels
>>>>>>> confusing to me. For example, users call Admin#describeConsumerGroups
>>>>> and
>>>>>>> see that a member has the 'consumer' protocol when it's in a classic
>>>>> group.
>>>>>>> Then, if the group is upgraded to a consumer group, calling
>>>>>>> Admin#describeConsumerGroups again shows the protocol as 'classic'
>> for
>>>>> the
>>>>>>> same member.
>>>>>>>
>>>>>>> I don't fully follow your comment so please excuse me if I say
>>>> something
>>>>>>> wrong.
>>>>>>>
>>>>>>> I think that it should work as follows:
>>>>>>> * A member in a classic group should report "classic".
>>>>>>> * A member using the consumer protocol in a consumer group should
>>>> report
>>>>>>> "consumer".
>>>>>>> * A member using the classic protocol in a consumer group should
>>>> report
>>>>>>> "classic".
>>>>>>> * A member in a share group should report "share".
>>>>>>>
>>>>>>>> If all we want to do is distinguish classic members in the
>>>>>>> ConsumerGroupDescription, then we shouldn't modify the common
>>>>>>> MemberDescription structure. Instead, we could add a collection to
>>>>>>> ConsumerGroupDescription to trace members using the classic protocol,
>>>>> which
>>>>>>> would allow us to enrich the output of command-line tools.
>>>>>>>
>>>>>>> Personally, I believe that using different collections per protocol
>>>>> types
>>>>>>> is not necessary. A field in the member is more than enough.
>>>>>>>
>>>>>>> Best,
>>>>>>> DJ
>>>>>>>
>>>>>>> On Fri, Nov 22, 2024 at 12:01 PM Chia-Ping Tsai <ch...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> hi PoAn
>>>>>>>>
>>>>>>>>> LM4: I agree it’s better to use a type like GroupProtocol enum, but
>>>> I
>>>>>>>> would like to align the format with kafka-groups.sh.
>>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and
>>>>> change
>>>>>>>> return type of ModernGroup#protocolType?
>>>>>>>>
>>>>>>>> MemberDescription is also used by ClassicGroupDescription, and in
>>>>> classic
>>>>>>>> groups, the protocol can be an arbitrary string, so we can't use an
>>>>> enum.
>>>>>>>>
>>>>>>>> In fact, I'm wondering if we should even add 'protocol' to
>>>>>>>> MemberDescription,
>>>>>>>> since the field has different values between classic and consumer
>>>>>>> groups—it
>>>>>>>> shows 'consumer' in classic groups, whereas it shows 'classic' in
>>>>>>> consumer
>>>>>>>> groups. Yes, it seems like a typo, but it's true. This inconsistency
>>>>>>> feels
>>>>>>>> confusing to me. For example, users call
>> Admin#describeConsumerGroups
>>>>> and
>>>>>>>> see that a member has the 'consumer' protocol when it's in a classic
>>>>>>> group.
>>>>>>>> Then, if the group is upgraded to a consumer group, calling
>>>>>>>> Admin#describeConsumerGroups again shows the protocol as 'classic'
>>>> for
>>>>>>> the
>>>>>>>> same member.
>>>>>>>>
>>>>>>>> If all we want to do is distinguish classic members in the
>>>>>>>> ConsumerGroupDescription, then we shouldn't modify the common
>>>>>>>> MemberDescription structure. Instead, we could add a collection to
>>>>>>>> ConsumerGroupDescription to trace members using the classic
>> protocol,
>>>>>>> which
>>>>>>>> would allow us to enrich the output of command-line tools.
>>>>>>>> Best,
>>>>>>>> Chia-Ping
>>>>>>>>
>>>>>>>> PoAn Yang <ya...@gmail.com> 於 2024年11月22日 週五 下午3:46寫道：
>>>>>>>>
>>>>>>>>> Hi Lianet / Jeff,
>>>>>>>>>
>>>>>>>>> Thanks for the review.
>>>>>>>>>
>>>>>>>>> LM5: The kafka-share-groups.sh also uses “—describe” to show
>> offsets
>>>>>>>>> information.
>>>>>>>>> Change to use “—describe —state —verbose” to show group level
>>>>>>> information
>>>>>>>>> in kafka-share-groups.sh.
>>>>>>>>>
>>>>>>>>> LM6: Update the example.
>>>>>>>>>
>>>>>>>>> LM7: Add a description to mention “—verbose” is a new option in
>>>>>>>>> ShareGroupCommandOptions.
>>>>>>>>>
>>>>>>>>> JK2: Yes, add the statement about format change for ASSIGNMENT
>>>> value.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> PoAn
>>>>>>>>>
>>>>>>>>>> On Nov 22, 2024, at 7:35 AM, Jeff Kim <je...@apache.org>
>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi PoAn,
>>>>>>>>>>
>>>>>>>>>> JK2: Can we state in the KIP that we will reformat the existing
>>>>>>>>>> classic group's `--members --verbose` as well, specifically the
>>>>>>>>>> "ASSIGNMENT" column to include the topic name?
>>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>>
>>>>>>>>>> On 2024/11/21 14:33:37 "Lianet M." wrote:
>>>>>>>>>>> Thanks for the updates PoAn!
>>>>>>>>>>>
>>>>>>>>>>> LM6. Just a nit, under the --describe --state --verbose the
>> header
>>>>>>> is
>>>>>>>>> fine
>>>>>>>>>>> but the example is still missing the --state argument.
>>>>>>>>>>>
>>>>>>>>>>> LM7. Under the Proposed changes for kafka-share-groups.sh, should
>>>> we
>>>>>>>>>>> clearly mention that the KIP adds a new --verbose option? I think
>>>>>>> it's
>>>>>>>>>>> confusing because it's presented as if the option already exists
>>>> and
>>>>>>>>> we're
>>>>>>>>>>> only changing the output (which is the case for
>>>>>>> kafka-consumer-groups
>>>>>>>>> but
>>>>>>>>>>> not for kafka-share-groups)
>>>>>>>>>>>
>>>>>>>>>>> That's all on my side. Thanks!
>>>>>>>>>>>
>>>>>>>>>>> Lianet
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Nov 20, 2024 at 11:12 PM PoAn Yang <ya...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Lianet / Jeff,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for review and suggestions.
>>>>>>>>>>>>
>>>>>>>>>>>> LM4: At group level, the protocol field is from
>>>>>>>>> ModernGroup#protocolType.
>>>>>>>>>>>> It’s a string.
>>>>>>>>>>>> The value is defined in different places like
>>>>>>>> ShareGroup#PROTOCOL_TYPE
>>>>>>>>> and
>>>>>>>>>>>> ConsumerProtocol#PROTOCOL_TYPE.
>>>>>>>>>>>> In KIP-1043 <
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=305171038#KIP1043:Administrationofgroups-kafka-groups.sh
>>>>>>>>>> ,
>>>>>>>>>>>> the kafka-groups.sh also shows protocol like “classic” /
>>>>>>> “consumer” /
>>>>>>>>>>>> “share”.
>>>>>>>>>>>> I agree it’s better to use a type like GroupProtocol enum, but I
>>>>>>>> would
>>>>>>>>>>>> like to align the format with kafka-groups.sh.
>>>>>>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol
>> and
>>>>>>>>> change
>>>>>>>>>>>> return type of ModernGroup#protocolType?
>>>>>>>>>>>>
>>>>>>>>>>>> LM5 / JK1: Sorry, I misunderstood LM3, so I deleted —state in
>> the
>>>>>>>>> title.
>>>>>>>>>>>> You’re correct. I would like to use `—describe —state —verbose`
>>>> to
>>>>>>>> show
>>>>>>>>>>>> group level information.
>>>>>>>>>>>> For `—describe —verbose`, it will show output as same as
>>>> `—describe
>>>>>>>>>>>> —offsets —verbose`,
>>>>>>>>>>>> because `—describe` also shows output as same as `—describe
>>>>>>>> —offsets`.
>>>>>>>>>>>>
>>>>>>>>>>>> JK2: Yes, I only use consumer group in the sample output, but
>> the
>>>>>>>>> classic
>>>>>>>>>>>> group will use the same format as well.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> PoAn
>>>>>>>>>>>>
>>>>>>>>>>>>> On Nov 21, 2024, at 8:28 AM, Jeff Kim <je...@apache.org>
>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the KIP. I have some questions:
>>>>>>>>>>>>>
>>>>>>>>>>>>> JK1: (Realized this is similar to LM5) On `--describe
>> --verbose`
>>>>>>>>>>>> proposed changes - doesn't `--describe $group` default to
>>>> printing
>>>>>>>> the
>>>>>>>>>>>> offsets? Perhaps you're referring to the `--state` argument?
>>>> Also,
>>>>>>>>> would
>>>>>>>>>>>> that mean the default `--describe $group --verbose` command
>> would
>>>>>>> not
>>>>>>>>> print
>>>>>>>>>>>> the added field to `--offsets verbose` (leader-epoch) or would
>>>> it?
>>>>>>>>>>>>>
>>>>>>>>>>>>> JK2: Looking at ConsumerGroupCommand.java, the existing
>>>>>>> "ASSIGNMENT"
>>>>>>>>>>>> column under `--members --verbose` does group the topic
>>>> partitions
>>>>>>> by
>>>>>>>>> topic
>>>>>>>>>>>> but does not prefix the grouping with the topic name like you're
>>>>>>>>> proposal:
>>>>>>>>>>>> "my_topic:0,1;new_topic:0,1". Should we do apply the same format
>>>>>>> for
>>>>>>>>> the
>>>>>>>>>>>> classic group as well?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 2024/11/20 21:32:41 "Lianet M." wrote:
>>>>>>>>>>>>>> Hello PoAn, just of couple more minor comments:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> LM4. Regarding the new “protocol” field added to
>>>>>>> MemberDescription.
>>>>>>>>>>>> Should
>>>>>>>>>>>>>> we consider reusing the existing GroupProtocol enum instead of
>>>>>>>>> String?
>>>>>>>>>>>>>> (It’s the one we use from the consumer side to refer to the
>>>>>>>> protocol
>>>>>>>>> in
>>>>>>>>>>>>>> use, just missing Share I notice).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> LM5. Regarding the change to the output for
>>>>>>> kafka-consumer-groups,
>>>>>>>>> the
>>>>>>>>>>>>>> command shown does not include the —state option, but the
>>>> output
>>>>>>>>> shows
>>>>>>>>>>>>>> state info (state, #members, epochs). I would guess that we
>>>> want
>>>>>>> to
>>>>>>>>>>>> modify
>>>>>>>>>>>>>> the output only when we describe a group with the —state
>>>> —verbose
>>>>>>>>>>>> option,
>>>>>>>>>>>>>> is my understanding right? If my understanding is right we’re
>>>>>>> just
>>>>>>>>>>>> missing
>>>>>>>>>>>>>> adding the —state in the example, and the KIP does not
>>>> introduce
>>>>>>>> any
>>>>>>>>>>>>>> changes to the —describe —verbose option. (If not, it would
>>>> mean
>>>>>>> a
>>>>>>>>>>>> bigger
>>>>>>>>>>>>>> change to the output of —describe —verbose which I expect is
>>>> not
>>>>>>>> the
>>>>>>>>>>>>>> intention?)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Lianet
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Nov 20, 2024 at 2:12 AM PoAn Yang <yangpoan@gmail.com
>>>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Chia-Ping,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for the review and suggestions.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> CI0: Thanks for the reminder. Update validVersions in
>>>>>>>>>>>>>>> ConsumerGroupDescribeRequest to 0-1.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> CI1: Yes, I use `ConsumerGroupMember#useClassicProtocol` to
>>>>>>> check
>>>>>>>>>>>> whether
>>>>>>>>>>>>>>> a member in consumer group uses “classic” or “consumer”
>>>>>>> protocol.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> CI2: Yes, a member in share group always uses “share”
>>>> protocol.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> CI3: Add a table to show meaning of “classic”, “consumer”,
>> and
>>>>>>>>> “share”
>>>>>>>>>>>>>>> protocol.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> BTW, the vote thread is in
>>>>>>>>>>>>>>>
>>>>>>> https://lists.apache.org/thread/rb25tf75tzf4c7jqqldlo5jh9w8chsq6.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Nov 20, 2024, at 11:46 AM, Chia-Ping Tsai <
>>>>>>>> chia7712@apache.org>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> hi PoAn
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> CI0: We have to bump the version of
>>>>>>> ConsumerGroupDescribeRequest
>>>>>>>>> as
>>>>>>>>>>>>>>> well, so server can distinguish the new and old behavior.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> CI1: the type of new filed is string, so I guess you plan to
>>>>>>> use
>>>>>>>>>>>>>>> `ConsumerGroupMember#useClassicProtocol` [0] flag to return
>>>>>>> either
>>>>>>>>>>>>>>> "classic" or "consumer", right?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> CI2: `MemberDescription` is used by `ShareGroupDescription`
>>>>>>> too,
>>>>>>>> so
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> new filed protocol in shared group is always "shared", right?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> CI3: Could you consider adding a table to show the value of
>>>> the
>>>>>>>>>>>> protocol
>>>>>>>>>>>>>>> field in each case? Andrew has a beautiful table in KIP-1043
>>>>>>> that
>>>>>>>>>>>> lists all
>>>>>>>>>>>>>>> possible protocol names.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [0]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>> https://github.com/apache/kafka/blob/441a6d0b790f4a17b454caeea7588a6b90fbd9db/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/consumer/ConsumerGroupMember.java#L454
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 2024/11/19 15:45:16 PoAn Yang wrote:
>>>>>>>>>>>>>>>>> Hi David,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for the reminder.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> DJ4: Add related content to the KIP.
>>>>>>>>>>>>>>>>> Bump validVersions in ConsumerGroupDescribeResponse to
>>>> “0-1”.
>>>>>>>>>>>>>>>>> Add a new field “Protocol” to
>> ConsumerGroupDescribeResponse.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Nov 19, 2024, at 3:57 PM, David Jacot
>>>>>>>>>>>> <dj...@confluent.io.INVALID>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> DJ4: Could you please also include the required changes to
>>>>>>> the
>>>>>>>>>>>>>>>>>> ConsumerGroupDescribed API in the public interface
>> section?
>>>>>>> We
>>>>>>>>>>>>>>> basically
>>>>>>>>>>>>>>>>>> need to bump the version, add the new field, etc.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> David
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang <
>>>>>>> yangpoan@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi David,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> DJ4: Add a new field protocol to MemberDescription,
>>>>>>>>>>>>>>>>>>> so the command line tool can show protocol information
>>>> when
>>>>>>>>> users
>>>>>>>>>>>>>>> describe
>>>>>>>>>>>>>>>>>>> members.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> If there is no further suggestion, I will start a vote
>>>>>>> thread
>>>>>>>>>>>> today.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Nov 15, 2024, at 11:50 PM, David Jacot
>>>>>>>>>>>>>>> <dj...@confluent.io.INVALID>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> DJ2: Using "-" sounds good to me.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> DJ3: That seems reasonable to me.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> DJ4: Why not add it right now? I don't want to change
>> the
>>>>>>>>> output
>>>>>>>>>>>> of
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> tool too many times.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>> David
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang <
>>>>>>>> yangpoan@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi David / Andrew,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks for review. Thanks Andrew for picking up
>>>>>>>>>>>>>>> kafka-share-groups.sh
>>>>>>>>>>>>>>>>>>>>> implementation.
>>>>>>>>>>>>>>>>>>>>> I will handle kafka-consumer-groups.sh.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> DJ3: After discussing with @Chia-Ping Tsai, we think
>>>> that
>>>>>>>>> using
>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>> format
>>>>>>>>>>>>>>>>>>>>> is more clear.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> The new format will be like
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>> <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Using colon(:) to concat topic name and partition IDs.
>>>>>>>>>>>>>>>>>>>>> Using comma(,) to concat partition IDs within same
>> topic
>>>>>>>> name.
>>>>>>>>>>>>>>>>>>>>> Using semicolon(;) to concat topic strings.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> AS3: Fix it with kafka-share-groups.sh. Thanks.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> If there is no further suggestion, I will start a vote
>>>>>>>> thread
>>>>>>>>>>>> next
>>>>>>>>>>>>>>>>>>> Monday.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield <
>>>>>>>>>>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>>>>>>>>>> DJ2: I was just going to comment that "-" would be a
>>>> more
>>>>>>>>>>>>>>> appropriate
>>>>>>>>>>>>>>>>>>>>> missing value, but
>>>>>>>>>>>>>>>>>>>>>> you got there first.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> AS3: The examples for kafka-share-groups.sh include
>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups.sh in the
>>>>>>>>>>>>>>>>>>>>>> command line.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> If this is accepted in time, I'm happy to pick up the
>>>>>>>>>>>>>>> implementation of
>>>>>>>>>>>>>>>>>>>>> the share groups
>>>>>>>>>>>>>>>>>>>>>> part of this if it helps.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> Andrew
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>>>>>>>> From: Frank Yang <ya...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>> Sent: 14 November 2024 10:48
>>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
>>>>>>>> kafka-consumer-groups
>>>>>>>>>>>>>>> command
>>>>>>>>>>>>>>>>>>>>> line tool to support new consumer group
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi David,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks for the review and suggestion! I would like to
>>>> get
>>>>>>>>> this
>>>>>>>>>>>> in
>>>>>>>>>>>>>>> AK
>>>>>>>>>>>>>>>>>>> 4.0
>>>>>>>>>>>>>>>>>>>>> as well. I will do my best.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> DJ1: Update KIP to put GROUP-EPOCH and
>>>>>>>>> TARGET-ASSIGNMENT-EPOCH
>>>>>>>>>>>>>>> before
>>>>>>>>>>>>>>>>>>>>> #MEMBERS.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> DJ2: I prefer to follow current missing column value
>>>> “-“.
>>>>>>>>>>>>>>> (reference <
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92
>>>>>>>>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> DJ3: Update KIP to use CURRENT-EPOCH
>> CURRENT-ASSIGNMENT
>>>>>>>>>>>>>>> TARGET-EPOCH
>>>>>>>>>>>>>>>>>>>>> TARGET-ASSIGNMENT.
>>>>>>>>>>>>>>>>>>>>>> Remove GROUP-EPOCH.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> For assignment value, it follows current output
>>>>>>> (reference
>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418
>>>>>>>>>>>>>>>>>>>> ).
>>>>>>>>>>>>>>>>>>>>> I think the form of `topicid-partitionid` is more
>> clear.
>>>>>>>>>>>>>>>>>>>>>> If we would like to use this form, I will update both
>>>>>>>> output
>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups.sh and kafka-share-groups.sh.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> DJ4: It looks like DescribeGroupsResponseData only has
>>>>>>>>> protocol
>>>>>>>>>>>>>>> type at
>>>>>>>>>>>>>>>>>>>>> group level.
>>>>>>>>>>>>>>>>>>>>>> Both DescribeGroupsResponseData and
>>>>>>>>>>>>>>> ConsumerGroupDescribeResponseData
>>>>>>>>>>>>>>>>>>>>> don’t have protocol at member level.
>>>>>>>>>>>>>>>>>>>>>> Could we use a followup to add it?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> DJ5: Update KIP to put LEADER-EPOCH before
>>>>>>> CURRENT-OFFSET.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot
>>>>>>>>>>>>>>> <djacot@confluent.io.INVALID
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP! I have a few minor
>>>>>>>> comments/suggestions:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> DJ1: In the output of `--describe --verbose`, I would
>>>>>>>>> suggest
>>>>>>>>>>>>>>> putting
>>>>>>>>>>>>>>>>>>>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before
>>>>>>>>> `#MEMBERS`.
>>>>>>>>>>>>>>>>>>>>>>> DJ2: Continuing on the above, I assume that we will
>>>>>>> print
>>>>>>>>> out
>>>>>>>>>>>> N/A
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> fields not supported by classic groups. Is this
>>>> correct?
>>>>>>>>>>>>>>>>>>>>>>> DJ3: In the output of `--describe --members
>>>> --verbose`,
>>>>>>> I
>>>>>>>>>>>> wonder
>>>>>>>>>>>>>>> if we
>>>>>>>>>>>>>>>>>>>>>>> should use the following order and terms:
>>>> CURRENT-EPOCH
>>>>>>>>>>>>>>>>>>>>> CURRENT-ASSIGNMENT
>>>>>>>>>>>>>>>>>>>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the
>>>>>>>>> GROUP-EPOCH
>>>>>>>>>>>>>>>>>>> because
>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>> is already in the group description. The value `(0)`
>>>>>>> used
>>>>>>>>> for
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> assignment is incorrect. Here I suppose that we will
>>>>>>> print
>>>>>>>>> out
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> list
>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>> partitions in the form of `topicid-partitionid`.
>>>>>>>>>>>>>>>>>>>>>>> DJ4: Continuing on the above, I wonder if we should
>>>> also
>>>>>>>> add
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> protocol
>>>>>>>>>>>>>>>>>>>>>>> used `classic` or `consumer`. For context, it is
>>>>>>> possible
>>>>>>>> to
>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>> `classic`
>>>>>>>>>>>>>>>>>>>>>>> members and `consumer` members in a `consumer` group
>>>>>>>> during
>>>>>>>>> an
>>>>>>>>>>>>>>> online
>>>>>>>>>>>>>>>>>>>>>>> upgrade from the classic protocol to the consumer
>>>>>>>> protocol.
>>>>>>>>>>>> Having
>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>> information may be handy for administrators. What do
>>>> you
>>>>>>>>> think?
>>>>>>>>>>>>>>>>>>>>>>> DJ5: In the output of `--describe --offsets
>>>> --verbose`,
>>>>>>> I
>>>>>>>>> would
>>>>>>>>>>>>>>>>>>> suggest
>>>>>>>>>>>>>>>>>>>>>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> It would be great if we could get this one in AK 4.0
>>>> as
>>>>>>> it
>>>>>>>>>>>>>>> changes the
>>>>>>>>>>>>>>>>>>>>>>> output of the command.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> DJ
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <
>>>>>>>>> yangpoan@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi Sean / Andrew / Lianet,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks for all your review and suggestions.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> AS1, LM1, LM4: Change to add KIP-848 information
>> when
>>>>>>>> users
>>>>>>>>>>>> give
>>>>>>>>>>>>>>>>>>>>> —verbose
>>>>>>>>>>>>>>>>>>>>>>>> option.
>>>>>>>>>>>>>>>>>>>>>>>> —describe —verbose: shows group epoch and target
>>>>>>>> assignment
>>>>>>>>>>>>>>> epoch.
>>>>>>>>>>>>>>>>>>>>>>>> —describe —members —verbose: shows above
>> information,
>>>>>>>>> member
>>>>>>>>>>>>>>> epoch,
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>> target assignment.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> AS2: Change to use MEMBER-EPOCH to align with
>> KIP-848
>>>>>>>>>>>> definition.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> LM2: For classic group, it doesn’t have epoch, so I
>>>> use
>>>>>>>>>>>> Optional
>>>>>>>>>>>>>>>>>>>>> fields in
>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupDescription.
>>>>>>>>>>>>>>>>>>>>>>>> For share group, it relies on KIP-848. It must have
>>>>>>>> epoch,
>>>>>>>>> so
>>>>>>>>>>>> I
>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>>> int
>>>>>>>>>>>>>>>>>>>>>>>> fields in ShareGroupDescription.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> LM3: Remove —state change. We can get group level
>>>>>>>>> information
>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>> —describe
>>>>>>>>>>>>>>>>>>>>>>>> —verbose.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> SQ1: Add LEADER-EPOCH when users give —describe
>>>>>>> —offsets
>>>>>>>>>>>>>>> —verbose.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <
>>>>>>>> lianetmr@gmail.com
>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Hello Frank, thanks for the KIP! A few comments:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> LM1. I strongly agree with Andrew's suggestion of
>>>>>>> moving
>>>>>>>>> this
>>>>>>>>>>>>>>> into a
>>>>>>>>>>>>>>>>>>>>>>>>> --verbose option. I expect someone would only
>>>> attempt
>>>>>>> to
>>>>>>>>> make
>>>>>>>>>>>>>>> sense
>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> epochs while debugging an issue, not while trying
>> to
>>>>>>>> view
>>>>>>>>> the
>>>>>>>>>>>>>>> group
>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>>>> member's info. (Also member-epoch makes more sense
>>>> to
>>>>>>>> me,
>>>>>>>>>>>> even
>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>>> if we
>>>>>>>>>>>>>>>>>>>>>>>>> end up agreeing in a --verbose). Related to both
>>>>>>> issues,
>>>>>>>>> my
>>>>>>>>>>>>>>> take is
>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>> whoever is close to the protocol will expect
>>>>>>>> member-epoch,
>>>>>>>>>>>>>>> whoever
>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>> will probably won't even need to see the epochs at
>>>>>>> all.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> LM2. Why are the epochs defined as Optional (don't
>>>> we
>>>>>>>>> expect
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> always
>>>>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>>>>> them? with 0 initially, defined on the broker side
>>>> for
>>>>>>>> the
>>>>>>>>>>>> group
>>>>>>>>>>>>>>>>>>> ones,
>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>> client side for the member)
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> LM3. Why is the KIP including the –state option in
>>>> the
>>>>>>>>>>>> proposed
>>>>>>>>>>>>>>>>>>>>> changes?
>>>>>>>>>>>>>>>>>>>>>>>> (
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
>>>>>>>>>>>>>>>>>>>>>>>>> ).
>>>>>>>>>>>>>>>>>>>>>>>>> I get that the output would change in that example,
>>>>>>> but
>>>>>>>>> it’s
>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>> because
>>>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>> any change to the –state option itself. It's
>> because
>>>>>>> of
>>>>>>>>> the
>>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>>>>>>>>>> proposed
>>>>>>>>>>>>>>>>>>>>>>>>> to the –described (required with the --state), and
>>>> the
>>>>>>>>>>>> changes
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> --describe are already explained above (seems
>>>>>>> confusing,
>>>>>>>>> got
>>>>>>>>>>>> me
>>>>>>>>>>>>>>>>>>>>> looking
>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>> a change to the state filter which seemed
>>>> unrelated).
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> LM4. Group epoch and target assignment epoch are
>>>>>>>>> conceptually
>>>>>>>>>>>>>>> at the
>>>>>>>>>>>>>>>>>>>>>>>> group
>>>>>>>>>>>>>>>>>>>>>>>>> level. vs member epoch that lands at a member
>> level.
>>>>>>> So
>>>>>>>>>>>> wonder
>>>>>>>>>>>>>>> if we
>>>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>> show them accordingly (ex. using the --verbose
>>>> option)
>>>>>>>>>>>>>>>>>>>>>>>>> –describe –verbose => shows group epoch and target
>>>>>>>>> assignment
>>>>>>>>>>>>>>> epoch
>>>>>>>>>>>>>>>>>>>>>>>>> –describe –members –verbose => shows member
>>>>>>>> epoch
>>>>>>>>>>>>>>> (along
>>>>>>>>>>>>>>>>>>>>>>>>> with group epoch and target assignment epoch)
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Lianet
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield <
>>>>>>>>>>>>>>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I have a few comments.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> AS1: It seems to me that these new additions are
>>>> most
>>>>>>>>>>>> useful to
>>>>>>>>>>>>>>>>>>>>> people
>>>>>>>>>>>>>>>>>>>>>>>>>> trying to understand
>>>>>>>>>>>>>>>>>>>>>>>>>> the progress of rebalancing in quite some detail.
>>>> The
>>>>>>>>>>>>>>> information
>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>> really not understandable
>>>>>>>>>>>>>>>>>>>>>>>>>> for most users who do not have deep knowledge of
>>>>>>>>>>>> KIP-848/932.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> As a result, I suggest for kafka-share-groups.sh
>>>> that
>>>>>>>> you
>>>>>>>>>>>> add a
>>>>>>>>>>>>>>>>>>>>>>>> --members
>>>>>>>>>>>>>>>>>>>>>>>>>> --verbose option
>>>>>>>>>>>>>>>>>>>>>>>>>> and only include the new information in the output
>>>>>>> for
>>>>>>>>> that
>>>>>>>>>>>>>>> option,
>>>>>>>>>>>>>>>>>>>>>>>> rather
>>>>>>>>>>>>>>>>>>>>>>>>>> than changing the
>>>>>>>>>>>>>>>>>>>>>>>>>> non-verbose --members output.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I also make a similar suggestion for
>>>>>>>>>>>> kafka-consumer-groups.sh
>>>>>>>>>>>>>>>>>>>>> --members
>>>>>>>>>>>>>>>>>>>>>>>>>> and only add the
>>>>>>>>>>>>>>>>>>>>>>>>>> new information for the --verbose output.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> AS2: I strongly suggest that you use MEMBER-EPOCH
>>>>>>>>> instead of
>>>>>>>>>>>>>>>>>>>>>>>>>> CONSUMER-EPOCH.
>>>>>>>>>>>>>>>>>>>>>>>>>> I think it's more confusing not following the
>>>>>>>>> terminology of
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> KIPs.
>>>>>>>>>>>>>>>>>>>>>>>> For
>>>>>>>>>>>>>>>>>>>>>>>>>> one thing,
>>>>>>>>>>>>>>>>>>>>>>>>>> we already have "member" in the admin client such
>>>> as
>>>>>>>>>>>>>>>>>>>>> MemberDescription.
>>>>>>>>>>>>>>>>>>>>>>>>>> It's not a
>>>>>>>>>>>>>>>>>>>>>>>>>> new term.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> Andrew
>>>>>>>>>>>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>>>>>>>>>>>> From: PoAn Yang <pa...@apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>> Sent: 25 October 2024 13:55
>>>>>>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
>>>>>>>>>>>> kafka-consumer-groups
>>>>>>>>>>>>>>>>>>> command
>>>>>>>>>>>>>>>>>>>>>>>> line
>>>>>>>>>>>>>>>>>>>>>>>>>> tool to support new consumer group
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Lucas,
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the review!
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Yes, I add related change for
>>>>>>> kafka-share-groups.sh
>>>>>>>> to
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> KIP.
>>>>>>>>>>>>>>>>>>>>> Could
>>>>>>>>>>>>>>>>>>>>>>>>>> you take a look? Thanks for the suggestion.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use
>>>>>>>>> MEMBER-EPOCH
>>>>>>>>>>>>>>> here,
>>>>>>>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>>>>>>>>>>>> confuse what is different between CONSUMER and
>>>>>>> MEMBER.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Frank,
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks for the KIP!
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) For consistency, should we do the same for
>>>>>>>>>>>>>>>>>>>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription,
>>>> etc. ?
>>>>>>>>> Even
>>>>>>>>>>>> if
>>>>>>>>>>>>>>> we do
>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>>> implement it right now if the share group
>>>>>>>> implementation
>>>>>>>>>>>> may
>>>>>>>>>>>>>>> still
>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>> incomplete, it may make sense to include it in
>> the
>>>>>>>> KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH?
>>>>>>> That
>>>>>>>>> would
>>>>>>>>>>>>>>> seem
>>>>>>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>>>>>>>>> consistent.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>> Lucas
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <
>>>>>>>>>>>>>>> yangpoan@gmail.com>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to kick off the discussion of
>>>>>>> KIP-1099.
>>>>>>>>> This
>>>>>>>>>>>> KIP
>>>>>>>>>>>>>>>>>>>>> enhances
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups tools to include state
>>>> which
>>>>>>> is
>>>>>>>>>>>>>>> introduced
>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>>>> KIP-848.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups command
>>>> line
>>>>>>>>> tool
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> support
>>>>>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer group - Apache Kafka - Apache Software
>>>>>>>>> Foundation
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> cwiki.apache.org
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> [image: favicon.ico]
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>
>>
>>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by Chia-Ping Tsai <ch...@gmail.com>.
hi PoAn
`isClassic` is good to me :)
Best,
Chia-Ping
PoAn Yang <ya...@gmail.com> 於 2024年11月23日 週六 下午12:35寫道：
> Hi Chia-Ping / David,
>
> Thanks for the great suggestion.
>
> How about using “isClassic”?
> The term like “isLegacy” may be equivocal in the future.
> For example, if we have a newer version of consumer / share consumer, the
> meaning will be different for different cases.
> If we use “isClassic”, for MemberDescription in ClassicGroupDescription,
> the value can be always “true”.
> In ConsumerGroupDescription, it’s “true” when it’s a classic member.
> In ShareGroupDescription, it’s always “false”.
>
> Thanks,
> PoAn
>
> > On Nov 23, 2024, at 1:02 AM, Chia-Ping Tsai <ch...@gmail.com> wrote:
> >
> > hi DJ
> >
> >> My goal was to have a way to tell the user that his (new) consumer group
> > may have non-upgraded members yet. I wonder if we could use a boolean to
> > signal this, e.g. isLegacy. Then, in the CLI we could have a column which
> > is only displayed when they are non upgraded members. What do you think?
> >
> > Yes, this is a good approach, but the downside is that the new isLegacy
> > field in MemberDescription would be irrelevant for classic and shared
> > groups. This would require thorough documentation to clarify its purpose.
> >
> > Additionally, we could use a boolean instead of a string for the new
> field
> > in the RPC to reduce the request size.
> > Best,
> > Chia-Ping
> >
> >
> >
> > David Jacot <dj...@confluent.io.invalid> 於 2024年11月23日 週六 上午12:41寫道：
> >
> >> Hi both,
> >>
> >> Ah, I understand what you mean now. You're saying that we have two
> protocol
> >> fields. One at the group level and one at the member level (the new
> one).
> >> The two fields are actually two different things. The former is the
> name of
> >> the embedded protocol used in the classic protocol whereas the latter is
> >> the name of the protocol/api used. This is indeed confusing.
> >>
> >> My goal was to have a way to tell the user that his (new) consumer group
> >> may have non-upgraded members yet. I wonder if we could use a boolean to
> >> signal this, e.g. isLegacy. Then, in the CLI we could have a column
> which
> >> is only displayed when they are non upgraded members. What do you think?
> >>
> >> If we cannot find a good way to represent this, we could drop it from
> this
> >> KIP too.
> >>
> >> Best,
> >> David
> >>
> >> On Fri, Nov 22, 2024 at 5:12 PM PoAn Yang <ya...@gmail.com> wrote:
> >>
> >>> Hi Chia-Ping / David,
> >>>
> >>> Thanks for the review and suggestions.
> >>>
> >>>>> I don't fully follow your comment so please excuse me if I say
> >> something
> >>>>> wrong.
> >>>
> >>> I think Chia-Ping wants to say: originally, the default protocol in
> >>> classic group is “consumer”.
> >>> After we have a new consumer group, the protocol of a classic member in
> >>> consumer group is “classic”.
> >>> The MemberDescription is used by both ClassicGroupDescription and
> >>> ConsumerGroupDescription.
> >>> If we add the protocol field to MemberDescription, the protocol of a
> >>> classic member in classic group is “consumer”, but it’s “classic” in
> >>> consumer group.
> >>> To avoid misleading word, we can add a new collection of member
> protocols
> >>> to ConsumerGroupDescription only.
> >>> The original goal is to show member protocol during consumer migration,
> >> so
> >>> we don’t need to add this information to ClassicGroupDescription.
> >>>
> >>> Please correct me if I misunderstood anything.
> >>>
> >>> Thanks,
> >>> PoAn
> >>>
> >>>> On Nov 22, 2024, at 7:00 PM, Chia-Ping Tsai <ch...@gmail.com>
> >> wrote:
> >>>>
> >>>> hi DJ
> >>>>
> >>>>> How about we use another Jira to add “SHARE” to GroupProtocol and
> >> change
> >>>> return type of ModernGroup#protocolType?
> >>>>
> >>>> This is not my comment ... sorry for the incorrect quote :(
> >>>>
> >>>>> A member in a classic group should report "classic".
> >>>>
> >>>> I completely agree. However, in this case,
> >>> ClassicGroupDescription#protocol
> >>>> returns 'consumer', while ClassicGroupDescription#members[x].protocol
> >>>> returns 'classic'. This inconsistency feels a bit strange to me.
> >>>>
> >>>> Best,
> >>>> Chia-Ping
> >>>>
> >>>>
> >>>> David Jacot <dj...@confluent.io.invalid> 於 2024年11月22日 週五 下午6:51寫道：
> >>>>
> >>>>> Hi Chia,
> >>>>>
> >>>>>>> LM4: I agree it’s better to use a type like GroupProtocol enum, but
> >> I
> >>>>> would like to align the format with kafka-groups.sh.
> >>>>> How about we use another Jira to add “SHARE” to GroupProtocol and
> >> change
> >>>>> return type of ModernGroup#protocolType?
> >>>>>
> >>>>> My understanding is that `GroupProtocol` is used to drive the config
> >> of
> >>> the
> >>>>> `Consumer`. Hence adding `Share` seems wrong to me. Or are you
> >>> referring to
> >>>>> another GroupProtocol enum? Perhaps GroupType enum?
> >>>>>
> >>>>>> In fact, I'm wondering if we should even add 'protocol' to
> >>>>> MemberDescription,
> >>>>> since the field has different values between classic and consumer
> >>> groups—it
> >>>>> shows 'consumer' in classic groups, whereas it shows 'classic' in
> >>> consumer
> >>>>> groups. Yes, it seems like a typo, but it's true. This inconsistency
> >>> feels
> >>>>> confusing to me. For example, users call Admin#describeConsumerGroups
> >>> and
> >>>>> see that a member has the 'consumer' protocol when it's in a classic
> >>> group.
> >>>>> Then, if the group is upgraded to a consumer group, calling
> >>>>> Admin#describeConsumerGroups again shows the protocol as 'classic'
> for
> >>> the
> >>>>> same member.
> >>>>>
> >>>>> I don't fully follow your comment so please excuse me if I say
> >> something
> >>>>> wrong.
> >>>>>
> >>>>> I think that it should work as follows:
> >>>>> * A member in a classic group should report "classic".
> >>>>> * A member using the consumer protocol in a consumer group should
> >> report
> >>>>> "consumer".
> >>>>> * A member using the classic protocol in a consumer group should
> >> report
> >>>>> "classic".
> >>>>> * A member in a share group should report "share".
> >>>>>
> >>>>>> If all we want to do is distinguish classic members in the
> >>>>> ConsumerGroupDescription, then we shouldn't modify the common
> >>>>> MemberDescription structure. Instead, we could add a collection to
> >>>>> ConsumerGroupDescription to trace members using the classic protocol,
> >>> which
> >>>>> would allow us to enrich the output of command-line tools.
> >>>>>
> >>>>> Personally, I believe that using different collections per protocol
> >>> types
> >>>>> is not necessary. A field in the member is more than enough.
> >>>>>
> >>>>> Best,
> >>>>> DJ
> >>>>>
> >>>>> On Fri, Nov 22, 2024 at 12:01 PM Chia-Ping Tsai <ch...@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> hi PoAn
> >>>>>>
> >>>>>>> LM4: I agree it’s better to use a type like GroupProtocol enum, but
> >> I
> >>>>>> would like to align the format with kafka-groups.sh.
> >>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and
> >>> change
> >>>>>> return type of ModernGroup#protocolType?
> >>>>>>
> >>>>>> MemberDescription is also used by ClassicGroupDescription, and in
> >>> classic
> >>>>>> groups, the protocol can be an arbitrary string, so we can't use an
> >>> enum.
> >>>>>>
> >>>>>> In fact, I'm wondering if we should even add 'protocol' to
> >>>>>> MemberDescription,
> >>>>>> since the field has different values between classic and consumer
> >>>>> groups—it
> >>>>>> shows 'consumer' in classic groups, whereas it shows 'classic' in
> >>>>> consumer
> >>>>>> groups. Yes, it seems like a typo, but it's true. This inconsistency
> >>>>> feels
> >>>>>> confusing to me. For example, users call
> Admin#describeConsumerGroups
> >>> and
> >>>>>> see that a member has the 'consumer' protocol when it's in a classic
> >>>>> group.
> >>>>>> Then, if the group is upgraded to a consumer group, calling
> >>>>>> Admin#describeConsumerGroups again shows the protocol as 'classic'
> >> for
> >>>>> the
> >>>>>> same member.
> >>>>>>
> >>>>>> If all we want to do is distinguish classic members in the
> >>>>>> ConsumerGroupDescription, then we shouldn't modify the common
> >>>>>> MemberDescription structure. Instead, we could add a collection to
> >>>>>> ConsumerGroupDescription to trace members using the classic
> protocol,
> >>>>> which
> >>>>>> would allow us to enrich the output of command-line tools.
> >>>>>> Best,
> >>>>>> Chia-Ping
> >>>>>>
> >>>>>> PoAn Yang <ya...@gmail.com> 於 2024年11月22日 週五 下午3:46寫道：
> >>>>>>
> >>>>>>> Hi Lianet / Jeff,
> >>>>>>>
> >>>>>>> Thanks for the review.
> >>>>>>>
> >>>>>>> LM5: The kafka-share-groups.sh also uses “—describe” to show
> offsets
> >>>>>>> information.
> >>>>>>> Change to use “—describe —state —verbose” to show group level
> >>>>> information
> >>>>>>> in kafka-share-groups.sh.
> >>>>>>>
> >>>>>>> LM6: Update the example.
> >>>>>>>
> >>>>>>> LM7: Add a description to mention “—verbose” is a new option in
> >>>>>>> ShareGroupCommandOptions.
> >>>>>>>
> >>>>>>> JK2: Yes, add the statement about format change for ASSIGNMENT
> >> value.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> PoAn
> >>>>>>>
> >>>>>>>> On Nov 22, 2024, at 7:35 AM, Jeff Kim <je...@apache.org>
> >> wrote:
> >>>>>>>>
> >>>>>>>> Hi PoAn,
> >>>>>>>>
> >>>>>>>> JK2: Can we state in the KIP that we will reformat the existing
> >>>>>>>> classic group's `--members --verbose` as well, specifically the
> >>>>>>>> "ASSIGNMENT" column to include the topic name?
> >>>>>>>>
> >>>>>>>> Thanks!
> >>>>>>>>
> >>>>>>>> On 2024/11/21 14:33:37 "Lianet M." wrote:
> >>>>>>>>> Thanks for the updates PoAn!
> >>>>>>>>>
> >>>>>>>>> LM6. Just a nit, under the --describe --state --verbose the
> header
> >>>>> is
> >>>>>>> fine
> >>>>>>>>> but the example is still missing the --state argument.
> >>>>>>>>>
> >>>>>>>>> LM7. Under the Proposed changes for kafka-share-groups.sh, should
> >> we
> >>>>>>>>> clearly mention that the KIP adds a new --verbose option? I think
> >>>>> it's
> >>>>>>>>> confusing because it's presented as if the option already exists
> >> and
> >>>>>>> we're
> >>>>>>>>> only changing the output (which is the case for
> >>>>> kafka-consumer-groups
> >>>>>>> but
> >>>>>>>>> not for kafka-share-groups)
> >>>>>>>>>
> >>>>>>>>> That's all on my side. Thanks!
> >>>>>>>>>
> >>>>>>>>> Lianet
> >>>>>>>>>
> >>>>>>>>> On Wed, Nov 20, 2024 at 11:12 PM PoAn Yang <ya...@gmail.com>
> >>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi Lianet / Jeff,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for review and suggestions.
> >>>>>>>>>>
> >>>>>>>>>> LM4: At group level, the protocol field is from
> >>>>>>> ModernGroup#protocolType.
> >>>>>>>>>> It’s a string.
> >>>>>>>>>> The value is defined in different places like
> >>>>>> ShareGroup#PROTOCOL_TYPE
> >>>>>>> and
> >>>>>>>>>> ConsumerProtocol#PROTOCOL_TYPE.
> >>>>>>>>>> In KIP-1043 <
> >>>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=305171038#KIP1043:Administrationofgroups-kafka-groups.sh
> >>>>>>>> ,
> >>>>>>>>>> the kafka-groups.sh also shows protocol like “classic” /
> >>>>> “consumer” /
> >>>>>>>>>> “share”.
> >>>>>>>>>> I agree it’s better to use a type like GroupProtocol enum, but I
> >>>>>> would
> >>>>>>>>>> like to align the format with kafka-groups.sh.
> >>>>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol
> and
> >>>>>>> change
> >>>>>>>>>> return type of ModernGroup#protocolType?
> >>>>>>>>>>
> >>>>>>>>>> LM5 / JK1: Sorry, I misunderstood LM3, so I deleted —state in
> the
> >>>>>>> title.
> >>>>>>>>>> You’re correct. I would like to use `—describe —state —verbose`
> >> to
> >>>>>> show
> >>>>>>>>>> group level information.
> >>>>>>>>>> For `—describe —verbose`, it will show output as same as
> >> `—describe
> >>>>>>>>>> —offsets —verbose`,
> >>>>>>>>>> because `—describe` also shows output as same as `—describe
> >>>>>> —offsets`.
> >>>>>>>>>>
> >>>>>>>>>> JK2: Yes, I only use consumer group in the sample output, but
> the
> >>>>>>> classic
> >>>>>>>>>> group will use the same format as well.
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> PoAn
> >>>>>>>>>>
> >>>>>>>>>>> On Nov 21, 2024, at 8:28 AM, Jeff Kim <je...@apache.org>
> >>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for the KIP. I have some questions:
> >>>>>>>>>>>
> >>>>>>>>>>> JK1: (Realized this is similar to LM5) On `--describe
> --verbose`
> >>>>>>>>>> proposed changes - doesn't `--describe $group` default to
> >> printing
> >>>>>> the
> >>>>>>>>>> offsets? Perhaps you're referring to the `--state` argument?
> >> Also,
> >>>>>>> would
> >>>>>>>>>> that mean the default `--describe $group --verbose` command
> would
> >>>>> not
> >>>>>>> print
> >>>>>>>>>> the added field to `--offsets verbose` (leader-epoch) or would
> >> it?
> >>>>>>>>>>>
> >>>>>>>>>>> JK2: Looking at ConsumerGroupCommand.java, the existing
> >>>>> "ASSIGNMENT"
> >>>>>>>>>> column under `--members --verbose` does group the topic
> >> partitions
> >>>>> by
> >>>>>>> topic
> >>>>>>>>>> but does not prefix the grouping with the topic name like you're
> >>>>>>> proposal:
> >>>>>>>>>> "my_topic:0,1;new_topic:0,1". Should we do apply the same format
> >>>>> for
> >>>>>>> the
> >>>>>>>>>> classic group as well?
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On 2024/11/20 21:32:41 "Lianet M." wrote:
> >>>>>>>>>>>> Hello PoAn, just of couple more minor comments:
> >>>>>>>>>>>>
> >>>>>>>>>>>> LM4. Regarding the new “protocol” field added to
> >>>>> MemberDescription.
> >>>>>>>>>> Should
> >>>>>>>>>>>> we consider reusing the existing GroupProtocol enum instead of
> >>>>>>> String?
> >>>>>>>>>>>> (It’s the one we use from the consumer side to refer to the
> >>>>>> protocol
> >>>>>>> in
> >>>>>>>>>>>> use, just missing Share I notice).
> >>>>>>>>>>>>
> >>>>>>>>>>>> LM5. Regarding the change to the output for
> >>>>> kafka-consumer-groups,
> >>>>>>> the
> >>>>>>>>>>>> command shown does not include the —state option, but the
> >> output
> >>>>>>> shows
> >>>>>>>>>>>> state info (state, #members, epochs). I would guess that we
> >> want
> >>>>> to
> >>>>>>>>>> modify
> >>>>>>>>>>>> the output only when we describe a group with the —state
> >> —verbose
> >>>>>>>>>> option,
> >>>>>>>>>>>> is my understanding right? If my understanding is right we’re
> >>>>> just
> >>>>>>>>>> missing
> >>>>>>>>>>>> adding the —state in the example, and the KIP does not
> >> introduce
> >>>>>> any
> >>>>>>>>>>>> changes to the —describe —verbose option. (If not, it would
> >> mean
> >>>>> a
> >>>>>>>>>> bigger
> >>>>>>>>>>>> change to the output of —describe —verbose which I expect is
> >> not
> >>>>>> the
> >>>>>>>>>>>> intention?)
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>
> >>>>>>>>>>>> Lianet
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Wed, Nov 20, 2024 at 2:12 AM PoAn Yang <yangpoan@gmail.com
> >
> >>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Chia-Ping,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for the review and suggestions.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> CI0: Thanks for the reminder. Update validVersions in
> >>>>>>>>>>>>> ConsumerGroupDescribeRequest to 0-1.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> CI1: Yes, I use `ConsumerGroupMember#useClassicProtocol` to
> >>>>> check
> >>>>>>>>>> whether
> >>>>>>>>>>>>> a member in consumer group uses “classic” or “consumer”
> >>>>> protocol.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> CI2: Yes, a member in share group always uses “share”
> >> protocol.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> CI3: Add a table to show meaning of “classic”, “consumer”,
> and
> >>>>>>> “share”
> >>>>>>>>>>>>> protocol.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> BTW, the vote thread is in
> >>>>>>>>>>>>>
> >>>>> https://lists.apache.org/thread/rb25tf75tzf4c7jqqldlo5jh9w8chsq6.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Nov 20, 2024, at 11:46 AM, Chia-Ping Tsai <
> >>>>>> chia7712@apache.org>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> hi PoAn
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> CI0: We have to bump the version of
> >>>>> ConsumerGroupDescribeRequest
> >>>>>>> as
> >>>>>>>>>>>>> well, so server can distinguish the new and old behavior.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> CI1: the type of new filed is string, so I guess you plan to
> >>>>> use
> >>>>>>>>>>>>> `ConsumerGroupMember#useClassicProtocol` [0] flag to return
> >>>>> either
> >>>>>>>>>>>>> "classic" or "consumer", right?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> CI2: `MemberDescription` is used by `ShareGroupDescription`
> >>>>> too,
> >>>>>> so
> >>>>>>>>>> the
> >>>>>>>>>>>>> new filed protocol in shared group is always "shared", right?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> CI3: Could you consider adding a table to show the value of
> >> the
> >>>>>>>>>> protocol
> >>>>>>>>>>>>> field in each case? Andrew has a beautiful table in KIP-1043
> >>>>> that
> >>>>>>>>>> lists all
> >>>>>>>>>>>>> possible protocol names.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> [0]
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>
> https://github.com/apache/kafka/blob/441a6d0b790f4a17b454caeea7588a6b90fbd9db/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/consumer/ConsumerGroupMember.java#L454
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 2024/11/19 15:45:16 PoAn Yang wrote:
> >>>>>>>>>>>>>>> Hi David,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks for the reminder.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> DJ4: Add related content to the KIP.
> >>>>>>>>>>>>>>> Bump validVersions in ConsumerGroupDescribeResponse to
> >> “0-1”.
> >>>>>>>>>>>>>>> Add a new field “Protocol” to
> ConsumerGroupDescribeResponse.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Nov 19, 2024, at 3:57 PM, David Jacot
> >>>>>>>>>> <dj...@confluent.io.INVALID>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> DJ4: Could you please also include the required changes to
> >>>>> the
> >>>>>>>>>>>>>>>> ConsumerGroupDescribed API in the public interface
> section?
> >>>>> We
> >>>>>>>>>>>>> basically
> >>>>>>>>>>>>>>>> need to bump the version, add the new field, etc.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>> David
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang <
> >>>>> yangpoan@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi David,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> DJ4: Add a new field protocol to MemberDescription,
> >>>>>>>>>>>>>>>>> so the command line tool can show protocol information
> >> when
> >>>>>>> users
> >>>>>>>>>>>>> describe
> >>>>>>>>>>>>>>>>> members.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> If there is no further suggestion, I will start a vote
> >>>>> thread
> >>>>>>>>>> today.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Nov 15, 2024, at 11:50 PM, David Jacot
> >>>>>>>>>>>>> <dj...@confluent.io.INVALID>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> DJ2: Using "-" sounds good to me.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> DJ3: That seems reasonable to me.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> DJ4: Why not add it right now? I don't want to change
> the
> >>>>>>> output
> >>>>>>>>>> of
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> tool too many times.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>> David
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang <
> >>>>>> yangpoan@gmail.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi David / Andrew,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks for review. Thanks Andrew for picking up
> >>>>>>>>>>>>> kafka-share-groups.sh
> >>>>>>>>>>>>>>>>>>> implementation.
> >>>>>>>>>>>>>>>>>>> I will handle kafka-consumer-groups.sh.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> DJ3: After discussing with @Chia-Ping Tsai, we think
> >> that
> >>>>>>> using
> >>>>>>>>>> new
> >>>>>>>>>>>>>>>>> format
> >>>>>>>>>>>>>>>>>>> is more clear.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> The new format will be like
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>
> <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Using colon(:) to concat topic name and partition IDs.
> >>>>>>>>>>>>>>>>>>> Using comma(,) to concat partition IDs within same
> topic
> >>>>>> name.
> >>>>>>>>>>>>>>>>>>> Using semicolon(;) to concat topic strings.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> AS3: Fix it with kafka-share-groups.sh. Thanks.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> If there is no further suggestion, I will start a vote
> >>>>>> thread
> >>>>>>>>>> next
> >>>>>>>>>>>>>>>>> Monday.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield <
> >>>>>>>>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>>>>>>>>>>> DJ2: I was just going to comment that "-" would be a
> >> more
> >>>>>>>>>>>>> appropriate
> >>>>>>>>>>>>>>>>>>> missing value, but
> >>>>>>>>>>>>>>>>>>>> you got there first.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> AS3: The examples for kafka-share-groups.sh include
> >>>>>>>>>>>>>>>>>>> kafka-consumer-groups.sh in the
> >>>>>>>>>>>>>>>>>>>> command line.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> If this is accepted in time, I'm happy to pick up the
> >>>>>>>>>>>>> implementation of
> >>>>>>>>>>>>>>>>>>> the share groups
> >>>>>>>>>>>>>>>>>>>> part of this if it helps.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>> Andrew
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>>>>>>>> From: Frank Yang <ya...@gmail.com>
> >>>>>>>>>>>>>>>>>>>> Sent: 14 November 2024 10:48
> >>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
> >>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
> >>>>>> kafka-consumer-groups
> >>>>>>>>>>>>> command
> >>>>>>>>>>>>>>>>>>> line tool to support new consumer group
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hi David,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks for the review and suggestion! I would like to
> >> get
> >>>>>>> this
> >>>>>>>>>> in
> >>>>>>>>>>>>> AK
> >>>>>>>>>>>>>>>>> 4.0
> >>>>>>>>>>>>>>>>>>> as well. I will do my best.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> DJ1: Update KIP to put GROUP-EPOCH and
> >>>>>>> TARGET-ASSIGNMENT-EPOCH
> >>>>>>>>>>>>> before
> >>>>>>>>>>>>>>>>>>> #MEMBERS.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> DJ2: I prefer to follow current missing column value
> >> “-“.
> >>>>>>>>>>>>> (reference <
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>
> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92
> >>>>>>>>>>>>>>>>>>>> )
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> DJ3: Update KIP to use CURRENT-EPOCH
> CURRENT-ASSIGNMENT
> >>>>>>>>>>>>> TARGET-EPOCH
> >>>>>>>>>>>>>>>>>>> TARGET-ASSIGNMENT.
> >>>>>>>>>>>>>>>>>>>> Remove GROUP-EPOCH.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> For assignment value, it follows current output
> >>>>> (reference
> >>>>>> <
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>
> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418
> >>>>>>>>>>>>>>>>>> ).
> >>>>>>>>>>>>>>>>>>> I think the form of `topicid-partitionid` is more
> clear.
> >>>>>>>>>>>>>>>>>>>> If we would like to use this form, I will update both
> >>>>>> output
> >>>>>>> in
> >>>>>>>>>>>>>>>>>>> kafka-consumer-groups.sh and kafka-share-groups.sh.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> DJ4: It looks like DescribeGroupsResponseData only has
> >>>>>>> protocol
> >>>>>>>>>>>>> type at
> >>>>>>>>>>>>>>>>>>> group level.
> >>>>>>>>>>>>>>>>>>>> Both DescribeGroupsResponseData and
> >>>>>>>>>>>>> ConsumerGroupDescribeResponseData
> >>>>>>>>>>>>>>>>>>> don’t have protocol at member level.
> >>>>>>>>>>>>>>>>>>>> Could we use a followup to add it?
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> DJ5: Update KIP to put LEADER-EPOCH before
> >>>>> CURRENT-OFFSET.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot
> >>>>>>>>>>>>> <djacot@confluent.io.INVALID
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks for the KIP! I have a few minor
> >>>>>> comments/suggestions:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> DJ1: In the output of `--describe --verbose`, I would
> >>>>>>> suggest
> >>>>>>>>>>>>> putting
> >>>>>>>>>>>>>>>>>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before
> >>>>>>> `#MEMBERS`.
> >>>>>>>>>>>>>>>>>>>>> DJ2: Continuing on the above, I assume that we will
> >>>>> print
> >>>>>>> out
> >>>>>>>>>> N/A
> >>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> fields not supported by classic groups. Is this
> >> correct?
> >>>>>>>>>>>>>>>>>>>>> DJ3: In the output of `--describe --members
> >> --verbose`,
> >>>>> I
> >>>>>>>>>> wonder
> >>>>>>>>>>>>> if we
> >>>>>>>>>>>>>>>>>>>>> should use the following order and terms:
> >> CURRENT-EPOCH
> >>>>>>>>>>>>>>>>>>> CURRENT-ASSIGNMENT
> >>>>>>>>>>>>>>>>>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the
> >>>>>>> GROUP-EPOCH
> >>>>>>>>>>>>>>>>> because
> >>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>> is already in the group description. The value `(0)`
> >>>>> used
> >>>>>>> for
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> assignment is incorrect. Here I suppose that we will
> >>>>> print
> >>>>>>> out
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>>> list
> >>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>> partitions in the form of `topicid-partitionid`.
> >>>>>>>>>>>>>>>>>>>>> DJ4: Continuing on the above, I wonder if we should
> >> also
> >>>>>> add
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> protocol
> >>>>>>>>>>>>>>>>>>>>> used `classic` or `consumer`. For context, it is
> >>>>> possible
> >>>>>> to
> >>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>> `classic`
> >>>>>>>>>>>>>>>>>>>>> members and `consumer` members in a `consumer` group
> >>>>>> during
> >>>>>>> an
> >>>>>>>>>>>>> online
> >>>>>>>>>>>>>>>>>>>>> upgrade from the classic protocol to the consumer
> >>>>>> protocol.
> >>>>>>>>>> Having
> >>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>> information may be handy for administrators. What do
> >> you
> >>>>>>> think?
> >>>>>>>>>>>>>>>>>>>>> DJ5: In the output of `--describe --offsets
> >> --verbose`,
> >>>>> I
> >>>>>>> would
> >>>>>>>>>>>>>>>>> suggest
> >>>>>>>>>>>>>>>>>>>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> It would be great if we could get this one in AK 4.0
> >> as
> >>>>> it
> >>>>>>>>>>>>> changes the
> >>>>>>>>>>>>>>>>>>>>> output of the command.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>> DJ
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <
> >>>>>>> yangpoan@gmail.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Hi Sean / Andrew / Lianet,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Thanks for all your review and suggestions.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> AS1, LM1, LM4: Change to add KIP-848 information
> when
> >>>>>> users
> >>>>>>>>>> give
> >>>>>>>>>>>>>>>>>>> —verbose
> >>>>>>>>>>>>>>>>>>>>>> option.
> >>>>>>>>>>>>>>>>>>>>>> —describe —verbose: shows group epoch and target
> >>>>>> assignment
> >>>>>>>>>>>>> epoch.
> >>>>>>>>>>>>>>>>>>>>>> —describe —members —verbose: shows above
> information,
> >>>>>>> member
> >>>>>>>>>>>>> epoch,
> >>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>> target assignment.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> AS2: Change to use MEMBER-EPOCH to align with
> KIP-848
> >>>>>>>>>> definition.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> LM2: For classic group, it doesn’t have epoch, so I
> >> use
> >>>>>>>>>> Optional
> >>>>>>>>>>>>>>>>>>> fields in
> >>>>>>>>>>>>>>>>>>>>>> ConsumerGroupDescription.
> >>>>>>>>>>>>>>>>>>>>>> For share group, it relies on KIP-848. It must have
> >>>>>> epoch,
> >>>>>>> so
> >>>>>>>>>> I
> >>>>>>>>>>>>> use
> >>>>>>>>>>>>>>>>> int
> >>>>>>>>>>>>>>>>>>>>>> fields in ShareGroupDescription.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> LM3: Remove —state change. We can get group level
> >>>>>>> information
> >>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>> —describe
> >>>>>>>>>>>>>>>>>>>>>> —verbose.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> SQ1: Add LEADER-EPOCH when users give —describe
> >>>>> —offsets
> >>>>>>>>>>>>> —verbose.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Thanks.
> >>>>>>>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <
> >>>>>> lianetmr@gmail.com
> >>>>>>>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Hello Frank, thanks for the KIP! A few comments:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> LM1. I strongly agree with Andrew's suggestion of
> >>>>> moving
> >>>>>>> this
> >>>>>>>>>>>>> into a
> >>>>>>>>>>>>>>>>>>>>>>> --verbose option. I expect someone would only
> >> attempt
> >>>>> to
> >>>>>>> make
> >>>>>>>>>>>>> sense
> >>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> epochs while debugging an issue, not while trying
> to
> >>>>>> view
> >>>>>>> the
> >>>>>>>>>>>>> group
> >>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>> member's info. (Also member-epoch makes more sense
> >> to
> >>>>>> me,
> >>>>>>>>>> even
> >>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>>> if we
> >>>>>>>>>>>>>>>>>>>>>>> end up agreeing in a --verbose). Related to both
> >>>>> issues,
> >>>>>>> my
> >>>>>>>>>>>>> take is
> >>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>> whoever is close to the protocol will expect
> >>>>>> member-epoch,
> >>>>>>>>>>>>> whoever
> >>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>> will probably won't even need to see the epochs at
> >>>>> all.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> LM2. Why are the epochs defined as Optional (don't
> >> we
> >>>>>>> expect
> >>>>>>>>>> to
> >>>>>>>>>>>>>>>>> always
> >>>>>>>>>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>>>>>> them? with 0 initially, defined on the broker side
> >> for
> >>>>>> the
> >>>>>>>>>> group
> >>>>>>>>>>>>>>>>> ones,
> >>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>> client side for the member)
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> LM3. Why is the KIP including the –state option in
> >> the
> >>>>>>>>>> proposed
> >>>>>>>>>>>>>>>>>>> changes?
> >>>>>>>>>>>>>>>>>>>>>> (
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
> >>>>>>>>>>>>>>>>>>>>>>> ).
> >>>>>>>>>>>>>>>>>>>>>>> I get that the output would change in that example,
> >>>>> but
> >>>>>>> it’s
> >>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>> because
> >>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>> any change to the –state option itself. It's
> because
> >>>>> of
> >>>>>>> the
> >>>>>>>>>>>>> change
> >>>>>>>>>>>>>>>>>>>>>> proposed
> >>>>>>>>>>>>>>>>>>>>>>> to the –described (required with the --state), and
> >> the
> >>>>>>>>>> changes
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>> --describe are already explained above (seems
> >>>>> confusing,
> >>>>>>> got
> >>>>>>>>>> me
> >>>>>>>>>>>>>>>>>>> looking
> >>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>> a change to the state filter which seemed
> >> unrelated).
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> LM4. Group epoch and target assignment epoch are
> >>>>>>> conceptually
> >>>>>>>>>>>>> at the
> >>>>>>>>>>>>>>>>>>>>>> group
> >>>>>>>>>>>>>>>>>>>>>>> level. vs member epoch that lands at a member
> level.
> >>>>> So
> >>>>>>>>>> wonder
> >>>>>>>>>>>>> if we
> >>>>>>>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>>>>> show them accordingly (ex. using the --verbose
> >> option)
> >>>>>>>>>>>>>>>>>>>>>>> –describe –verbose => shows group epoch and target
> >>>>>>> assignment
> >>>>>>>>>>>>> epoch
> >>>>>>>>>>>>>>>>>>>>>>> –describe –members –verbose => shows member
> >>>>>> epoch
> >>>>>>>>>>>>> (along
> >>>>>>>>>>>>>>>>>>>>>>> with group epoch and target assignment epoch)
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Lianet
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield <
> >>>>>>>>>>>>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I have a few comments.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> AS1: It seems to me that these new additions are
> >> most
> >>>>>>>>>> useful to
> >>>>>>>>>>>>>>>>>>> people
> >>>>>>>>>>>>>>>>>>>>>>>> trying to understand
> >>>>>>>>>>>>>>>>>>>>>>>> the progress of rebalancing in quite some detail.
> >> The
> >>>>>>>>>>>>> information
> >>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>> really not understandable
> >>>>>>>>>>>>>>>>>>>>>>>> for most users who do not have deep knowledge of
> >>>>>>>>>> KIP-848/932.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> As a result, I suggest for kafka-share-groups.sh
> >> that
> >>>>>> you
> >>>>>>>>>> add a
> >>>>>>>>>>>>>>>>>>>>>> --members
> >>>>>>>>>>>>>>>>>>>>>>>> --verbose option
> >>>>>>>>>>>>>>>>>>>>>>>> and only include the new information in the output
> >>>>> for
> >>>>>>> that
> >>>>>>>>>>>>> option,
> >>>>>>>>>>>>>>>>>>>>>> rather
> >>>>>>>>>>>>>>>>>>>>>>>> than changing the
> >>>>>>>>>>>>>>>>>>>>>>>> non-verbose --members output.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> I also make a similar suggestion for
> >>>>>>>>>> kafka-consumer-groups.sh
> >>>>>>>>>>>>>>>>>>> --members
> >>>>>>>>>>>>>>>>>>>>>>>> and only add the
> >>>>>>>>>>>>>>>>>>>>>>>> new information for the --verbose output.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> AS2: I strongly suggest that you use MEMBER-EPOCH
> >>>>>>> instead of
> >>>>>>>>>>>>>>>>>>>>>>>> CONSUMER-EPOCH.
> >>>>>>>>>>>>>>>>>>>>>>>> I think it's more confusing not following the
> >>>>>>> terminology of
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> KIPs.
> >>>>>>>>>>>>>>>>>>>>>> For
> >>>>>>>>>>>>>>>>>>>>>>>> one thing,
> >>>>>>>>>>>>>>>>>>>>>>>> we already have "member" in the admin client such
> >> as
> >>>>>>>>>>>>>>>>>>> MemberDescription.
> >>>>>>>>>>>>>>>>>>>>>>>> It's not a
> >>>>>>>>>>>>>>>>>>>>>>>> new term.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>> Andrew
> >>>>>>>>>>>>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>>>>>>>>>>>> From: PoAn Yang <pa...@apache.org>
> >>>>>>>>>>>>>>>>>>>>>>>> Sent: 25 October 2024 13:55
> >>>>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
> >>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
> >>>>>>>>>> kafka-consumer-groups
> >>>>>>>>>>>>>>>>> command
> >>>>>>>>>>>>>>>>>>>>>> line
> >>>>>>>>>>>>>>>>>>>>>>>> tool to support new consumer group
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Hi Lucas,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the review!
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> 1) Yes, I add related change for
> >>>>> kafka-share-groups.sh
> >>>>>> to
> >>>>>>>>>> the
> >>>>>>>>>>>>> KIP.
> >>>>>>>>>>>>>>>>>>> Could
> >>>>>>>>>>>>>>>>>>>>>>>> you take a look? Thanks for the suggestion.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use
> >>>>>>> MEMBER-EPOCH
> >>>>>>>>>>>>> here,
> >>>>>>>>>>>>>>>>>>> users
> >>>>>>>>>>>>>>>>>>>>>> may
> >>>>>>>>>>>>>>>>>>>>>>>> confuse what is different between CONSUMER and
> >>>>> MEMBER.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>> Hi Frank,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> thanks for the KIP!
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> 1) For consistency, should we do the same for
> >>>>>>>>>>>>>>>>>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription,
> >> etc. ?
> >>>>>>> Even
> >>>>>>>>>> if
> >>>>>>>>>>>>> we do
> >>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>> implement it right now if the share group
> >>>>>> implementation
> >>>>>>>>>> may
> >>>>>>>>>>>>> still
> >>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>> incomplete, it may make sense to include it in
> the
> >>>>>> KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH?
> >>>>> That
> >>>>>>> would
> >>>>>>>>>>>>> seem
> >>>>>>>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>>>>>>>>> consistent.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>>>>> Lucas
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <
> >>>>>>>>>>>>> yangpoan@gmail.com>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> I would like to kick off the discussion of
> >>>>> KIP-1099.
> >>>>>>> This
> >>>>>>>>>> KIP
> >>>>>>>>>>>>>>>>>>> enhances
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups tools to include state
> >> which
> >>>>> is
> >>>>>>>>>>>>> introduced
> >>>>>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>>>>>>> KIP-848.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups command
> >> line
> >>>>>>> tool
> >>>>>>>>>> to
> >>>>>>>>>>>>>>>>> support
> >>>>>>>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>>>>>>> consumer group - Apache Kafka - Apache Software
> >>>>>>> Foundation
> >>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> cwiki.apache.org
> >>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> [image: favicon.ico]
> >>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thank you,
> >>>>>>>>>>>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>>
> >>>
> >>
>
>
>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by PoAn Yang <ya...@gmail.com>.
Hi Chia-Ping / David,
Thanks for the great suggestion.
How about using “isClassic”?
The term like “isLegacy” may be equivocal in the future.
For example, if we have a newer version of consumer / share consumer, the meaning will be different for different cases.
If we use “isClassic”, for MemberDescription in ClassicGroupDescription, the value can be always “true”.
In ConsumerGroupDescription, it’s “true” when it’s a classic member.
In ShareGroupDescription, it’s always “false”.
Thanks,
PoAn
> On Nov 23, 2024, at 1:02 AM, Chia-Ping Tsai <ch...@gmail.com> wrote:
>
> hi DJ
>
>> My goal was to have a way to tell the user that his (new) consumer group
> may have non-upgraded members yet. I wonder if we could use a boolean to
> signal this, e.g. isLegacy. Then, in the CLI we could have a column which
> is only displayed when they are non upgraded members. What do you think?
>
> Yes, this is a good approach, but the downside is that the new isLegacy
> field in MemberDescription would be irrelevant for classic and shared
> groups. This would require thorough documentation to clarify its purpose.
>
> Additionally, we could use a boolean instead of a string for the new field
> in the RPC to reduce the request size.
> Best,
> Chia-Ping
>
>
>
> David Jacot <dj...@confluent.io.invalid> 於 2024年11月23日 週六 上午12:41寫道：
>
>> Hi both,
>>
>> Ah, I understand what you mean now. You're saying that we have two protocol
>> fields. One at the group level and one at the member level (the new one).
>> The two fields are actually two different things. The former is the name of
>> the embedded protocol used in the classic protocol whereas the latter is
>> the name of the protocol/api used. This is indeed confusing.
>>
>> My goal was to have a way to tell the user that his (new) consumer group
>> may have non-upgraded members yet. I wonder if we could use a boolean to
>> signal this, e.g. isLegacy. Then, in the CLI we could have a column which
>> is only displayed when they are non upgraded members. What do you think?
>>
>> If we cannot find a good way to represent this, we could drop it from this
>> KIP too.
>>
>> Best,
>> David
>>
>> On Fri, Nov 22, 2024 at 5:12 PM PoAn Yang <ya...@gmail.com> wrote:
>>
>>> Hi Chia-Ping / David,
>>>
>>> Thanks for the review and suggestions.
>>>
>>>>> I don't fully follow your comment so please excuse me if I say
>> something
>>>>> wrong.
>>>
>>> I think Chia-Ping wants to say: originally, the default protocol in
>>> classic group is “consumer”.
>>> After we have a new consumer group, the protocol of a classic member in
>>> consumer group is “classic”.
>>> The MemberDescription is used by both ClassicGroupDescription and
>>> ConsumerGroupDescription.
>>> If we add the protocol field to MemberDescription, the protocol of a
>>> classic member in classic group is “consumer”, but it’s “classic” in
>>> consumer group.
>>> To avoid misleading word, we can add a new collection of member protocols
>>> to ConsumerGroupDescription only.
>>> The original goal is to show member protocol during consumer migration,
>> so
>>> we don’t need to add this information to ClassicGroupDescription.
>>>
>>> Please correct me if I misunderstood anything.
>>>
>>> Thanks,
>>> PoAn
>>>
>>>> On Nov 22, 2024, at 7:00 PM, Chia-Ping Tsai <ch...@gmail.com>
>> wrote:
>>>>
>>>> hi DJ
>>>>
>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and
>> change
>>>> return type of ModernGroup#protocolType?
>>>>
>>>> This is not my comment ... sorry for the incorrect quote :(
>>>>
>>>>> A member in a classic group should report "classic".
>>>>
>>>> I completely agree. However, in this case,
>>> ClassicGroupDescription#protocol
>>>> returns 'consumer', while ClassicGroupDescription#members[x].protocol
>>>> returns 'classic'. This inconsistency feels a bit strange to me.
>>>>
>>>> Best,
>>>> Chia-Ping
>>>>
>>>>
>>>> David Jacot <dj...@confluent.io.invalid> 於 2024年11月22日 週五 下午6:51寫道：
>>>>
>>>>> Hi Chia,
>>>>>
>>>>>>> LM4: I agree it’s better to use a type like GroupProtocol enum, but
>> I
>>>>> would like to align the format with kafka-groups.sh.
>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and
>> change
>>>>> return type of ModernGroup#protocolType?
>>>>>
>>>>> My understanding is that `GroupProtocol` is used to drive the config
>> of
>>> the
>>>>> `Consumer`. Hence adding `Share` seems wrong to me. Or are you
>>> referring to
>>>>> another GroupProtocol enum? Perhaps GroupType enum?
>>>>>
>>>>>> In fact, I'm wondering if we should even add 'protocol' to
>>>>> MemberDescription,
>>>>> since the field has different values between classic and consumer
>>> groups—it
>>>>> shows 'consumer' in classic groups, whereas it shows 'classic' in
>>> consumer
>>>>> groups. Yes, it seems like a typo, but it's true. This inconsistency
>>> feels
>>>>> confusing to me. For example, users call Admin#describeConsumerGroups
>>> and
>>>>> see that a member has the 'consumer' protocol when it's in a classic
>>> group.
>>>>> Then, if the group is upgraded to a consumer group, calling
>>>>> Admin#describeConsumerGroups again shows the protocol as 'classic' for
>>> the
>>>>> same member.
>>>>>
>>>>> I don't fully follow your comment so please excuse me if I say
>> something
>>>>> wrong.
>>>>>
>>>>> I think that it should work as follows:
>>>>> * A member in a classic group should report "classic".
>>>>> * A member using the consumer protocol in a consumer group should
>> report
>>>>> "consumer".
>>>>> * A member using the classic protocol in a consumer group should
>> report
>>>>> "classic".
>>>>> * A member in a share group should report "share".
>>>>>
>>>>>> If all we want to do is distinguish classic members in the
>>>>> ConsumerGroupDescription, then we shouldn't modify the common
>>>>> MemberDescription structure. Instead, we could add a collection to
>>>>> ConsumerGroupDescription to trace members using the classic protocol,
>>> which
>>>>> would allow us to enrich the output of command-line tools.
>>>>>
>>>>> Personally, I believe that using different collections per protocol
>>> types
>>>>> is not necessary. A field in the member is more than enough.
>>>>>
>>>>> Best,
>>>>> DJ
>>>>>
>>>>> On Fri, Nov 22, 2024 at 12:01 PM Chia-Ping Tsai <ch...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> hi PoAn
>>>>>>
>>>>>>> LM4: I agree it’s better to use a type like GroupProtocol enum, but
>> I
>>>>>> would like to align the format with kafka-groups.sh.
>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and
>>> change
>>>>>> return type of ModernGroup#protocolType?
>>>>>>
>>>>>> MemberDescription is also used by ClassicGroupDescription, and in
>>> classic
>>>>>> groups, the protocol can be an arbitrary string, so we can't use an
>>> enum.
>>>>>>
>>>>>> In fact, I'm wondering if we should even add 'protocol' to
>>>>>> MemberDescription,
>>>>>> since the field has different values between classic and consumer
>>>>> groups—it
>>>>>> shows 'consumer' in classic groups, whereas it shows 'classic' in
>>>>> consumer
>>>>>> groups. Yes, it seems like a typo, but it's true. This inconsistency
>>>>> feels
>>>>>> confusing to me. For example, users call Admin#describeConsumerGroups
>>> and
>>>>>> see that a member has the 'consumer' protocol when it's in a classic
>>>>> group.
>>>>>> Then, if the group is upgraded to a consumer group, calling
>>>>>> Admin#describeConsumerGroups again shows the protocol as 'classic'
>> for
>>>>> the
>>>>>> same member.
>>>>>>
>>>>>> If all we want to do is distinguish classic members in the
>>>>>> ConsumerGroupDescription, then we shouldn't modify the common
>>>>>> MemberDescription structure. Instead, we could add a collection to
>>>>>> ConsumerGroupDescription to trace members using the classic protocol,
>>>>> which
>>>>>> would allow us to enrich the output of command-line tools.
>>>>>> Best,
>>>>>> Chia-Ping
>>>>>>
>>>>>> PoAn Yang <ya...@gmail.com> 於 2024年11月22日 週五 下午3:46寫道：
>>>>>>
>>>>>>> Hi Lianet / Jeff,
>>>>>>>
>>>>>>> Thanks for the review.
>>>>>>>
>>>>>>> LM5: The kafka-share-groups.sh also uses “—describe” to show offsets
>>>>>>> information.
>>>>>>> Change to use “—describe —state —verbose” to show group level
>>>>> information
>>>>>>> in kafka-share-groups.sh.
>>>>>>>
>>>>>>> LM6: Update the example.
>>>>>>>
>>>>>>> LM7: Add a description to mention “—verbose” is a new option in
>>>>>>> ShareGroupCommandOptions.
>>>>>>>
>>>>>>> JK2: Yes, add the statement about format change for ASSIGNMENT
>> value.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> PoAn
>>>>>>>
>>>>>>>> On Nov 22, 2024, at 7:35 AM, Jeff Kim <je...@apache.org>
>> wrote:
>>>>>>>>
>>>>>>>> Hi PoAn,
>>>>>>>>
>>>>>>>> JK2: Can we state in the KIP that we will reformat the existing
>>>>>>>> classic group's `--members --verbose` as well, specifically the
>>>>>>>> "ASSIGNMENT" column to include the topic name?
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> On 2024/11/21 14:33:37 "Lianet M." wrote:
>>>>>>>>> Thanks for the updates PoAn!
>>>>>>>>>
>>>>>>>>> LM6. Just a nit, under the --describe --state --verbose the header
>>>>> is
>>>>>>> fine
>>>>>>>>> but the example is still missing the --state argument.
>>>>>>>>>
>>>>>>>>> LM7. Under the Proposed changes for kafka-share-groups.sh, should
>> we
>>>>>>>>> clearly mention that the KIP adds a new --verbose option? I think
>>>>> it's
>>>>>>>>> confusing because it's presented as if the option already exists
>> and
>>>>>>> we're
>>>>>>>>> only changing the output (which is the case for
>>>>> kafka-consumer-groups
>>>>>>> but
>>>>>>>>> not for kafka-share-groups)
>>>>>>>>>
>>>>>>>>> That's all on my side. Thanks!
>>>>>>>>>
>>>>>>>>> Lianet
>>>>>>>>>
>>>>>>>>> On Wed, Nov 20, 2024 at 11:12 PM PoAn Yang <ya...@gmail.com>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Lianet / Jeff,
>>>>>>>>>>
>>>>>>>>>> Thanks for review and suggestions.
>>>>>>>>>>
>>>>>>>>>> LM4: At group level, the protocol field is from
>>>>>>> ModernGroup#protocolType.
>>>>>>>>>> It’s a string.
>>>>>>>>>> The value is defined in different places like
>>>>>> ShareGroup#PROTOCOL_TYPE
>>>>>>> and
>>>>>>>>>> ConsumerProtocol#PROTOCOL_TYPE.
>>>>>>>>>> In KIP-1043 <
>>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=305171038#KIP1043:Administrationofgroups-kafka-groups.sh
>>>>>>>> ,
>>>>>>>>>> the kafka-groups.sh also shows protocol like “classic” /
>>>>> “consumer” /
>>>>>>>>>> “share”.
>>>>>>>>>> I agree it’s better to use a type like GroupProtocol enum, but I
>>>>>> would
>>>>>>>>>> like to align the format with kafka-groups.sh.
>>>>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and
>>>>>>> change
>>>>>>>>>> return type of ModernGroup#protocolType?
>>>>>>>>>>
>>>>>>>>>> LM5 / JK1: Sorry, I misunderstood LM3, so I deleted —state in the
>>>>>>> title.
>>>>>>>>>> You’re correct. I would like to use `—describe —state —verbose`
>> to
>>>>>> show
>>>>>>>>>> group level information.
>>>>>>>>>> For `—describe —verbose`, it will show output as same as
>> `—describe
>>>>>>>>>> —offsets —verbose`,
>>>>>>>>>> because `—describe` also shows output as same as `—describe
>>>>>> —offsets`.
>>>>>>>>>>
>>>>>>>>>> JK2: Yes, I only use consumer group in the sample output, but the
>>>>>>> classic
>>>>>>>>>> group will use the same format as well.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> PoAn
>>>>>>>>>>
>>>>>>>>>>> On Nov 21, 2024, at 8:28 AM, Jeff Kim <je...@apache.org>
>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>
>>>>>>>>>>> Thanks for the KIP. I have some questions:
>>>>>>>>>>>
>>>>>>>>>>> JK1: (Realized this is similar to LM5) On `--describe --verbose`
>>>>>>>>>> proposed changes - doesn't `--describe $group` default to
>> printing
>>>>>> the
>>>>>>>>>> offsets? Perhaps you're referring to the `--state` argument?
>> Also,
>>>>>>> would
>>>>>>>>>> that mean the default `--describe $group --verbose` command would
>>>>> not
>>>>>>> print
>>>>>>>>>> the added field to `--offsets verbose` (leader-epoch) or would
>> it?
>>>>>>>>>>>
>>>>>>>>>>> JK2: Looking at ConsumerGroupCommand.java, the existing
>>>>> "ASSIGNMENT"
>>>>>>>>>> column under `--members --verbose` does group the topic
>> partitions
>>>>> by
>>>>>>> topic
>>>>>>>>>> but does not prefix the grouping with the topic name like you're
>>>>>>> proposal:
>>>>>>>>>> "my_topic:0,1;new_topic:0,1". Should we do apply the same format
>>>>> for
>>>>>>> the
>>>>>>>>>> classic group as well?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 2024/11/20 21:32:41 "Lianet M." wrote:
>>>>>>>>>>>> Hello PoAn, just of couple more minor comments:
>>>>>>>>>>>>
>>>>>>>>>>>> LM4. Regarding the new “protocol” field added to
>>>>> MemberDescription.
>>>>>>>>>> Should
>>>>>>>>>>>> we consider reusing the existing GroupProtocol enum instead of
>>>>>>> String?
>>>>>>>>>>>> (It’s the one we use from the consumer side to refer to the
>>>>>> protocol
>>>>>>> in
>>>>>>>>>>>> use, just missing Share I notice).
>>>>>>>>>>>>
>>>>>>>>>>>> LM5. Regarding the change to the output for
>>>>> kafka-consumer-groups,
>>>>>>> the
>>>>>>>>>>>> command shown does not include the —state option, but the
>> output
>>>>>>> shows
>>>>>>>>>>>> state info (state, #members, epochs). I would guess that we
>> want
>>>>> to
>>>>>>>>>> modify
>>>>>>>>>>>> the output only when we describe a group with the —state
>> —verbose
>>>>>>>>>> option,
>>>>>>>>>>>> is my understanding right? If my understanding is right we’re
>>>>> just
>>>>>>>>>> missing
>>>>>>>>>>>> adding the —state in the example, and the KIP does not
>> introduce
>>>>>> any
>>>>>>>>>>>> changes to the —describe —verbose option. (If not, it would
>> mean
>>>>> a
>>>>>>>>>> bigger
>>>>>>>>>>>> change to the output of —describe —verbose which I expect is
>> not
>>>>>> the
>>>>>>>>>>>> intention?)
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>
>>>>>>>>>>>> Lianet
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Nov 20, 2024 at 2:12 AM PoAn Yang <ya...@gmail.com>
>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Chia-Ping,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the review and suggestions.
>>>>>>>>>>>>>
>>>>>>>>>>>>> CI0: Thanks for the reminder. Update validVersions in
>>>>>>>>>>>>> ConsumerGroupDescribeRequest to 0-1.
>>>>>>>>>>>>>
>>>>>>>>>>>>> CI1: Yes, I use `ConsumerGroupMember#useClassicProtocol` to
>>>>> check
>>>>>>>>>> whether
>>>>>>>>>>>>> a member in consumer group uses “classic” or “consumer”
>>>>> protocol.
>>>>>>>>>>>>>
>>>>>>>>>>>>> CI2: Yes, a member in share group always uses “share”
>> protocol.
>>>>>>>>>>>>>
>>>>>>>>>>>>> CI3: Add a table to show meaning of “classic”, “consumer”, and
>>>>>>> “share”
>>>>>>>>>>>>> protocol.
>>>>>>>>>>>>>
>>>>>>>>>>>>> BTW, the vote thread is in
>>>>>>>>>>>>>
>>>>> https://lists.apache.org/thread/rb25tf75tzf4c7jqqldlo5jh9w8chsq6.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Nov 20, 2024, at 11:46 AM, Chia-Ping Tsai <
>>>>>> chia7712@apache.org>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> hi PoAn
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> CI0: We have to bump the version of
>>>>> ConsumerGroupDescribeRequest
>>>>>>> as
>>>>>>>>>>>>> well, so server can distinguish the new and old behavior.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> CI1: the type of new filed is string, so I guess you plan to
>>>>> use
>>>>>>>>>>>>> `ConsumerGroupMember#useClassicProtocol` [0] flag to return
>>>>> either
>>>>>>>>>>>>> "classic" or "consumer", right?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> CI2: `MemberDescription` is used by `ShareGroupDescription`
>>>>> too,
>>>>>> so
>>>>>>>>>> the
>>>>>>>>>>>>> new filed protocol in shared group is always "shared", right?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> CI3: Could you consider adding a table to show the value of
>> the
>>>>>>>>>> protocol
>>>>>>>>>>>>> field in each case? Andrew has a beautiful table in KIP-1043
>>>>> that
>>>>>>>>>> lists all
>>>>>>>>>>>>> possible protocol names.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [0]
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
>> https://github.com/apache/kafka/blob/441a6d0b790f4a17b454caeea7588a6b90fbd9db/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/consumer/ConsumerGroupMember.java#L454
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 2024/11/19 15:45:16 PoAn Yang wrote:
>>>>>>>>>>>>>>> Hi David,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for the reminder.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> DJ4: Add related content to the KIP.
>>>>>>>>>>>>>>> Bump validVersions in ConsumerGroupDescribeResponse to
>> “0-1”.
>>>>>>>>>>>>>>> Add a new field “Protocol” to ConsumerGroupDescribeResponse.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Nov 19, 2024, at 3:57 PM, David Jacot
>>>>>>>>>> <dj...@confluent.io.INVALID>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> DJ4: Could you please also include the required changes to
>>>>> the
>>>>>>>>>>>>>>>> ConsumerGroupDescribed API in the public interface section?
>>>>> We
>>>>>>>>>>>>> basically
>>>>>>>>>>>>>>>> need to bump the version, add the new field, etc.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> David
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang <
>>>>> yangpoan@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi David,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> DJ4: Add a new field protocol to MemberDescription,
>>>>>>>>>>>>>>>>> so the command line tool can show protocol information
>> when
>>>>>>> users
>>>>>>>>>>>>> describe
>>>>>>>>>>>>>>>>> members.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If there is no further suggestion, I will start a vote
>>>>> thread
>>>>>>>>>> today.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Nov 15, 2024, at 11:50 PM, David Jacot
>>>>>>>>>>>>> <dj...@confluent.io.INVALID>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> DJ2: Using "-" sounds good to me.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> DJ3: That seems reasonable to me.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> DJ4: Why not add it right now? I don't want to change the
>>>>>>> output
>>>>>>>>>> of
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> tool too many times.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>> David
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang <
>>>>>> yangpoan@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi David / Andrew,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks for review. Thanks Andrew for picking up
>>>>>>>>>>>>> kafka-share-groups.sh
>>>>>>>>>>>>>>>>>>> implementation.
>>>>>>>>>>>>>>>>>>> I will handle kafka-consumer-groups.sh.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> DJ3: After discussing with @Chia-Ping Tsai, we think
>> that
>>>>>>> using
>>>>>>>>>> new
>>>>>>>>>>>>>>>>> format
>>>>>>>>>>>>>>>>>>> is more clear.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The new format will be like
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
>> <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Using colon(:) to concat topic name and partition IDs.
>>>>>>>>>>>>>>>>>>> Using comma(,) to concat partition IDs within same topic
>>>>>> name.
>>>>>>>>>>>>>>>>>>> Using semicolon(;) to concat topic strings.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> AS3: Fix it with kafka-share-groups.sh. Thanks.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> If there is no further suggestion, I will start a vote
>>>>>> thread
>>>>>>>>>> next
>>>>>>>>>>>>>>>>> Monday.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield <
>>>>>>>>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>>>>>>>> DJ2: I was just going to comment that "-" would be a
>> more
>>>>>>>>>>>>> appropriate
>>>>>>>>>>>>>>>>>>> missing value, but
>>>>>>>>>>>>>>>>>>>> you got there first.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> AS3: The examples for kafka-share-groups.sh include
>>>>>>>>>>>>>>>>>>> kafka-consumer-groups.sh in the
>>>>>>>>>>>>>>>>>>>> command line.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> If this is accepted in time, I'm happy to pick up the
>>>>>>>>>>>>> implementation of
>>>>>>>>>>>>>>>>>>> the share groups
>>>>>>>>>>>>>>>>>>>> part of this if it helps.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> Andrew
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>>>>>> From: Frank Yang <ya...@gmail.com>
>>>>>>>>>>>>>>>>>>>> Sent: 14 November 2024 10:48
>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
>>>>>> kafka-consumer-groups
>>>>>>>>>>>>> command
>>>>>>>>>>>>>>>>>>> line tool to support new consumer group
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi David,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks for the review and suggestion! I would like to
>> get
>>>>>>> this
>>>>>>>>>> in
>>>>>>>>>>>>> AK
>>>>>>>>>>>>>>>>> 4.0
>>>>>>>>>>>>>>>>>>> as well. I will do my best.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> DJ1: Update KIP to put GROUP-EPOCH and
>>>>>>> TARGET-ASSIGNMENT-EPOCH
>>>>>>>>>>>>> before
>>>>>>>>>>>>>>>>>>> #MEMBERS.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> DJ2: I prefer to follow current missing column value
>> “-“.
>>>>>>>>>>>>> (reference <
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
>> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92
>>>>>>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> DJ3: Update KIP to use CURRENT-EPOCH CURRENT-ASSIGNMENT
>>>>>>>>>>>>> TARGET-EPOCH
>>>>>>>>>>>>>>>>>>> TARGET-ASSIGNMENT.
>>>>>>>>>>>>>>>>>>>> Remove GROUP-EPOCH.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> For assignment value, it follows current output
>>>>> (reference
>>>>>> <
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
>> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418
>>>>>>>>>>>>>>>>>> ).
>>>>>>>>>>>>>>>>>>> I think the form of `topicid-partitionid` is more clear.
>>>>>>>>>>>>>>>>>>>> If we would like to use this form, I will update both
>>>>>> output
>>>>>>> in
>>>>>>>>>>>>>>>>>>> kafka-consumer-groups.sh and kafka-share-groups.sh.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> DJ4: It looks like DescribeGroupsResponseData only has
>>>>>>> protocol
>>>>>>>>>>>>> type at
>>>>>>>>>>>>>>>>>>> group level.
>>>>>>>>>>>>>>>>>>>> Both DescribeGroupsResponseData and
>>>>>>>>>>>>> ConsumerGroupDescribeResponseData
>>>>>>>>>>>>>>>>>>> don’t have protocol at member level.
>>>>>>>>>>>>>>>>>>>> Could we use a followup to add it?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> DJ5: Update KIP to put LEADER-EPOCH before
>>>>> CURRENT-OFFSET.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot
>>>>>>>>>>>>> <djacot@confluent.io.INVALID
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP! I have a few minor
>>>>>> comments/suggestions:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> DJ1: In the output of `--describe --verbose`, I would
>>>>>>> suggest
>>>>>>>>>>>>> putting
>>>>>>>>>>>>>>>>>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before
>>>>>>> `#MEMBERS`.
>>>>>>>>>>>>>>>>>>>>> DJ2: Continuing on the above, I assume that we will
>>>>> print
>>>>>>> out
>>>>>>>>>> N/A
>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> fields not supported by classic groups. Is this
>> correct?
>>>>>>>>>>>>>>>>>>>>> DJ3: In the output of `--describe --members
>> --verbose`,
>>>>> I
>>>>>>>>>> wonder
>>>>>>>>>>>>> if we
>>>>>>>>>>>>>>>>>>>>> should use the following order and terms:
>> CURRENT-EPOCH
>>>>>>>>>>>>>>>>>>> CURRENT-ASSIGNMENT
>>>>>>>>>>>>>>>>>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the
>>>>>>> GROUP-EPOCH
>>>>>>>>>>>>>>>>> because
>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>> is already in the group description. The value `(0)`
>>>>> used
>>>>>>> for
>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> assignment is incorrect. Here I suppose that we will
>>>>> print
>>>>>>> out
>>>>>>>>>> the
>>>>>>>>>>>>>>>>> list
>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>> partitions in the form of `topicid-partitionid`.
>>>>>>>>>>>>>>>>>>>>> DJ4: Continuing on the above, I wonder if we should
>> also
>>>>>> add
>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> protocol
>>>>>>>>>>>>>>>>>>>>> used `classic` or `consumer`. For context, it is
>>>>> possible
>>>>>> to
>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>> `classic`
>>>>>>>>>>>>>>>>>>>>> members and `consumer` members in a `consumer` group
>>>>>> during
>>>>>>> an
>>>>>>>>>>>>> online
>>>>>>>>>>>>>>>>>>>>> upgrade from the classic protocol to the consumer
>>>>>> protocol.
>>>>>>>>>> Having
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>> information may be handy for administrators. What do
>> you
>>>>>>> think?
>>>>>>>>>>>>>>>>>>>>> DJ5: In the output of `--describe --offsets
>> --verbose`,
>>>>> I
>>>>>>> would
>>>>>>>>>>>>>>>>> suggest
>>>>>>>>>>>>>>>>>>>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> It would be great if we could get this one in AK 4.0
>> as
>>>>> it
>>>>>>>>>>>>> changes the
>>>>>>>>>>>>>>>>>>>>> output of the command.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> DJ
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <
>>>>>>> yangpoan@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi Sean / Andrew / Lianet,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks for all your review and suggestions.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> AS1, LM1, LM4: Change to add KIP-848 information when
>>>>>> users
>>>>>>>>>> give
>>>>>>>>>>>>>>>>>>> —verbose
>>>>>>>>>>>>>>>>>>>>>> option.
>>>>>>>>>>>>>>>>>>>>>> —describe —verbose: shows group epoch and target
>>>>>> assignment
>>>>>>>>>>>>> epoch.
>>>>>>>>>>>>>>>>>>>>>> —describe —members —verbose: shows above information,
>>>>>>> member
>>>>>>>>>>>>> epoch,
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>> target assignment.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> AS2: Change to use MEMBER-EPOCH to align with KIP-848
>>>>>>>>>> definition.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> LM2: For classic group, it doesn’t have epoch, so I
>> use
>>>>>>>>>> Optional
>>>>>>>>>>>>>>>>>>> fields in
>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupDescription.
>>>>>>>>>>>>>>>>>>>>>> For share group, it relies on KIP-848. It must have
>>>>>> epoch,
>>>>>>> so
>>>>>>>>>> I
>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>> int
>>>>>>>>>>>>>>>>>>>>>> fields in ShareGroupDescription.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> LM3: Remove —state change. We can get group level
>>>>>>> information
>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>> —describe
>>>>>>>>>>>>>>>>>>>>>> —verbose.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> SQ1: Add LEADER-EPOCH when users give —describe
>>>>> —offsets
>>>>>>>>>>>>> —verbose.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <
>>>>>> lianetmr@gmail.com
>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hello Frank, thanks for the KIP! A few comments:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> LM1. I strongly agree with Andrew's suggestion of
>>>>> moving
>>>>>>> this
>>>>>>>>>>>>> into a
>>>>>>>>>>>>>>>>>>>>>>> --verbose option. I expect someone would only
>> attempt
>>>>> to
>>>>>>> make
>>>>>>>>>>>>> sense
>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> epochs while debugging an issue, not while trying to
>>>>>> view
>>>>>>> the
>>>>>>>>>>>>> group
>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>> member's info. (Also member-epoch makes more sense
>> to
>>>>>> me,
>>>>>>>>>> even
>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>> if we
>>>>>>>>>>>>>>>>>>>>>>> end up agreeing in a --verbose). Related to both
>>>>> issues,
>>>>>>> my
>>>>>>>>>>>>> take is
>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>> whoever is close to the protocol will expect
>>>>>> member-epoch,
>>>>>>>>>>>>> whoever
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>> will probably won't even need to see the epochs at
>>>>> all.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> LM2. Why are the epochs defined as Optional (don't
>> we
>>>>>>> expect
>>>>>>>>>> to
>>>>>>>>>>>>>>>>> always
>>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>>> them? with 0 initially, defined on the broker side
>> for
>>>>>> the
>>>>>>>>>> group
>>>>>>>>>>>>>>>>> ones,
>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>> client side for the member)
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> LM3. Why is the KIP including the –state option in
>> the
>>>>>>>>>> proposed
>>>>>>>>>>>>>>>>>>> changes?
>>>>>>>>>>>>>>>>>>>>>> (
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
>>>>>>>>>>>>>>>>>>>>>>> ).
>>>>>>>>>>>>>>>>>>>>>>> I get that the output would change in that example,
>>>>> but
>>>>>>> it’s
>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>> because
>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>> any change to the –state option itself. It's because
>>>>> of
>>>>>>> the
>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>>>>>>>> proposed
>>>>>>>>>>>>>>>>>>>>>>> to the –described (required with the --state), and
>> the
>>>>>>>>>> changes
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> --describe are already explained above (seems
>>>>> confusing,
>>>>>>> got
>>>>>>>>>> me
>>>>>>>>>>>>>>>>>>> looking
>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>> a change to the state filter which seemed
>> unrelated).
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> LM4. Group epoch and target assignment epoch are
>>>>>>> conceptually
>>>>>>>>>>>>> at the
>>>>>>>>>>>>>>>>>>>>>> group
>>>>>>>>>>>>>>>>>>>>>>> level. vs member epoch that lands at a member level.
>>>>> So
>>>>>>>>>> wonder
>>>>>>>>>>>>> if we
>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>> show them accordingly (ex. using the --verbose
>> option)
>>>>>>>>>>>>>>>>>>>>>>> –describe –verbose => shows group epoch and target
>>>>>>> assignment
>>>>>>>>>>>>> epoch
>>>>>>>>>>>>>>>>>>>>>>> –describe –members –verbose => shows member
>>>>>> epoch
>>>>>>>>>>>>> (along
>>>>>>>>>>>>>>>>>>>>>>> with group epoch and target assignment epoch)
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Lianet
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield <
>>>>>>>>>>>>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I have a few comments.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> AS1: It seems to me that these new additions are
>> most
>>>>>>>>>> useful to
>>>>>>>>>>>>>>>>>>> people
>>>>>>>>>>>>>>>>>>>>>>>> trying to understand
>>>>>>>>>>>>>>>>>>>>>>>> the progress of rebalancing in quite some detail.
>> The
>>>>>>>>>>>>> information
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>> really not understandable
>>>>>>>>>>>>>>>>>>>>>>>> for most users who do not have deep knowledge of
>>>>>>>>>> KIP-848/932.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> As a result, I suggest for kafka-share-groups.sh
>> that
>>>>>> you
>>>>>>>>>> add a
>>>>>>>>>>>>>>>>>>>>>> --members
>>>>>>>>>>>>>>>>>>>>>>>> --verbose option
>>>>>>>>>>>>>>>>>>>>>>>> and only include the new information in the output
>>>>> for
>>>>>>> that
>>>>>>>>>>>>> option,
>>>>>>>>>>>>>>>>>>>>>> rather
>>>>>>>>>>>>>>>>>>>>>>>> than changing the
>>>>>>>>>>>>>>>>>>>>>>>> non-verbose --members output.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I also make a similar suggestion for
>>>>>>>>>> kafka-consumer-groups.sh
>>>>>>>>>>>>>>>>>>> --members
>>>>>>>>>>>>>>>>>>>>>>>> and only add the
>>>>>>>>>>>>>>>>>>>>>>>> new information for the --verbose output.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> AS2: I strongly suggest that you use MEMBER-EPOCH
>>>>>>> instead of
>>>>>>>>>>>>>>>>>>>>>>>> CONSUMER-EPOCH.
>>>>>>>>>>>>>>>>>>>>>>>> I think it's more confusing not following the
>>>>>>> terminology of
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> KIPs.
>>>>>>>>>>>>>>>>>>>>>> For
>>>>>>>>>>>>>>>>>>>>>>>> one thing,
>>>>>>>>>>>>>>>>>>>>>>>> we already have "member" in the admin client such
>> as
>>>>>>>>>>>>>>>>>>> MemberDescription.
>>>>>>>>>>>>>>>>>>>>>>>> It's not a
>>>>>>>>>>>>>>>>>>>>>>>> new term.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> Andrew
>>>>>>>>>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>>>>>>>>>> From: PoAn Yang <pa...@apache.org>
>>>>>>>>>>>>>>>>>>>>>>>> Sent: 25 October 2024 13:55
>>>>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
>>>>>>>>>> kafka-consumer-groups
>>>>>>>>>>>>>>>>> command
>>>>>>>>>>>>>>>>>>>>>> line
>>>>>>>>>>>>>>>>>>>>>>>> tool to support new consumer group
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi Lucas,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the review!
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> 1) Yes, I add related change for
>>>>> kafka-share-groups.sh
>>>>>> to
>>>>>>>>>> the
>>>>>>>>>>>>> KIP.
>>>>>>>>>>>>>>>>>>> Could
>>>>>>>>>>>>>>>>>>>>>>>> you take a look? Thanks for the suggestion.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use
>>>>>>> MEMBER-EPOCH
>>>>>>>>>>>>> here,
>>>>>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>>>>>>>>>> confuse what is different between CONSUMER and
>>>>> MEMBER.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> Hi Frank,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> thanks for the KIP!
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> 1) For consistency, should we do the same for
>>>>>>>>>>>>>>>>>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription,
>> etc. ?
>>>>>>> Even
>>>>>>>>>> if
>>>>>>>>>>>>> we do
>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>> implement it right now if the share group
>>>>>> implementation
>>>>>>>>>> may
>>>>>>>>>>>>> still
>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>> incomplete, it may make sense to include it in the
>>>>>> KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH?
>>>>> That
>>>>>>> would
>>>>>>>>>>>>> seem
>>>>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>>>>>>> consistent.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>> Lucas
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <
>>>>>>>>>>>>> yangpoan@gmail.com>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to kick off the discussion of
>>>>> KIP-1099.
>>>>>>> This
>>>>>>>>>> KIP
>>>>>>>>>>>>>>>>>>> enhances
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups tools to include state
>> which
>>>>> is
>>>>>>>>>>>>> introduced
>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>> KIP-848.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups command
>> line
>>>>>>> tool
>>>>>>>>>> to
>>>>>>>>>>>>>>>>> support
>>>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>>>> consumer group - Apache Kafka - Apache Software
>>>>>>> Foundation
>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> cwiki.apache.org
>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> [image: favicon.ico]
>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
>>>
>>>
>>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by Chia-Ping Tsai <ch...@gmail.com>.
hi DJ
> My goal was to have a way to tell the user that his (new) consumer group
may have non-upgraded members yet. I wonder if we could use a boolean to
signal this, e.g. isLegacy. Then, in the CLI we could have a column which
is only displayed when they are non upgraded members. What do you think?
Yes, this is a good approach, but the downside is that the new isLegacy
field in MemberDescription would be irrelevant for classic and shared
groups. This would require thorough documentation to clarify its purpose.
Additionally, we could use a boolean instead of a string for the new field
in the RPC to reduce the request size.
Best,
Chia-Ping
David Jacot <dj...@confluent.io.invalid> 於 2024年11月23日 週六 上午12:41寫道：
> Hi both,
>
> Ah, I understand what you mean now. You're saying that we have two protocol
> fields. One at the group level and one at the member level (the new one).
> The two fields are actually two different things. The former is the name of
> the embedded protocol used in the classic protocol whereas the latter is
> the name of the protocol/api used. This is indeed confusing.
>
> My goal was to have a way to tell the user that his (new) consumer group
> may have non-upgraded members yet. I wonder if we could use a boolean to
> signal this, e.g. isLegacy. Then, in the CLI we could have a column which
> is only displayed when they are non upgraded members. What do you think?
>
> If we cannot find a good way to represent this, we could drop it from this
> KIP too.
>
> Best,
> David
>
> On Fri, Nov 22, 2024 at 5:12 PM PoAn Yang <ya...@gmail.com> wrote:
>
> > Hi Chia-Ping / David,
> >
> > Thanks for the review and suggestions.
> >
> > >> I don't fully follow your comment so please excuse me if I say
> something
> > >> wrong.
> >
> > I think Chia-Ping wants to say: originally, the default protocol in
> > classic group is “consumer”.
> > After we have a new consumer group, the protocol of a classic member in
> > consumer group is “classic”.
> > The MemberDescription is used by both ClassicGroupDescription and
> > ConsumerGroupDescription.
> > If we add the protocol field to MemberDescription, the protocol of a
> > classic member in classic group is “consumer”, but it’s “classic” in
> > consumer group.
> > To avoid misleading word, we can add a new collection of member protocols
> > to ConsumerGroupDescription only.
> > The original goal is to show member protocol during consumer migration,
> so
> > we don’t need to add this information to ClassicGroupDescription.
> >
> > Please correct me if I misunderstood anything.
> >
> > Thanks,
> > PoAn
> >
> > > On Nov 22, 2024, at 7:00 PM, Chia-Ping Tsai <ch...@gmail.com>
> wrote:
> > >
> > > hi DJ
> > >
> > >> How about we use another Jira to add “SHARE” to GroupProtocol and
> change
> > > return type of ModernGroup#protocolType?
> > >
> > > This is not my comment ... sorry for the incorrect quote :(
> > >
> > >> A member in a classic group should report "classic".
> > >
> > > I completely agree. However, in this case,
> > ClassicGroupDescription#protocol
> > > returns 'consumer', while ClassicGroupDescription#members[x].protocol
> > > returns 'classic'. This inconsistency feels a bit strange to me.
> > >
> > > Best,
> > > Chia-Ping
> > >
> > >
> > > David Jacot <dj...@confluent.io.invalid> 於 2024年11月22日 週五 下午6:51寫道：
> > >
> > >> Hi Chia,
> > >>
> > >>>> LM4: I agree it’s better to use a type like GroupProtocol enum, but
> I
> > >> would like to align the format with kafka-groups.sh.
> > >> How about we use another Jira to add “SHARE” to GroupProtocol and
> change
> > >> return type of ModernGroup#protocolType?
> > >>
> > >> My understanding is that `GroupProtocol` is used to drive the config
> of
> > the
> > >> `Consumer`. Hence adding `Share` seems wrong to me. Or are you
> > referring to
> > >> another GroupProtocol enum? Perhaps GroupType enum?
> > >>
> > >>> In fact, I'm wondering if we should even add 'protocol' to
> > >> MemberDescription,
> > >> since the field has different values between classic and consumer
> > groups—it
> > >> shows 'consumer' in classic groups, whereas it shows 'classic' in
> > consumer
> > >> groups. Yes, it seems like a typo, but it's true. This inconsistency
> > feels
> > >> confusing to me. For example, users call Admin#describeConsumerGroups
> > and
> > >> see that a member has the 'consumer' protocol when it's in a classic
> > group.
> > >> Then, if the group is upgraded to a consumer group, calling
> > >> Admin#describeConsumerGroups again shows the protocol as 'classic' for
> > the
> > >> same member.
> > >>
> > >> I don't fully follow your comment so please excuse me if I say
> something
> > >> wrong.
> > >>
> > >> I think that it should work as follows:
> > >> * A member in a classic group should report "classic".
> > >> * A member using the consumer protocol in a consumer group should
> report
> > >> "consumer".
> > >> * A member using the classic protocol in a consumer group should
> report
> > >> "classic".
> > >> * A member in a share group should report "share".
> > >>
> > >>> If all we want to do is distinguish classic members in the
> > >> ConsumerGroupDescription, then we shouldn't modify the common
> > >> MemberDescription structure. Instead, we could add a collection to
> > >> ConsumerGroupDescription to trace members using the classic protocol,
> > which
> > >> would allow us to enrich the output of command-line tools.
> > >>
> > >> Personally, I believe that using different collections per protocol
> > types
> > >> is not necessary. A field in the member is more than enough.
> > >>
> > >> Best,
> > >> DJ
> > >>
> > >> On Fri, Nov 22, 2024 at 12:01 PM Chia-Ping Tsai <ch...@gmail.com>
> > >> wrote:
> > >>
> > >>> hi PoAn
> > >>>
> > >>>> LM4: I agree it’s better to use a type like GroupProtocol enum, but
> I
> > >>> would like to align the format with kafka-groups.sh.
> > >>> How about we use another Jira to add “SHARE” to GroupProtocol and
> > change
> > >>> return type of ModernGroup#protocolType?
> > >>>
> > >>> MemberDescription is also used by ClassicGroupDescription, and in
> > classic
> > >>> groups, the protocol can be an arbitrary string, so we can't use an
> > enum.
> > >>>
> > >>> In fact, I'm wondering if we should even add 'protocol' to
> > >>> MemberDescription,
> > >>> since the field has different values between classic and consumer
> > >> groups—it
> > >>> shows 'consumer' in classic groups, whereas it shows 'classic' in
> > >> consumer
> > >>> groups. Yes, it seems like a typo, but it's true. This inconsistency
> > >> feels
> > >>> confusing to me. For example, users call Admin#describeConsumerGroups
> > and
> > >>> see that a member has the 'consumer' protocol when it's in a classic
> > >> group.
> > >>> Then, if the group is upgraded to a consumer group, calling
> > >>> Admin#describeConsumerGroups again shows the protocol as 'classic'
> for
> > >> the
> > >>> same member.
> > >>>
> > >>> If all we want to do is distinguish classic members in the
> > >>> ConsumerGroupDescription, then we shouldn't modify the common
> > >>> MemberDescription structure. Instead, we could add a collection to
> > >>> ConsumerGroupDescription to trace members using the classic protocol,
> > >> which
> > >>> would allow us to enrich the output of command-line tools.
> > >>> Best,
> > >>> Chia-Ping
> > >>>
> > >>> PoAn Yang <ya...@gmail.com> 於 2024年11月22日 週五 下午3:46寫道：
> > >>>
> > >>>> Hi Lianet / Jeff,
> > >>>>
> > >>>> Thanks for the review.
> > >>>>
> > >>>> LM5: The kafka-share-groups.sh also uses “—describe” to show offsets
> > >>>> information.
> > >>>> Change to use “—describe —state —verbose” to show group level
> > >> information
> > >>>> in kafka-share-groups.sh.
> > >>>>
> > >>>> LM6: Update the example.
> > >>>>
> > >>>> LM7: Add a description to mention “—verbose” is a new option in
> > >>>> ShareGroupCommandOptions.
> > >>>>
> > >>>> JK2: Yes, add the statement about format change for ASSIGNMENT
> value.
> > >>>>
> > >>>> Thanks,
> > >>>> PoAn
> > >>>>
> > >>>>> On Nov 22, 2024, at 7:35 AM, Jeff Kim <je...@apache.org>
> wrote:
> > >>>>>
> > >>>>> Hi PoAn,
> > >>>>>
> > >>>>> JK2: Can we state in the KIP that we will reformat the existing
> > >>>>> classic group's `--members --verbose` as well, specifically the
> > >>>>> "ASSIGNMENT" column to include the topic name?
> > >>>>>
> > >>>>> Thanks!
> > >>>>>
> > >>>>> On 2024/11/21 14:33:37 "Lianet M." wrote:
> > >>>>>> Thanks for the updates PoAn!
> > >>>>>>
> > >>>>>> LM6. Just a nit, under the --describe --state --verbose the header
> > >> is
> > >>>> fine
> > >>>>>> but the example is still missing the --state argument.
> > >>>>>>
> > >>>>>> LM7. Under the Proposed changes for kafka-share-groups.sh, should
> we
> > >>>>>> clearly mention that the KIP adds a new --verbose option? I think
> > >> it's
> > >>>>>> confusing because it's presented as if the option already exists
> and
> > >>>> we're
> > >>>>>> only changing the output (which is the case for
> > >> kafka-consumer-groups
> > >>>> but
> > >>>>>> not for kafka-share-groups)
> > >>>>>>
> > >>>>>> That's all on my side. Thanks!
> > >>>>>>
> > >>>>>> Lianet
> > >>>>>>
> > >>>>>> On Wed, Nov 20, 2024 at 11:12 PM PoAn Yang <ya...@gmail.com>
> > >>> wrote:
> > >>>>>>
> > >>>>>>> Hi Lianet / Jeff,
> > >>>>>>>
> > >>>>>>> Thanks for review and suggestions.
> > >>>>>>>
> > >>>>>>> LM4: At group level, the protocol field is from
> > >>>> ModernGroup#protocolType.
> > >>>>>>> It’s a string.
> > >>>>>>> The value is defined in different places like
> > >>> ShareGroup#PROTOCOL_TYPE
> > >>>> and
> > >>>>>>> ConsumerProtocol#PROTOCOL_TYPE.
> > >>>>>>> In KIP-1043 <
> > >>>>>>>
> > >>>>
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=305171038#KIP1043:Administrationofgroups-kafka-groups.sh
> > >>>>> ,
> > >>>>>>> the kafka-groups.sh also shows protocol like “classic” /
> > >> “consumer” /
> > >>>>>>> “share”.
> > >>>>>>> I agree it’s better to use a type like GroupProtocol enum, but I
> > >>> would
> > >>>>>>> like to align the format with kafka-groups.sh.
> > >>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and
> > >>>> change
> > >>>>>>> return type of ModernGroup#protocolType?
> > >>>>>>>
> > >>>>>>> LM5 / JK1: Sorry, I misunderstood LM3, so I deleted —state in the
> > >>>> title.
> > >>>>>>> You’re correct. I would like to use `—describe —state —verbose`
> to
> > >>> show
> > >>>>>>> group level information.
> > >>>>>>> For `—describe —verbose`, it will show output as same as
> `—describe
> > >>>>>>> —offsets —verbose`,
> > >>>>>>> because `—describe` also shows output as same as `—describe
> > >>> —offsets`.
> > >>>>>>>
> > >>>>>>> JK2: Yes, I only use consumer group in the sample output, but the
> > >>>> classic
> > >>>>>>> group will use the same format as well.
> > >>>>>>>
> > >>>>>>> Thanks,
> > >>>>>>> PoAn
> > >>>>>>>
> > >>>>>>>> On Nov 21, 2024, at 8:28 AM, Jeff Kim <je...@apache.org>
> > >> wrote:
> > >>>>>>>>
> > >>>>>>>> Hi PoAn,
> > >>>>>>>>
> > >>>>>>>> Thanks for the KIP. I have some questions:
> > >>>>>>>>
> > >>>>>>>> JK1: (Realized this is similar to LM5) On `--describe --verbose`
> > >>>>>>> proposed changes - doesn't `--describe $group` default to
> printing
> > >>> the
> > >>>>>>> offsets? Perhaps you're referring to the `--state` argument?
> Also,
> > >>>> would
> > >>>>>>> that mean the default `--describe $group --verbose` command would
> > >> not
> > >>>> print
> > >>>>>>> the added field to `--offsets verbose` (leader-epoch) or would
> it?
> > >>>>>>>>
> > >>>>>>>> JK2: Looking at ConsumerGroupCommand.java, the existing
> > >> "ASSIGNMENT"
> > >>>>>>> column under `--members --verbose` does group the topic
> partitions
> > >> by
> > >>>> topic
> > >>>>>>> but does not prefix the grouping with the topic name like you're
> > >>>> proposal:
> > >>>>>>> "my_topic:0,1;new_topic:0,1". Should we do apply the same format
> > >> for
> > >>>> the
> > >>>>>>> classic group as well?
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On 2024/11/20 21:32:41 "Lianet M." wrote:
> > >>>>>>>>> Hello PoAn, just of couple more minor comments:
> > >>>>>>>>>
> > >>>>>>>>> LM4. Regarding the new “protocol” field added to
> > >> MemberDescription.
> > >>>>>>> Should
> > >>>>>>>>> we consider reusing the existing GroupProtocol enum instead of
> > >>>> String?
> > >>>>>>>>> (It’s the one we use from the consumer side to refer to the
> > >>> protocol
> > >>>> in
> > >>>>>>>>> use, just missing Share I notice).
> > >>>>>>>>>
> > >>>>>>>>> LM5. Regarding the change to the output for
> > >> kafka-consumer-groups,
> > >>>> the
> > >>>>>>>>> command shown does not include the —state option, but the
> output
> > >>>> shows
> > >>>>>>>>> state info (state, #members, epochs). I would guess that we
> want
> > >> to
> > >>>>>>> modify
> > >>>>>>>>> the output only when we describe a group with the —state
> —verbose
> > >>>>>>> option,
> > >>>>>>>>> is my understanding right? If my understanding is right we’re
> > >> just
> > >>>>>>> missing
> > >>>>>>>>> adding the —state in the example, and the KIP does not
> introduce
> > >>> any
> > >>>>>>>>> changes to the —describe —verbose option. (If not, it would
> mean
> > >> a
> > >>>>>>> bigger
> > >>>>>>>>> change to the output of —describe —verbose which I expect is
> not
> > >>> the
> > >>>>>>>>> intention?)
> > >>>>>>>>>
> > >>>>>>>>> Thanks!
> > >>>>>>>>>
> > >>>>>>>>> Lianet
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> On Wed, Nov 20, 2024 at 2:12 AM PoAn Yang <ya...@gmail.com>
> > >>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Hi Chia-Ping,
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks for the review and suggestions.
> > >>>>>>>>>>
> > >>>>>>>>>> CI0: Thanks for the reminder. Update validVersions in
> > >>>>>>>>>> ConsumerGroupDescribeRequest to 0-1.
> > >>>>>>>>>>
> > >>>>>>>>>> CI1: Yes, I use `ConsumerGroupMember#useClassicProtocol` to
> > >> check
> > >>>>>>> whether
> > >>>>>>>>>> a member in consumer group uses “classic” or “consumer”
> > >> protocol.
> > >>>>>>>>>>
> > >>>>>>>>>> CI2: Yes, a member in share group always uses “share”
> protocol.
> > >>>>>>>>>>
> > >>>>>>>>>> CI3: Add a table to show meaning of “classic”, “consumer”, and
> > >>>> “share”
> > >>>>>>>>>> protocol.
> > >>>>>>>>>>
> > >>>>>>>>>> BTW, the vote thread is in
> > >>>>>>>>>>
> > >> https://lists.apache.org/thread/rb25tf75tzf4c7jqqldlo5jh9w8chsq6.
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks,
> > >>>>>>>>>> PoAn
> > >>>>>>>>>>
> > >>>>>>>>>>> On Nov 20, 2024, at 11:46 AM, Chia-Ping Tsai <
> > >>> chia7712@apache.org>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> hi PoAn
> > >>>>>>>>>>>
> > >>>>>>>>>>> CI0: We have to bump the version of
> > >> ConsumerGroupDescribeRequest
> > >>>> as
> > >>>>>>>>>> well, so server can distinguish the new and old behavior.
> > >>>>>>>>>>>
> > >>>>>>>>>>> CI1: the type of new filed is string, so I guess you plan to
> > >> use
> > >>>>>>>>>> `ConsumerGroupMember#useClassicProtocol` [0] flag to return
> > >> either
> > >>>>>>>>>> "classic" or "consumer", right?
> > >>>>>>>>>>>
> > >>>>>>>>>>> CI2: `MemberDescription` is used by `ShareGroupDescription`
> > >> too,
> > >>> so
> > >>>>>>> the
> > >>>>>>>>>> new filed protocol in shared group is always "shared", right?
> > >>>>>>>>>>>
> > >>>>>>>>>>> CI3: Could you consider adding a table to show the value of
> the
> > >>>>>>> protocol
> > >>>>>>>>>> field in each case? Andrew has a beautiful table in KIP-1043
> > >> that
> > >>>>>>> lists all
> > >>>>>>>>>> possible protocol names.
> > >>>>>>>>>>>
> > >>>>>>>>>>> [0]
> > >>>>>>>>>>
> > >>>>>>>
> > >>>>
> > >>>
> > >>
> >
> https://github.com/apache/kafka/blob/441a6d0b790f4a17b454caeea7588a6b90fbd9db/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/consumer/ConsumerGroupMember.java#L454
> > >>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Best,
> > >>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>
> > >>>>>>>>>>> On 2024/11/19 15:45:16 PoAn Yang wrote:
> > >>>>>>>>>>>> Hi David,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks for the reminder.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> DJ4: Add related content to the KIP.
> > >>>>>>>>>>>> Bump validVersions in ConsumerGroupDescribeResponse to
> “0-1”.
> > >>>>>>>>>>>> Add a new field “Protocol” to ConsumerGroupDescribeResponse.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>> PoAn
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> On Nov 19, 2024, at 3:57 PM, David Jacot
> > >>>>>>> <dj...@confluent.io.INVALID>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi PoAn,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> DJ4: Could you please also include the required changes to
> > >> the
> > >>>>>>>>>>>>> ConsumerGroupDescribed API in the public interface section?
> > >> We
> > >>>>>>>>>> basically
> > >>>>>>>>>>>>> need to bump the version, add the new field, etc.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>> David
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang <
> > >> yangpoan@gmail.com>
> > >>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hi David,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> DJ4: Add a new field protocol to MemberDescription,
> > >>>>>>>>>>>>>> so the command line tool can show protocol information
> when
> > >>>> users
> > >>>>>>>>>> describe
> > >>>>>>>>>>>>>> members.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> If there is no further suggestion, I will start a vote
> > >> thread
> > >>>>>>> today.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>> PoAn
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Nov 15, 2024, at 11:50 PM, David Jacot
> > >>>>>>>>>> <dj...@confluent.io.INVALID>
> > >>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Hi,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> DJ2: Using "-" sounds good to me.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> DJ3: That seems reasonable to me.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> DJ4: Why not add it right now? I don't want to change the
> > >>>> output
> > >>>>>>> of
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>>> tool too many times.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>> David
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang <
> > >>> yangpoan@gmail.com>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Hi David / Andrew,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thanks for review. Thanks Andrew for picking up
> > >>>>>>>>>> kafka-share-groups.sh
> > >>>>>>>>>>>>>>>> implementation.
> > >>>>>>>>>>>>>>>> I will handle kafka-consumer-groups.sh.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> DJ3: After discussing with @Chia-Ping Tsai, we think
> that
> > >>>> using
> > >>>>>>> new
> > >>>>>>>>>>>>>> format
> > >>>>>>>>>>>>>>>> is more clear.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> The new format will be like
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>
> > >>>>
> > >>>
> > >>
> >
> <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Using colon(:) to concat topic name and partition IDs.
> > >>>>>>>>>>>>>>>> Using comma(,) to concat partition IDs within same topic
> > >>> name.
> > >>>>>>>>>>>>>>>> Using semicolon(;) to concat topic strings.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> AS3: Fix it with kafka-share-groups.sh. Thanks.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> If there is no further suggestion, I will start a vote
> > >>> thread
> > >>>>>>> next
> > >>>>>>>>>>>>>> Monday.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>> PoAn
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield <
> > >>>>>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hi PoAn,
> > >>>>>>>>>>>>>>>>> DJ2: I was just going to comment that "-" would be a
> more
> > >>>>>>>>>> appropriate
> > >>>>>>>>>>>>>>>> missing value, but
> > >>>>>>>>>>>>>>>>> you got there first.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> AS3: The examples for kafka-share-groups.sh include
> > >>>>>>>>>>>>>>>> kafka-consumer-groups.sh in the
> > >>>>>>>>>>>>>>>>> command line.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> If this is accepted in time, I'm happy to pick up the
> > >>>>>>>>>> implementation of
> > >>>>>>>>>>>>>>>> the share groups
> > >>>>>>>>>>>>>>>>> part of this if it helps.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>> Andrew
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> ________________________________________
> > >>>>>>>>>>>>>>>>> From: Frank Yang <ya...@gmail.com>
> > >>>>>>>>>>>>>>>>> Sent: 14 November 2024 10:48
> > >>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
> > >>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
> > >>> kafka-consumer-groups
> > >>>>>>>>>> command
> > >>>>>>>>>>>>>>>> line tool to support new consumer group
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hi David,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Thanks for the review and suggestion! I would like to
> get
> > >>>> this
> > >>>>>>> in
> > >>>>>>>>>> AK
> > >>>>>>>>>>>>>> 4.0
> > >>>>>>>>>>>>>>>> as well. I will do my best.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> DJ1: Update KIP to put GROUP-EPOCH and
> > >>>> TARGET-ASSIGNMENT-EPOCH
> > >>>>>>>>>> before
> > >>>>>>>>>>>>>>>> #MEMBERS.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> DJ2: I prefer to follow current missing column value
> “-“.
> > >>>>>>>>>> (reference <
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>
> > >>>>
> > >>>
> > >>
> >
> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92
> > >>>>>>>>>>>>>>>>> )
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> DJ3: Update KIP to use CURRENT-EPOCH CURRENT-ASSIGNMENT
> > >>>>>>>>>> TARGET-EPOCH
> > >>>>>>>>>>>>>>>> TARGET-ASSIGNMENT.
> > >>>>>>>>>>>>>>>>> Remove GROUP-EPOCH.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> For assignment value, it follows current output
> > >> (reference
> > >>> <
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>
> > >>>>
> > >>>
> > >>
> >
> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418
> > >>>>>>>>>>>>>>> ).
> > >>>>>>>>>>>>>>>> I think the form of `topicid-partitionid` is more clear.
> > >>>>>>>>>>>>>>>>> If we would like to use this form, I will update both
> > >>> output
> > >>>> in
> > >>>>>>>>>>>>>>>> kafka-consumer-groups.sh and kafka-share-groups.sh.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> DJ4: It looks like DescribeGroupsResponseData only has
> > >>>> protocol
> > >>>>>>>>>> type at
> > >>>>>>>>>>>>>>>> group level.
> > >>>>>>>>>>>>>>>>> Both DescribeGroupsResponseData and
> > >>>>>>>>>> ConsumerGroupDescribeResponseData
> > >>>>>>>>>>>>>>>> don’t have protocol at member level.
> > >>>>>>>>>>>>>>>>> Could we use a followup to add it?
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> DJ5: Update KIP to put LEADER-EPOCH before
> > >> CURRENT-OFFSET.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>> PoAn
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot
> > >>>>>>>>>> <djacot@confluent.io.INVALID
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Hi PoAn,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Thanks for the KIP! I have a few minor
> > >>> comments/suggestions:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> DJ1: In the output of `--describe --verbose`, I would
> > >>>> suggest
> > >>>>>>>>>> putting
> > >>>>>>>>>>>>>>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before
> > >>>> `#MEMBERS`.
> > >>>>>>>>>>>>>>>>>> DJ2: Continuing on the above, I assume that we will
> > >> print
> > >>>> out
> > >>>>>>> N/A
> > >>>>>>>>>> for
> > >>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> fields not supported by classic groups. Is this
> correct?
> > >>>>>>>>>>>>>>>>>> DJ3: In the output of `--describe --members
> --verbose`,
> > >> I
> > >>>>>>> wonder
> > >>>>>>>>>> if we
> > >>>>>>>>>>>>>>>>>> should use the following order and terms:
> CURRENT-EPOCH
> > >>>>>>>>>>>>>>>> CURRENT-ASSIGNMENT
> > >>>>>>>>>>>>>>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the
> > >>>> GROUP-EPOCH
> > >>>>>>>>>>>>>> because
> > >>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>> is already in the group description. The value `(0)`
> > >> used
> > >>>> for
> > >>>>>>> the
> > >>>>>>>>>>>>>>>>>> assignment is incorrect. Here I suppose that we will
> > >> print
> > >>>> out
> > >>>>>>> the
> > >>>>>>>>>>>>>> list
> > >>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>> partitions in the form of `topicid-partitionid`.
> > >>>>>>>>>>>>>>>>>> DJ4: Continuing on the above, I wonder if we should
> also
> > >>> add
> > >>>>>>> the
> > >>>>>>>>>>>>>>>> protocol
> > >>>>>>>>>>>>>>>>>> used `classic` or `consumer`. For context, it is
> > >> possible
> > >>> to
> > >>>>>>> have
> > >>>>>>>>>>>>>>>> `classic`
> > >>>>>>>>>>>>>>>>>> members and `consumer` members in a `consumer` group
> > >>> during
> > >>>> an
> > >>>>>>>>>> online
> > >>>>>>>>>>>>>>>>>> upgrade from the classic protocol to the consumer
> > >>> protocol.
> > >>>>>>> Having
> > >>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>> information may be handy for administrators. What do
> you
> > >>>> think?
> > >>>>>>>>>>>>>>>>>> DJ5: In the output of `--describe --offsets
> --verbose`,
> > >> I
> > >>>> would
> > >>>>>>>>>>>>>> suggest
> > >>>>>>>>>>>>>>>>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> It would be great if we could get this one in AK 4.0
> as
> > >> it
> > >>>>>>>>>> changes the
> > >>>>>>>>>>>>>>>>>> output of the command.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>> DJ
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <
> > >>>> yangpoan@gmail.com>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Hi Sean / Andrew / Lianet,
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Thanks for all your review and suggestions.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> AS1, LM1, LM4: Change to add KIP-848 information when
> > >>> users
> > >>>>>>> give
> > >>>>>>>>>>>>>>>> —verbose
> > >>>>>>>>>>>>>>>>>>> option.
> > >>>>>>>>>>>>>>>>>>> —describe —verbose: shows group epoch and target
> > >>> assignment
> > >>>>>>>>>> epoch.
> > >>>>>>>>>>>>>>>>>>> —describe —members —verbose: shows above information,
> > >>>> member
> > >>>>>>>>>> epoch,
> > >>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>> target assignment.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> AS2: Change to use MEMBER-EPOCH to align with KIP-848
> > >>>>>>> definition.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> LM2: For classic group, it doesn’t have epoch, so I
> use
> > >>>>>>> Optional
> > >>>>>>>>>>>>>>>> fields in
> > >>>>>>>>>>>>>>>>>>> ConsumerGroupDescription.
> > >>>>>>>>>>>>>>>>>>> For share group, it relies on KIP-848. It must have
> > >>> epoch,
> > >>>> so
> > >>>>>>> I
> > >>>>>>>>>> use
> > >>>>>>>>>>>>>> int
> > >>>>>>>>>>>>>>>>>>> fields in ShareGroupDescription.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> LM3: Remove —state change. We can get group level
> > >>>> information
> > >>>>>>> by
> > >>>>>>>>>>>>>>>> —describe
> > >>>>>>>>>>>>>>>>>>> —verbose.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> SQ1: Add LEADER-EPOCH when users give —describe
> > >> —offsets
> > >>>>>>>>>> —verbose.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Thanks.
> > >>>>>>>>>>>>>>>>>>> PoAn
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <
> > >>> lianetmr@gmail.com
> > >>>>>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Hello Frank, thanks for the KIP! A few comments:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> LM1. I strongly agree with Andrew's suggestion of
> > >> moving
> > >>>> this
> > >>>>>>>>>> into a
> > >>>>>>>>>>>>>>>>>>>> --verbose option. I expect someone would only
> attempt
> > >> to
> > >>>> make
> > >>>>>>>>>> sense
> > >>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>> epochs while debugging an issue, not while trying to
> > >>> view
> > >>>> the
> > >>>>>>>>>> group
> > >>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>> member's info. (Also member-epoch makes more sense
> to
> > >>> me,
> > >>>>>>> even
> > >>>>>>>>>> more
> > >>>>>>>>>>>>>>>> if we
> > >>>>>>>>>>>>>>>>>>>> end up agreeing in a --verbose). Related to both
> > >> issues,
> > >>>> my
> > >>>>>>>>>> take is
> > >>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>> whoever is close to the protocol will expect
> > >>> member-epoch,
> > >>>>>>>>>> whoever
> > >>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>>> will probably won't even need to see the epochs at
> > >> all.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> LM2. Why are the epochs defined as Optional (don't
> we
> > >>>> expect
> > >>>>>>> to
> > >>>>>>>>>>>>>> always
> > >>>>>>>>>>>>>>>>>>> have
> > >>>>>>>>>>>>>>>>>>>> them? with 0 initially, defined on the broker side
> for
> > >>> the
> > >>>>>>> group
> > >>>>>>>>>>>>>> ones,
> > >>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>> client side for the member)
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> LM3. Why is the KIP including the –state option in
> the
> > >>>>>>> proposed
> > >>>>>>>>>>>>>>>> changes?
> > >>>>>>>>>>>>>>>>>>> (
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>
> > >>>>
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
> > >>>>>>>>>>>>>>>>>>>> ).
> > >>>>>>>>>>>>>>>>>>>> I get that the output would change in that example,
> > >> but
> > >>>> it’s
> > >>>>>>> not
> > >>>>>>>>>>>>>>>> because
> > >>>>>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>> any change to the –state option itself. It's because
> > >> of
> > >>>> the
> > >>>>>>>>>> change
> > >>>>>>>>>>>>>>>>>>> proposed
> > >>>>>>>>>>>>>>>>>>>> to the –described (required with the --state), and
> the
> > >>>>>>> changes
> > >>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>> --describe are already explained above (seems
> > >> confusing,
> > >>>> got
> > >>>>>>> me
> > >>>>>>>>>>>>>>>> looking
> > >>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>> a change to the state filter which seemed
> unrelated).
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> LM4. Group epoch and target assignment epoch are
> > >>>> conceptually
> > >>>>>>>>>> at the
> > >>>>>>>>>>>>>>>>>>> group
> > >>>>>>>>>>>>>>>>>>>> level. vs member epoch that lands at a member level.
> > >> So
> > >>>>>>> wonder
> > >>>>>>>>>> if we
> > >>>>>>>>>>>>>>>>>>> should
> > >>>>>>>>>>>>>>>>>>>> show them accordingly (ex. using the --verbose
> option)
> > >>>>>>>>>>>>>>>>>>>> –describe –verbose => shows group epoch and target
> > >>>> assignment
> > >>>>>>>>>> epoch
> > >>>>>>>>>>>>>>>>>>>> –describe –members –verbose => shows member
> > >>> epoch
> > >>>>>>>>>> (along
> > >>>>>>>>>>>>>>>>>>>> with group epoch and target assignment epoch)
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Thanks!
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Lianet
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield <
> > >>>>>>>>>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Hi PoAn,
> > >>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I have a few comments.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> AS1: It seems to me that these new additions are
> most
> > >>>>>>> useful to
> > >>>>>>>>>>>>>>>> people
> > >>>>>>>>>>>>>>>>>>>>> trying to understand
> > >>>>>>>>>>>>>>>>>>>>> the progress of rebalancing in quite some detail.
> The
> > >>>>>>>>>> information
> > >>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>> really not understandable
> > >>>>>>>>>>>>>>>>>>>>> for most users who do not have deep knowledge of
> > >>>>>>> KIP-848/932.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> As a result, I suggest for kafka-share-groups.sh
> that
> > >>> you
> > >>>>>>> add a
> > >>>>>>>>>>>>>>>>>>> --members
> > >>>>>>>>>>>>>>>>>>>>> --verbose option
> > >>>>>>>>>>>>>>>>>>>>> and only include the new information in the output
> > >> for
> > >>>> that
> > >>>>>>>>>> option,
> > >>>>>>>>>>>>>>>>>>> rather
> > >>>>>>>>>>>>>>>>>>>>> than changing the
> > >>>>>>>>>>>>>>>>>>>>> non-verbose --members output.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> I also make a similar suggestion for
> > >>>>>>> kafka-consumer-groups.sh
> > >>>>>>>>>>>>>>>> --members
> > >>>>>>>>>>>>>>>>>>>>> and only add the
> > >>>>>>>>>>>>>>>>>>>>> new information for the --verbose output.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> AS2: I strongly suggest that you use MEMBER-EPOCH
> > >>>> instead of
> > >>>>>>>>>>>>>>>>>>>>> CONSUMER-EPOCH.
> > >>>>>>>>>>>>>>>>>>>>> I think it's more confusing not following the
> > >>>> terminology of
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>>>> KIPs.
> > >>>>>>>>>>>>>>>>>>> For
> > >>>>>>>>>>>>>>>>>>>>> one thing,
> > >>>>>>>>>>>>>>>>>>>>> we already have "member" in the admin client such
> as
> > >>>>>>>>>>>>>>>> MemberDescription.
> > >>>>>>>>>>>>>>>>>>>>> It's not a
> > >>>>>>>>>>>>>>>>>>>>> new term.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>> Andrew
> > >>>>>>>>>>>>>>>>>>>>> ________________________________________
> > >>>>>>>>>>>>>>>>>>>>> From: PoAn Yang <pa...@apache.org>
> > >>>>>>>>>>>>>>>>>>>>> Sent: 25 October 2024 13:55
> > >>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
> > >>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
> > >>>>>>> kafka-consumer-groups
> > >>>>>>>>>>>>>> command
> > >>>>>>>>>>>>>>>>>>> line
> > >>>>>>>>>>>>>>>>>>>>> tool to support new consumer group
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Hi Lucas,
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Thanks for the review!
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> 1) Yes, I add related change for
> > >> kafka-share-groups.sh
> > >>> to
> > >>>>>>> the
> > >>>>>>>>>> KIP.
> > >>>>>>>>>>>>>>>> Could
> > >>>>>>>>>>>>>>>>>>>>> you take a look? Thanks for the suggestion.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use
> > >>>> MEMBER-EPOCH
> > >>>>>>>>>> here,
> > >>>>>>>>>>>>>>>> users
> > >>>>>>>>>>>>>>>>>>> may
> > >>>>>>>>>>>>>>>>>>>>> confuse what is different between CONSUMER and
> > >> MEMBER.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>> PoAn
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
> > >>>>>>>>>>>>>>>>>>>>>> Hi Frank,
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> thanks for the KIP!
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> 1) For consistency, should we do the same for
> > >>>>>>>>>>>>>>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription,
> etc. ?
> > >>>> Even
> > >>>>>>> if
> > >>>>>>>>>> we do
> > >>>>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>>>>> implement it right now if the share group
> > >>> implementation
> > >>>>>>> may
> > >>>>>>>>>> still
> > >>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>> incomplete, it may make sense to include it in the
> > >>> KIP.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH?
> > >> That
> > >>>> would
> > >>>>>>>>>> seem
> > >>>>>>>>>>>>>>>> more
> > >>>>>>>>>>>>>>>>>>>>>> consistent.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>>>>>> Lucas
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <
> > >>>>>>>>>> yangpoan@gmail.com>
> > >>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> I would like to kick off the discussion of
> > >> KIP-1099.
> > >>>> This
> > >>>>>>> KIP
> > >>>>>>>>>>>>>>>> enhances
> > >>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups tools to include state
> which
> > >> is
> > >>>>>>>>>> introduced
> > >>>>>>>>>>>>>> by
> > >>>>>>>>>>>>>>>>>>>>> KIP-848.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups command
> line
> > >>>> tool
> > >>>>>>> to
> > >>>>>>>>>>>>>> support
> > >>>>>>>>>>>>>>>>>>> new
> > >>>>>>>>>>>>>>>>>>>>>>> consumer group - Apache Kafka - Apache Software
> > >>>> Foundation
> > >>>>>>>>>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>
> > >>>>
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> cwiki.apache.org
> > >>>>>>>>>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>
> > >>>>
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> [image: favicon.ico]
> > >>>>>>>>>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>
> > >>>>
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>
> > >>>>
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Thank you,
> > >>>>>>>>>>>>>>>>>>>>>>> PoAn
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> >
> >
> >
>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by David Jacot <dj...@confluent.io.INVALID>.
Hi both,
Ah, I understand what you mean now. You're saying that we have two protocol
fields. One at the group level and one at the member level (the new one).
The two fields are actually two different things. The former is the name of
the embedded protocol used in the classic protocol whereas the latter is
the name of the protocol/api used. This is indeed confusing.
My goal was to have a way to tell the user that his (new) consumer group
may have non-upgraded members yet. I wonder if we could use a boolean to
signal this, e.g. isLegacy. Then, in the CLI we could have a column which
is only displayed when they are non upgraded members. What do you think?
If we cannot find a good way to represent this, we could drop it from this
KIP too.
Best,
David
On Fri, Nov 22, 2024 at 5:12 PM PoAn Yang <ya...@gmail.com> wrote:
> Hi Chia-Ping / David,
>
> Thanks for the review and suggestions.
>
> >> I don't fully follow your comment so please excuse me if I say something
> >> wrong.
>
> I think Chia-Ping wants to say: originally, the default protocol in
> classic group is “consumer”.
> After we have a new consumer group, the protocol of a classic member in
> consumer group is “classic”.
> The MemberDescription is used by both ClassicGroupDescription and
> ConsumerGroupDescription.
> If we add the protocol field to MemberDescription, the protocol of a
> classic member in classic group is “consumer”, but it’s “classic” in
> consumer group.
> To avoid misleading word, we can add a new collection of member protocols
> to ConsumerGroupDescription only.
> The original goal is to show member protocol during consumer migration, so
> we don’t need to add this information to ClassicGroupDescription.
>
> Please correct me if I misunderstood anything.
>
> Thanks,
> PoAn
>
> > On Nov 22, 2024, at 7:00 PM, Chia-Ping Tsai <ch...@gmail.com> wrote:
> >
> > hi DJ
> >
> >> How about we use another Jira to add “SHARE” to GroupProtocol and change
> > return type of ModernGroup#protocolType?
> >
> > This is not my comment ... sorry for the incorrect quote :(
> >
> >> A member in a classic group should report "classic".
> >
> > I completely agree. However, in this case,
> ClassicGroupDescription#protocol
> > returns 'consumer', while ClassicGroupDescription#members[x].protocol
> > returns 'classic'. This inconsistency feels a bit strange to me.
> >
> > Best,
> > Chia-Ping
> >
> >
> > David Jacot <dj...@confluent.io.invalid> 於 2024年11月22日 週五 下午6:51寫道：
> >
> >> Hi Chia,
> >>
> >>>> LM4: I agree it’s better to use a type like GroupProtocol enum, but I
> >> would like to align the format with kafka-groups.sh.
> >> How about we use another Jira to add “SHARE” to GroupProtocol and change
> >> return type of ModernGroup#protocolType?
> >>
> >> My understanding is that `GroupProtocol` is used to drive the config of
> the
> >> `Consumer`. Hence adding `Share` seems wrong to me. Or are you
> referring to
> >> another GroupProtocol enum? Perhaps GroupType enum?
> >>
> >>> In fact, I'm wondering if we should even add 'protocol' to
> >> MemberDescription,
> >> since the field has different values between classic and consumer
> groups—it
> >> shows 'consumer' in classic groups, whereas it shows 'classic' in
> consumer
> >> groups. Yes, it seems like a typo, but it's true. This inconsistency
> feels
> >> confusing to me. For example, users call Admin#describeConsumerGroups
> and
> >> see that a member has the 'consumer' protocol when it's in a classic
> group.
> >> Then, if the group is upgraded to a consumer group, calling
> >> Admin#describeConsumerGroups again shows the protocol as 'classic' for
> the
> >> same member.
> >>
> >> I don't fully follow your comment so please excuse me if I say something
> >> wrong.
> >>
> >> I think that it should work as follows:
> >> * A member in a classic group should report "classic".
> >> * A member using the consumer protocol in a consumer group should report
> >> "consumer".
> >> * A member using the classic protocol in a consumer group should report
> >> "classic".
> >> * A member in a share group should report "share".
> >>
> >>> If all we want to do is distinguish classic members in the
> >> ConsumerGroupDescription, then we shouldn't modify the common
> >> MemberDescription structure. Instead, we could add a collection to
> >> ConsumerGroupDescription to trace members using the classic protocol,
> which
> >> would allow us to enrich the output of command-line tools.
> >>
> >> Personally, I believe that using different collections per protocol
> types
> >> is not necessary. A field in the member is more than enough.
> >>
> >> Best,
> >> DJ
> >>
> >> On Fri, Nov 22, 2024 at 12:01 PM Chia-Ping Tsai <ch...@gmail.com>
> >> wrote:
> >>
> >>> hi PoAn
> >>>
> >>>> LM4: I agree it’s better to use a type like GroupProtocol enum, but I
> >>> would like to align the format with kafka-groups.sh.
> >>> How about we use another Jira to add “SHARE” to GroupProtocol and
> change
> >>> return type of ModernGroup#protocolType?
> >>>
> >>> MemberDescription is also used by ClassicGroupDescription, and in
> classic
> >>> groups, the protocol can be an arbitrary string, so we can't use an
> enum.
> >>>
> >>> In fact, I'm wondering if we should even add 'protocol' to
> >>> MemberDescription,
> >>> since the field has different values between classic and consumer
> >> groups—it
> >>> shows 'consumer' in classic groups, whereas it shows 'classic' in
> >> consumer
> >>> groups. Yes, it seems like a typo, but it's true. This inconsistency
> >> feels
> >>> confusing to me. For example, users call Admin#describeConsumerGroups
> and
> >>> see that a member has the 'consumer' protocol when it's in a classic
> >> group.
> >>> Then, if the group is upgraded to a consumer group, calling
> >>> Admin#describeConsumerGroups again shows the protocol as 'classic' for
> >> the
> >>> same member.
> >>>
> >>> If all we want to do is distinguish classic members in the
> >>> ConsumerGroupDescription, then we shouldn't modify the common
> >>> MemberDescription structure. Instead, we could add a collection to
> >>> ConsumerGroupDescription to trace members using the classic protocol,
> >> which
> >>> would allow us to enrich the output of command-line tools.
> >>> Best,
> >>> Chia-Ping
> >>>
> >>> PoAn Yang <ya...@gmail.com> 於 2024年11月22日 週五 下午3:46寫道：
> >>>
> >>>> Hi Lianet / Jeff,
> >>>>
> >>>> Thanks for the review.
> >>>>
> >>>> LM5: The kafka-share-groups.sh also uses “—describe” to show offsets
> >>>> information.
> >>>> Change to use “—describe —state —verbose” to show group level
> >> information
> >>>> in kafka-share-groups.sh.
> >>>>
> >>>> LM6: Update the example.
> >>>>
> >>>> LM7: Add a description to mention “—verbose” is a new option in
> >>>> ShareGroupCommandOptions.
> >>>>
> >>>> JK2: Yes, add the statement about format change for ASSIGNMENT value.
> >>>>
> >>>> Thanks,
> >>>> PoAn
> >>>>
> >>>>> On Nov 22, 2024, at 7:35 AM, Jeff Kim <je...@apache.org> wrote:
> >>>>>
> >>>>> Hi PoAn,
> >>>>>
> >>>>> JK2: Can we state in the KIP that we will reformat the existing
> >>>>> classic group's `--members --verbose` as well, specifically the
> >>>>> "ASSIGNMENT" column to include the topic name?
> >>>>>
> >>>>> Thanks!
> >>>>>
> >>>>> On 2024/11/21 14:33:37 "Lianet M." wrote:
> >>>>>> Thanks for the updates PoAn!
> >>>>>>
> >>>>>> LM6. Just a nit, under the --describe --state --verbose the header
> >> is
> >>>> fine
> >>>>>> but the example is still missing the --state argument.
> >>>>>>
> >>>>>> LM7. Under the Proposed changes for kafka-share-groups.sh, should we
> >>>>>> clearly mention that the KIP adds a new --verbose option? I think
> >> it's
> >>>>>> confusing because it's presented as if the option already exists and
> >>>> we're
> >>>>>> only changing the output (which is the case for
> >> kafka-consumer-groups
> >>>> but
> >>>>>> not for kafka-share-groups)
> >>>>>>
> >>>>>> That's all on my side. Thanks!
> >>>>>>
> >>>>>> Lianet
> >>>>>>
> >>>>>> On Wed, Nov 20, 2024 at 11:12 PM PoAn Yang <ya...@gmail.com>
> >>> wrote:
> >>>>>>
> >>>>>>> Hi Lianet / Jeff,
> >>>>>>>
> >>>>>>> Thanks for review and suggestions.
> >>>>>>>
> >>>>>>> LM4: At group level, the protocol field is from
> >>>> ModernGroup#protocolType.
> >>>>>>> It’s a string.
> >>>>>>> The value is defined in different places like
> >>> ShareGroup#PROTOCOL_TYPE
> >>>> and
> >>>>>>> ConsumerProtocol#PROTOCOL_TYPE.
> >>>>>>> In KIP-1043 <
> >>>>>>>
> >>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=305171038#KIP1043:Administrationofgroups-kafka-groups.sh
> >>>>> ,
> >>>>>>> the kafka-groups.sh also shows protocol like “classic” /
> >> “consumer” /
> >>>>>>> “share”.
> >>>>>>> I agree it’s better to use a type like GroupProtocol enum, but I
> >>> would
> >>>>>>> like to align the format with kafka-groups.sh.
> >>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and
> >>>> change
> >>>>>>> return type of ModernGroup#protocolType?
> >>>>>>>
> >>>>>>> LM5 / JK1: Sorry, I misunderstood LM3, so I deleted —state in the
> >>>> title.
> >>>>>>> You’re correct. I would like to use `—describe —state —verbose` to
> >>> show
> >>>>>>> group level information.
> >>>>>>> For `—describe —verbose`, it will show output as same as `—describe
> >>>>>>> —offsets —verbose`,
> >>>>>>> because `—describe` also shows output as same as `—describe
> >>> —offsets`.
> >>>>>>>
> >>>>>>> JK2: Yes, I only use consumer group in the sample output, but the
> >>>> classic
> >>>>>>> group will use the same format as well.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> PoAn
> >>>>>>>
> >>>>>>>> On Nov 21, 2024, at 8:28 AM, Jeff Kim <je...@apache.org>
> >> wrote:
> >>>>>>>>
> >>>>>>>> Hi PoAn,
> >>>>>>>>
> >>>>>>>> Thanks for the KIP. I have some questions:
> >>>>>>>>
> >>>>>>>> JK1: (Realized this is similar to LM5) On `--describe --verbose`
> >>>>>>> proposed changes - doesn't `--describe $group` default to printing
> >>> the
> >>>>>>> offsets? Perhaps you're referring to the `--state` argument? Also,
> >>>> would
> >>>>>>> that mean the default `--describe $group --verbose` command would
> >> not
> >>>> print
> >>>>>>> the added field to `--offsets verbose` (leader-epoch) or would it?
> >>>>>>>>
> >>>>>>>> JK2: Looking at ConsumerGroupCommand.java, the existing
> >> "ASSIGNMENT"
> >>>>>>> column under `--members --verbose` does group the topic partitions
> >> by
> >>>> topic
> >>>>>>> but does not prefix the grouping with the topic name like you're
> >>>> proposal:
> >>>>>>> "my_topic:0,1;new_topic:0,1". Should we do apply the same format
> >> for
> >>>> the
> >>>>>>> classic group as well?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 2024/11/20 21:32:41 "Lianet M." wrote:
> >>>>>>>>> Hello PoAn, just of couple more minor comments:
> >>>>>>>>>
> >>>>>>>>> LM4. Regarding the new “protocol” field added to
> >> MemberDescription.
> >>>>>>> Should
> >>>>>>>>> we consider reusing the existing GroupProtocol enum instead of
> >>>> String?
> >>>>>>>>> (It’s the one we use from the consumer side to refer to the
> >>> protocol
> >>>> in
> >>>>>>>>> use, just missing Share I notice).
> >>>>>>>>>
> >>>>>>>>> LM5. Regarding the change to the output for
> >> kafka-consumer-groups,
> >>>> the
> >>>>>>>>> command shown does not include the —state option, but the output
> >>>> shows
> >>>>>>>>> state info (state, #members, epochs). I would guess that we want
> >> to
> >>>>>>> modify
> >>>>>>>>> the output only when we describe a group with the —state —verbose
> >>>>>>> option,
> >>>>>>>>> is my understanding right? If my understanding is right we’re
> >> just
> >>>>>>> missing
> >>>>>>>>> adding the —state in the example, and the KIP does not introduce
> >>> any
> >>>>>>>>> changes to the —describe —verbose option. (If not, it would mean
> >> a
> >>>>>>> bigger
> >>>>>>>>> change to the output of —describe —verbose which I expect is not
> >>> the
> >>>>>>>>> intention?)
> >>>>>>>>>
> >>>>>>>>> Thanks!
> >>>>>>>>>
> >>>>>>>>> Lianet
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Wed, Nov 20, 2024 at 2:12 AM PoAn Yang <ya...@gmail.com>
> >>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi Chia-Ping,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for the review and suggestions.
> >>>>>>>>>>
> >>>>>>>>>> CI0: Thanks for the reminder. Update validVersions in
> >>>>>>>>>> ConsumerGroupDescribeRequest to 0-1.
> >>>>>>>>>>
> >>>>>>>>>> CI1: Yes, I use `ConsumerGroupMember#useClassicProtocol` to
> >> check
> >>>>>>> whether
> >>>>>>>>>> a member in consumer group uses “classic” or “consumer”
> >> protocol.
> >>>>>>>>>>
> >>>>>>>>>> CI2: Yes, a member in share group always uses “share” protocol.
> >>>>>>>>>>
> >>>>>>>>>> CI3: Add a table to show meaning of “classic”, “consumer”, and
> >>>> “share”
> >>>>>>>>>> protocol.
> >>>>>>>>>>
> >>>>>>>>>> BTW, the vote thread is in
> >>>>>>>>>>
> >> https://lists.apache.org/thread/rb25tf75tzf4c7jqqldlo5jh9w8chsq6.
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> PoAn
> >>>>>>>>>>
> >>>>>>>>>>> On Nov 20, 2024, at 11:46 AM, Chia-Ping Tsai <
> >>> chia7712@apache.org>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> hi PoAn
> >>>>>>>>>>>
> >>>>>>>>>>> CI0: We have to bump the version of
> >> ConsumerGroupDescribeRequest
> >>>> as
> >>>>>>>>>> well, so server can distinguish the new and old behavior.
> >>>>>>>>>>>
> >>>>>>>>>>> CI1: the type of new filed is string, so I guess you plan to
> >> use
> >>>>>>>>>> `ConsumerGroupMember#useClassicProtocol` [0] flag to return
> >> either
> >>>>>>>>>> "classic" or "consumer", right?
> >>>>>>>>>>>
> >>>>>>>>>>> CI2: `MemberDescription` is used by `ShareGroupDescription`
> >> too,
> >>> so
> >>>>>>> the
> >>>>>>>>>> new filed protocol in shared group is always "shared", right?
> >>>>>>>>>>>
> >>>>>>>>>>> CI3: Could you consider adding a table to show the value of the
> >>>>>>> protocol
> >>>>>>>>>> field in each case? Andrew has a beautiful table in KIP-1043
> >> that
> >>>>>>> lists all
> >>>>>>>>>> possible protocol names.
> >>>>>>>>>>>
> >>>>>>>>>>> [0]
> >>>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> https://github.com/apache/kafka/blob/441a6d0b790f4a17b454caeea7588a6b90fbd9db/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/consumer/ConsumerGroupMember.java#L454
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Best,
> >>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>
> >>>>>>>>>>> On 2024/11/19 15:45:16 PoAn Yang wrote:
> >>>>>>>>>>>> Hi David,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks for the reminder.
> >>>>>>>>>>>>
> >>>>>>>>>>>> DJ4: Add related content to the KIP.
> >>>>>>>>>>>> Bump validVersions in ConsumerGroupDescribeResponse to “0-1”.
> >>>>>>>>>>>> Add a new field “Protocol” to ConsumerGroupDescribeResponse.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>> PoAn
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On Nov 19, 2024, at 3:57 PM, David Jacot
> >>>>>>> <dj...@confluent.io.INVALID>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> DJ4: Could you please also include the required changes to
> >> the
> >>>>>>>>>>>>> ConsumerGroupDescribed API in the public interface section?
> >> We
> >>>>>>>>>> basically
> >>>>>>>>>>>>> need to bump the version, add the new field, etc.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>> David
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang <
> >> yangpoan@gmail.com>
> >>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi David,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> DJ4: Add a new field protocol to MemberDescription,
> >>>>>>>>>>>>>> so the command line tool can show protocol information when
> >>>> users
> >>>>>>>>>> describe
> >>>>>>>>>>>>>> members.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> If there is no further suggestion, I will start a vote
> >> thread
> >>>>>>> today.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Nov 15, 2024, at 11:50 PM, David Jacot
> >>>>>>>>>> <dj...@confluent.io.INVALID>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> DJ2: Using "-" sounds good to me.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> DJ3: That seems reasonable to me.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> DJ4: Why not add it right now? I don't want to change the
> >>>> output
> >>>>>>> of
> >>>>>>>>>> the
> >>>>>>>>>>>>>>> tool too many times.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>> David
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang <
> >>> yangpoan@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi David / Andrew,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks for review. Thanks Andrew for picking up
> >>>>>>>>>> kafka-share-groups.sh
> >>>>>>>>>>>>>>>> implementation.
> >>>>>>>>>>>>>>>> I will handle kafka-consumer-groups.sh.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> DJ3: After discussing with @Chia-Ping Tsai, we think that
> >>>> using
> >>>>>>> new
> >>>>>>>>>>>>>> format
> >>>>>>>>>>>>>>>> is more clear.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> The new format will be like
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Using colon(:) to concat topic name and partition IDs.
> >>>>>>>>>>>>>>>> Using comma(,) to concat partition IDs within same topic
> >>> name.
> >>>>>>>>>>>>>>>> Using semicolon(;) to concat topic strings.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> AS3: Fix it with kafka-share-groups.sh. Thanks.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> If there is no further suggestion, I will start a vote
> >>> thread
> >>>>>>> next
> >>>>>>>>>>>>>> Monday.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield <
> >>>>>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>>>>>>>> DJ2: I was just going to comment that "-" would be a more
> >>>>>>>>>> appropriate
> >>>>>>>>>>>>>>>> missing value, but
> >>>>>>>>>>>>>>>>> you got there first.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> AS3: The examples for kafka-share-groups.sh include
> >>>>>>>>>>>>>>>> kafka-consumer-groups.sh in the
> >>>>>>>>>>>>>>>>> command line.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> If this is accepted in time, I'm happy to pick up the
> >>>>>>>>>> implementation of
> >>>>>>>>>>>>>>>> the share groups
> >>>>>>>>>>>>>>>>> part of this if it helps.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>> Andrew
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>>>>> From: Frank Yang <ya...@gmail.com>
> >>>>>>>>>>>>>>>>> Sent: 14 November 2024 10:48
> >>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
> >>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
> >>> kafka-consumer-groups
> >>>>>>>>>> command
> >>>>>>>>>>>>>>>> line tool to support new consumer group
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi David,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks for the review and suggestion! I would like to get
> >>>> this
> >>>>>>> in
> >>>>>>>>>> AK
> >>>>>>>>>>>>>> 4.0
> >>>>>>>>>>>>>>>> as well. I will do my best.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> DJ1: Update KIP to put GROUP-EPOCH and
> >>>> TARGET-ASSIGNMENT-EPOCH
> >>>>>>>>>> before
> >>>>>>>>>>>>>>>> #MEMBERS.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> DJ2: I prefer to follow current missing column value “-“.
> >>>>>>>>>> (reference <
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92
> >>>>>>>>>>>>>>>>> )
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> DJ3: Update KIP to use CURRENT-EPOCH CURRENT-ASSIGNMENT
> >>>>>>>>>> TARGET-EPOCH
> >>>>>>>>>>>>>>>> TARGET-ASSIGNMENT.
> >>>>>>>>>>>>>>>>> Remove GROUP-EPOCH.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> For assignment value, it follows current output
> >> (reference
> >>> <
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418
> >>>>>>>>>>>>>>> ).
> >>>>>>>>>>>>>>>> I think the form of `topicid-partitionid` is more clear.
> >>>>>>>>>>>>>>>>> If we would like to use this form, I will update both
> >>> output
> >>>> in
> >>>>>>>>>>>>>>>> kafka-consumer-groups.sh and kafka-share-groups.sh.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> DJ4: It looks like DescribeGroupsResponseData only has
> >>>> protocol
> >>>>>>>>>> type at
> >>>>>>>>>>>>>>>> group level.
> >>>>>>>>>>>>>>>>> Both DescribeGroupsResponseData and
> >>>>>>>>>> ConsumerGroupDescribeResponseData
> >>>>>>>>>>>>>>>> don’t have protocol at member level.
> >>>>>>>>>>>>>>>>> Could we use a followup to add it?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> DJ5: Update KIP to put LEADER-EPOCH before
> >> CURRENT-OFFSET.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot
> >>>>>>>>>> <djacot@confluent.io.INVALID
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks for the KIP! I have a few minor
> >>> comments/suggestions:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> DJ1: In the output of `--describe --verbose`, I would
> >>>> suggest
> >>>>>>>>>> putting
> >>>>>>>>>>>>>>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before
> >>>> `#MEMBERS`.
> >>>>>>>>>>>>>>>>>> DJ2: Continuing on the above, I assume that we will
> >> print
> >>>> out
> >>>>>>> N/A
> >>>>>>>>>> for
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> fields not supported by classic groups. Is this correct?
> >>>>>>>>>>>>>>>>>> DJ3: In the output of `--describe --members --verbose`,
> >> I
> >>>>>>> wonder
> >>>>>>>>>> if we
> >>>>>>>>>>>>>>>>>> should use the following order and terms: CURRENT-EPOCH
> >>>>>>>>>>>>>>>> CURRENT-ASSIGNMENT
> >>>>>>>>>>>>>>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the
> >>>> GROUP-EPOCH
> >>>>>>>>>>>>>> because
> >>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>> is already in the group description. The value `(0)`
> >> used
> >>>> for
> >>>>>>> the
> >>>>>>>>>>>>>>>>>> assignment is incorrect. Here I suppose that we will
> >> print
> >>>> out
> >>>>>>> the
> >>>>>>>>>>>>>> list
> >>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>> partitions in the form of `topicid-partitionid`.
> >>>>>>>>>>>>>>>>>> DJ4: Continuing on the above, I wonder if we should also
> >>> add
> >>>>>>> the
> >>>>>>>>>>>>>>>> protocol
> >>>>>>>>>>>>>>>>>> used `classic` or `consumer`. For context, it is
> >> possible
> >>> to
> >>>>>>> have
> >>>>>>>>>>>>>>>> `classic`
> >>>>>>>>>>>>>>>>>> members and `consumer` members in a `consumer` group
> >>> during
> >>>> an
> >>>>>>>>>> online
> >>>>>>>>>>>>>>>>>> upgrade from the classic protocol to the consumer
> >>> protocol.
> >>>>>>> Having
> >>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>> information may be handy for administrators. What do you
> >>>> think?
> >>>>>>>>>>>>>>>>>> DJ5: In the output of `--describe --offsets --verbose`,
> >> I
> >>>> would
> >>>>>>>>>>>>>> suggest
> >>>>>>>>>>>>>>>>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> It would be great if we could get this one in AK 4.0 as
> >> it
> >>>>>>>>>> changes the
> >>>>>>>>>>>>>>>>>> output of the command.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>> DJ
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <
> >>>> yangpoan@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi Sean / Andrew / Lianet,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks for all your review and suggestions.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> AS1, LM1, LM4: Change to add KIP-848 information when
> >>> users
> >>>>>>> give
> >>>>>>>>>>>>>>>> —verbose
> >>>>>>>>>>>>>>>>>>> option.
> >>>>>>>>>>>>>>>>>>> —describe —verbose: shows group epoch and target
> >>> assignment
> >>>>>>>>>> epoch.
> >>>>>>>>>>>>>>>>>>> —describe —members —verbose: shows above information,
> >>>> member
> >>>>>>>>>> epoch,
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>> target assignment.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> AS2: Change to use MEMBER-EPOCH to align with KIP-848
> >>>>>>> definition.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> LM2: For classic group, it doesn’t have epoch, so I use
> >>>>>>> Optional
> >>>>>>>>>>>>>>>> fields in
> >>>>>>>>>>>>>>>>>>> ConsumerGroupDescription.
> >>>>>>>>>>>>>>>>>>> For share group, it relies on KIP-848. It must have
> >>> epoch,
> >>>> so
> >>>>>>> I
> >>>>>>>>>> use
> >>>>>>>>>>>>>> int
> >>>>>>>>>>>>>>>>>>> fields in ShareGroupDescription.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> LM3: Remove —state change. We can get group level
> >>>> information
> >>>>>>> by
> >>>>>>>>>>>>>>>> —describe
> >>>>>>>>>>>>>>>>>>> —verbose.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> SQ1: Add LEADER-EPOCH when users give —describe
> >> —offsets
> >>>>>>>>>> —verbose.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks.
> >>>>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <
> >>> lianetmr@gmail.com
> >>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hello Frank, thanks for the KIP! A few comments:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> LM1. I strongly agree with Andrew's suggestion of
> >> moving
> >>>> this
> >>>>>>>>>> into a
> >>>>>>>>>>>>>>>>>>>> --verbose option. I expect someone would only attempt
> >> to
> >>>> make
> >>>>>>>>>> sense
> >>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> epochs while debugging an issue, not while trying to
> >>> view
> >>>> the
> >>>>>>>>>> group
> >>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>> member's info. (Also member-epoch makes more sense to
> >>> me,
> >>>>>>> even
> >>>>>>>>>> more
> >>>>>>>>>>>>>>>> if we
> >>>>>>>>>>>>>>>>>>>> end up agreeing in a --verbose). Related to both
> >> issues,
> >>>> my
> >>>>>>>>>> take is
> >>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>> whoever is close to the protocol will expect
> >>> member-epoch,
> >>>>>>>>>> whoever
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>> will probably won't even need to see the epochs at
> >> all.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> LM2. Why are the epochs defined as Optional (don't we
> >>>> expect
> >>>>>>> to
> >>>>>>>>>>>>>> always
> >>>>>>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>>> them? with 0 initially, defined on the broker side for
> >>> the
> >>>>>>> group
> >>>>>>>>>>>>>> ones,
> >>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>> client side for the member)
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> LM3. Why is the KIP including the –state option in the
> >>>>>>> proposed
> >>>>>>>>>>>>>>>> changes?
> >>>>>>>>>>>>>>>>>>> (
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
> >>>>>>>>>>>>>>>>>>>> ).
> >>>>>>>>>>>>>>>>>>>> I get that the output would change in that example,
> >> but
> >>>> it’s
> >>>>>>> not
> >>>>>>>>>>>>>>>> because
> >>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>> any change to the –state option itself. It's because
> >> of
> >>>> the
> >>>>>>>>>> change
> >>>>>>>>>>>>>>>>>>> proposed
> >>>>>>>>>>>>>>>>>>>> to the –described (required with the --state), and the
> >>>>>>> changes
> >>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>> --describe are already explained above (seems
> >> confusing,
> >>>> got
> >>>>>>> me
> >>>>>>>>>>>>>>>> looking
> >>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>> a change to the state filter which seemed unrelated).
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> LM4. Group epoch and target assignment epoch are
> >>>> conceptually
> >>>>>>>>>> at the
> >>>>>>>>>>>>>>>>>>> group
> >>>>>>>>>>>>>>>>>>>> level. vs member epoch that lands at a member level.
> >> So
> >>>>>>> wonder
> >>>>>>>>>> if we
> >>>>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>> show them accordingly (ex. using the --verbose option)
> >>>>>>>>>>>>>>>>>>>> –describe –verbose => shows group epoch and target
> >>>> assignment
> >>>>>>>>>> epoch
> >>>>>>>>>>>>>>>>>>>> –describe –members –verbose => shows member
> >>> epoch
> >>>>>>>>>> (along
> >>>>>>>>>>>>>>>>>>>> with group epoch and target assignment epoch)
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Lianet
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield <
> >>>>>>>>>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I have a few comments.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> AS1: It seems to me that these new additions are most
> >>>>>>> useful to
> >>>>>>>>>>>>>>>> people
> >>>>>>>>>>>>>>>>>>>>> trying to understand
> >>>>>>>>>>>>>>>>>>>>> the progress of rebalancing in quite some detail. The
> >>>>>>>>>> information
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>> really not understandable
> >>>>>>>>>>>>>>>>>>>>> for most users who do not have deep knowledge of
> >>>>>>> KIP-848/932.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> As a result, I suggest for kafka-share-groups.sh that
> >>> you
> >>>>>>> add a
> >>>>>>>>>>>>>>>>>>> --members
> >>>>>>>>>>>>>>>>>>>>> --verbose option
> >>>>>>>>>>>>>>>>>>>>> and only include the new information in the output
> >> for
> >>>> that
> >>>>>>>>>> option,
> >>>>>>>>>>>>>>>>>>> rather
> >>>>>>>>>>>>>>>>>>>>> than changing the
> >>>>>>>>>>>>>>>>>>>>> non-verbose --members output.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> I also make a similar suggestion for
> >>>>>>> kafka-consumer-groups.sh
> >>>>>>>>>>>>>>>> --members
> >>>>>>>>>>>>>>>>>>>>> and only add the
> >>>>>>>>>>>>>>>>>>>>> new information for the --verbose output.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> AS2: I strongly suggest that you use MEMBER-EPOCH
> >>>> instead of
> >>>>>>>>>>>>>>>>>>>>> CONSUMER-EPOCH.
> >>>>>>>>>>>>>>>>>>>>> I think it's more confusing not following the
> >>>> terminology of
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>> KIPs.
> >>>>>>>>>>>>>>>>>>> For
> >>>>>>>>>>>>>>>>>>>>> one thing,
> >>>>>>>>>>>>>>>>>>>>> we already have "member" in the admin client such as
> >>>>>>>>>>>>>>>> MemberDescription.
> >>>>>>>>>>>>>>>>>>>>> It's not a
> >>>>>>>>>>>>>>>>>>>>> new term.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>> Andrew
> >>>>>>>>>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>>>>>>>>> From: PoAn Yang <pa...@apache.org>
> >>>>>>>>>>>>>>>>>>>>> Sent: 25 October 2024 13:55
> >>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
> >>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
> >>>>>>> kafka-consumer-groups
> >>>>>>>>>>>>>> command
> >>>>>>>>>>>>>>>>>>> line
> >>>>>>>>>>>>>>>>>>>>> tool to support new consumer group
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hi Lucas,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks for the review!
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> 1) Yes, I add related change for
> >> kafka-share-groups.sh
> >>> to
> >>>>>>> the
> >>>>>>>>>> KIP.
> >>>>>>>>>>>>>>>> Could
> >>>>>>>>>>>>>>>>>>>>> you take a look? Thanks for the suggestion.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use
> >>>> MEMBER-EPOCH
> >>>>>>>>>> here,
> >>>>>>>>>>>>>>>> users
> >>>>>>>>>>>>>>>>>>> may
> >>>>>>>>>>>>>>>>>>>>> confuse what is different between CONSUMER and
> >> MEMBER.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
> >>>>>>>>>>>>>>>>>>>>>> Hi Frank,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> thanks for the KIP!
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> 1) For consistency, should we do the same for
> >>>>>>>>>>>>>>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription, etc. ?
> >>>> Even
> >>>>>>> if
> >>>>>>>>>> we do
> >>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>> implement it right now if the share group
> >>> implementation
> >>>>>>> may
> >>>>>>>>>> still
> >>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>> incomplete, it may make sense to include it in the
> >>> KIP.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH?
> >> That
> >>>> would
> >>>>>>>>>> seem
> >>>>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>>>>>> consistent.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>> Lucas
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <
> >>>>>>>>>> yangpoan@gmail.com>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I would like to kick off the discussion of
> >> KIP-1099.
> >>>> This
> >>>>>>> KIP
> >>>>>>>>>>>>>>>> enhances
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups tools to include state which
> >> is
> >>>>>>>>>> introduced
> >>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>>>> KIP-848.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups command line
> >>>> tool
> >>>>>>> to
> >>>>>>>>>>>>>> support
> >>>>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>>>> consumer group - Apache Kafka - Apache Software
> >>>> Foundation
> >>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> cwiki.apache.org
> >>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> [image: favicon.ico]
> >>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thank you,
> >>>>>>>>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>
> >>
>
>
>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by PoAn Yang <ya...@gmail.com>.
Hi Chia-Ping / David,
Thanks for the review and suggestions.
>> I don't fully follow your comment so please excuse me if I say something
>> wrong.
I think Chia-Ping wants to say: originally, the default protocol in classic group is “consumer”.
After we have a new consumer group, the protocol of a classic member in consumer group is “classic”.
The MemberDescription is used by both ClassicGroupDescription and ConsumerGroupDescription.
If we add the protocol field to MemberDescription, the protocol of a classic member in classic group is “consumer”, but it’s “classic” in consumer group.
To avoid misleading word, we can add a new collection of member protocols to ConsumerGroupDescription only.
The original goal is to show member protocol during consumer migration, so we don’t need to add this information to ClassicGroupDescription.
Please correct me if I misunderstood anything.
Thanks,
PoAn
> On Nov 22, 2024, at 7:00 PM, Chia-Ping Tsai <ch...@gmail.com> wrote:
>
> hi DJ
>
>> How about we use another Jira to add “SHARE” to GroupProtocol and change
> return type of ModernGroup#protocolType?
>
> This is not my comment ... sorry for the incorrect quote :(
>
>> A member in a classic group should report "classic".
>
> I completely agree. However, in this case, ClassicGroupDescription#protocol
> returns 'consumer', while ClassicGroupDescription#members[x].protocol
> returns 'classic'. This inconsistency feels a bit strange to me.
>
> Best,
> Chia-Ping
>
>
> David Jacot <dj...@confluent.io.invalid> 於 2024年11月22日 週五 下午6:51寫道：
>
>> Hi Chia,
>>
>>>> LM4: I agree it’s better to use a type like GroupProtocol enum, but I
>> would like to align the format with kafka-groups.sh.
>> How about we use another Jira to add “SHARE” to GroupProtocol and change
>> return type of ModernGroup#protocolType?
>>
>> My understanding is that `GroupProtocol` is used to drive the config of the
>> `Consumer`. Hence adding `Share` seems wrong to me. Or are you referring to
>> another GroupProtocol enum? Perhaps GroupType enum?
>>
>>> In fact, I'm wondering if we should even add 'protocol' to
>> MemberDescription,
>> since the field has different values between classic and consumer groups—it
>> shows 'consumer' in classic groups, whereas it shows 'classic' in consumer
>> groups. Yes, it seems like a typo, but it's true. This inconsistency feels
>> confusing to me. For example, users call Admin#describeConsumerGroups and
>> see that a member has the 'consumer' protocol when it's in a classic group.
>> Then, if the group is upgraded to a consumer group, calling
>> Admin#describeConsumerGroups again shows the protocol as 'classic' for the
>> same member.
>>
>> I don't fully follow your comment so please excuse me if I say something
>> wrong.
>>
>> I think that it should work as follows:
>> * A member in a classic group should report "classic".
>> * A member using the consumer protocol in a consumer group should report
>> "consumer".
>> * A member using the classic protocol in a consumer group should report
>> "classic".
>> * A member in a share group should report "share".
>>
>>> If all we want to do is distinguish classic members in the
>> ConsumerGroupDescription, then we shouldn't modify the common
>> MemberDescription structure. Instead, we could add a collection to
>> ConsumerGroupDescription to trace members using the classic protocol, which
>> would allow us to enrich the output of command-line tools.
>>
>> Personally, I believe that using different collections per protocol types
>> is not necessary. A field in the member is more than enough.
>>
>> Best,
>> DJ
>>
>> On Fri, Nov 22, 2024 at 12:01 PM Chia-Ping Tsai <ch...@gmail.com>
>> wrote:
>>
>>> hi PoAn
>>>
>>>> LM4: I agree it’s better to use a type like GroupProtocol enum, but I
>>> would like to align the format with kafka-groups.sh.
>>> How about we use another Jira to add “SHARE” to GroupProtocol and change
>>> return type of ModernGroup#protocolType?
>>>
>>> MemberDescription is also used by ClassicGroupDescription, and in classic
>>> groups, the protocol can be an arbitrary string, so we can't use an enum.
>>>
>>> In fact, I'm wondering if we should even add 'protocol' to
>>> MemberDescription,
>>> since the field has different values between classic and consumer
>> groups—it
>>> shows 'consumer' in classic groups, whereas it shows 'classic' in
>> consumer
>>> groups. Yes, it seems like a typo, but it's true. This inconsistency
>> feels
>>> confusing to me. For example, users call Admin#describeConsumerGroups and
>>> see that a member has the 'consumer' protocol when it's in a classic
>> group.
>>> Then, if the group is upgraded to a consumer group, calling
>>> Admin#describeConsumerGroups again shows the protocol as 'classic' for
>> the
>>> same member.
>>>
>>> If all we want to do is distinguish classic members in the
>>> ConsumerGroupDescription, then we shouldn't modify the common
>>> MemberDescription structure. Instead, we could add a collection to
>>> ConsumerGroupDescription to trace members using the classic protocol,
>> which
>>> would allow us to enrich the output of command-line tools.
>>> Best,
>>> Chia-Ping
>>>
>>> PoAn Yang <ya...@gmail.com> 於 2024年11月22日 週五 下午3:46寫道：
>>>
>>>> Hi Lianet / Jeff,
>>>>
>>>> Thanks for the review.
>>>>
>>>> LM5: The kafka-share-groups.sh also uses “—describe” to show offsets
>>>> information.
>>>> Change to use “—describe —state —verbose” to show group level
>> information
>>>> in kafka-share-groups.sh.
>>>>
>>>> LM6: Update the example.
>>>>
>>>> LM7: Add a description to mention “—verbose” is a new option in
>>>> ShareGroupCommandOptions.
>>>>
>>>> JK2: Yes, add the statement about format change for ASSIGNMENT value.
>>>>
>>>> Thanks,
>>>> PoAn
>>>>
>>>>> On Nov 22, 2024, at 7:35 AM, Jeff Kim <je...@apache.org> wrote:
>>>>>
>>>>> Hi PoAn,
>>>>>
>>>>> JK2: Can we state in the KIP that we will reformat the existing
>>>>> classic group's `--members --verbose` as well, specifically the
>>>>> "ASSIGNMENT" column to include the topic name?
>>>>>
>>>>> Thanks!
>>>>>
>>>>> On 2024/11/21 14:33:37 "Lianet M." wrote:
>>>>>> Thanks for the updates PoAn!
>>>>>>
>>>>>> LM6. Just a nit, under the --describe --state --verbose the header
>> is
>>>> fine
>>>>>> but the example is still missing the --state argument.
>>>>>>
>>>>>> LM7. Under the Proposed changes for kafka-share-groups.sh, should we
>>>>>> clearly mention that the KIP adds a new --verbose option? I think
>> it's
>>>>>> confusing because it's presented as if the option already exists and
>>>> we're
>>>>>> only changing the output (which is the case for
>> kafka-consumer-groups
>>>> but
>>>>>> not for kafka-share-groups)
>>>>>>
>>>>>> That's all on my side. Thanks!
>>>>>>
>>>>>> Lianet
>>>>>>
>>>>>> On Wed, Nov 20, 2024 at 11:12 PM PoAn Yang <ya...@gmail.com>
>>> wrote:
>>>>>>
>>>>>>> Hi Lianet / Jeff,
>>>>>>>
>>>>>>> Thanks for review and suggestions.
>>>>>>>
>>>>>>> LM4: At group level, the protocol field is from
>>>> ModernGroup#protocolType.
>>>>>>> It’s a string.
>>>>>>> The value is defined in different places like
>>> ShareGroup#PROTOCOL_TYPE
>>>> and
>>>>>>> ConsumerProtocol#PROTOCOL_TYPE.
>>>>>>> In KIP-1043 <
>>>>>>>
>>>>
>>>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=305171038#KIP1043:Administrationofgroups-kafka-groups.sh
>>>>> ,
>>>>>>> the kafka-groups.sh also shows protocol like “classic” /
>> “consumer” /
>>>>>>> “share”.
>>>>>>> I agree it’s better to use a type like GroupProtocol enum, but I
>>> would
>>>>>>> like to align the format with kafka-groups.sh.
>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and
>>>> change
>>>>>>> return type of ModernGroup#protocolType?
>>>>>>>
>>>>>>> LM5 / JK1: Sorry, I misunderstood LM3, so I deleted —state in the
>>>> title.
>>>>>>> You’re correct. I would like to use `—describe —state —verbose` to
>>> show
>>>>>>> group level information.
>>>>>>> For `—describe —verbose`, it will show output as same as `—describe
>>>>>>> —offsets —verbose`,
>>>>>>> because `—describe` also shows output as same as `—describe
>>> —offsets`.
>>>>>>>
>>>>>>> JK2: Yes, I only use consumer group in the sample output, but the
>>>> classic
>>>>>>> group will use the same format as well.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> PoAn
>>>>>>>
>>>>>>>> On Nov 21, 2024, at 8:28 AM, Jeff Kim <je...@apache.org>
>> wrote:
>>>>>>>>
>>>>>>>> Hi PoAn,
>>>>>>>>
>>>>>>>> Thanks for the KIP. I have some questions:
>>>>>>>>
>>>>>>>> JK1: (Realized this is similar to LM5) On `--describe --verbose`
>>>>>>> proposed changes - doesn't `--describe $group` default to printing
>>> the
>>>>>>> offsets? Perhaps you're referring to the `--state` argument? Also,
>>>> would
>>>>>>> that mean the default `--describe $group --verbose` command would
>> not
>>>> print
>>>>>>> the added field to `--offsets verbose` (leader-epoch) or would it?
>>>>>>>>
>>>>>>>> JK2: Looking at ConsumerGroupCommand.java, the existing
>> "ASSIGNMENT"
>>>>>>> column under `--members --verbose` does group the topic partitions
>> by
>>>> topic
>>>>>>> but does not prefix the grouping with the topic name like you're
>>>> proposal:
>>>>>>> "my_topic:0,1;new_topic:0,1". Should we do apply the same format
>> for
>>>> the
>>>>>>> classic group as well?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2024/11/20 21:32:41 "Lianet M." wrote:
>>>>>>>>> Hello PoAn, just of couple more minor comments:
>>>>>>>>>
>>>>>>>>> LM4. Regarding the new “protocol” field added to
>> MemberDescription.
>>>>>>> Should
>>>>>>>>> we consider reusing the existing GroupProtocol enum instead of
>>>> String?
>>>>>>>>> (It’s the one we use from the consumer side to refer to the
>>> protocol
>>>> in
>>>>>>>>> use, just missing Share I notice).
>>>>>>>>>
>>>>>>>>> LM5. Regarding the change to the output for
>> kafka-consumer-groups,
>>>> the
>>>>>>>>> command shown does not include the —state option, but the output
>>>> shows
>>>>>>>>> state info (state, #members, epochs). I would guess that we want
>> to
>>>>>>> modify
>>>>>>>>> the output only when we describe a group with the —state —verbose
>>>>>>> option,
>>>>>>>>> is my understanding right? If my understanding is right we’re
>> just
>>>>>>> missing
>>>>>>>>> adding the —state in the example, and the KIP does not introduce
>>> any
>>>>>>>>> changes to the —describe —verbose option. (If not, it would mean
>> a
>>>>>>> bigger
>>>>>>>>> change to the output of —describe —verbose which I expect is not
>>> the
>>>>>>>>> intention?)
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>>
>>>>>>>>> Lianet
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Nov 20, 2024 at 2:12 AM PoAn Yang <ya...@gmail.com>
>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Chia-Ping,
>>>>>>>>>>
>>>>>>>>>> Thanks for the review and suggestions.
>>>>>>>>>>
>>>>>>>>>> CI0: Thanks for the reminder. Update validVersions in
>>>>>>>>>> ConsumerGroupDescribeRequest to 0-1.
>>>>>>>>>>
>>>>>>>>>> CI1: Yes, I use `ConsumerGroupMember#useClassicProtocol` to
>> check
>>>>>>> whether
>>>>>>>>>> a member in consumer group uses “classic” or “consumer”
>> protocol.
>>>>>>>>>>
>>>>>>>>>> CI2: Yes, a member in share group always uses “share” protocol.
>>>>>>>>>>
>>>>>>>>>> CI3: Add a table to show meaning of “classic”, “consumer”, and
>>>> “share”
>>>>>>>>>> protocol.
>>>>>>>>>>
>>>>>>>>>> BTW, the vote thread is in
>>>>>>>>>>
>> https://lists.apache.org/thread/rb25tf75tzf4c7jqqldlo5jh9w8chsq6.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> PoAn
>>>>>>>>>>
>>>>>>>>>>> On Nov 20, 2024, at 11:46 AM, Chia-Ping Tsai <
>>> chia7712@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> hi PoAn
>>>>>>>>>>>
>>>>>>>>>>> CI0: We have to bump the version of
>> ConsumerGroupDescribeRequest
>>>> as
>>>>>>>>>> well, so server can distinguish the new and old behavior.
>>>>>>>>>>>
>>>>>>>>>>> CI1: the type of new filed is string, so I guess you plan to
>> use
>>>>>>>>>> `ConsumerGroupMember#useClassicProtocol` [0] flag to return
>> either
>>>>>>>>>> "classic" or "consumer", right?
>>>>>>>>>>>
>>>>>>>>>>> CI2: `MemberDescription` is used by `ShareGroupDescription`
>> too,
>>> so
>>>>>>> the
>>>>>>>>>> new filed protocol in shared group is always "shared", right?
>>>>>>>>>>>
>>>>>>>>>>> CI3: Could you consider adding a table to show the value of the
>>>>>>> protocol
>>>>>>>>>> field in each case? Andrew has a beautiful table in KIP-1043
>> that
>>>>>>> lists all
>>>>>>>>>> possible protocol names.
>>>>>>>>>>>
>>>>>>>>>>> [0]
>>>>>>>>>>
>>>>>>>
>>>>
>>>
>> https://github.com/apache/kafka/blob/441a6d0b790f4a17b454caeea7588a6b90fbd9db/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/consumer/ConsumerGroupMember.java#L454
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>
>>>>>>>>>>> On 2024/11/19 15:45:16 PoAn Yang wrote:
>>>>>>>>>>>> Hi David,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for the reminder.
>>>>>>>>>>>>
>>>>>>>>>>>> DJ4: Add related content to the KIP.
>>>>>>>>>>>> Bump validVersions in ConsumerGroupDescribeResponse to “0-1”.
>>>>>>>>>>>> Add a new field “Protocol” to ConsumerGroupDescribeResponse.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> PoAn
>>>>>>>>>>>>
>>>>>>>>>>>>> On Nov 19, 2024, at 3:57 PM, David Jacot
>>>>>>> <dj...@confluent.io.INVALID>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>
>>>>>>>>>>>>> DJ4: Could you please also include the required changes to
>> the
>>>>>>>>>>>>> ConsumerGroupDescribed API in the public interface section?
>> We
>>>>>>>>>> basically
>>>>>>>>>>>>> need to bump the version, add the new field, etc.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> David
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang <
>> yangpoan@gmail.com>
>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi David,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> DJ4: Add a new field protocol to MemberDescription,
>>>>>>>>>>>>>> so the command line tool can show protocol information when
>>>> users
>>>>>>>>>> describe
>>>>>>>>>>>>>> members.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If there is no further suggestion, I will start a vote
>> thread
>>>>>>> today.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Nov 15, 2024, at 11:50 PM, David Jacot
>>>>>>>>>> <dj...@confluent.io.INVALID>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> DJ2: Using "-" sounds good to me.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> DJ3: That seems reasonable to me.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> DJ4: Why not add it right now? I don't want to change the
>>>> output
>>>>>>> of
>>>>>>>>>> the
>>>>>>>>>>>>>>> tool too many times.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>> David
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang <
>>> yangpoan@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi David / Andrew,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks for review. Thanks Andrew for picking up
>>>>>>>>>> kafka-share-groups.sh
>>>>>>>>>>>>>>>> implementation.
>>>>>>>>>>>>>>>> I will handle kafka-consumer-groups.sh.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> DJ3: After discussing with @Chia-Ping Tsai, we think that
>>>> using
>>>>>>> new
>>>>>>>>>>>>>> format
>>>>>>>>>>>>>>>> is more clear.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The new format will be like
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>
>>>
>> <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Using colon(:) to concat topic name and partition IDs.
>>>>>>>>>>>>>>>> Using comma(,) to concat partition IDs within same topic
>>> name.
>>>>>>>>>>>>>>>> Using semicolon(;) to concat topic strings.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> AS3: Fix it with kafka-share-groups.sh. Thanks.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If there is no further suggestion, I will start a vote
>>> thread
>>>>>>> next
>>>>>>>>>>>>>> Monday.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield <
>>>>>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>>>>> DJ2: I was just going to comment that "-" would be a more
>>>>>>>>>> appropriate
>>>>>>>>>>>>>>>> missing value, but
>>>>>>>>>>>>>>>>> you got there first.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> AS3: The examples for kafka-share-groups.sh include
>>>>>>>>>>>>>>>> kafka-consumer-groups.sh in the
>>>>>>>>>>>>>>>>> command line.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If this is accepted in time, I'm happy to pick up the
>>>>>>>>>> implementation of
>>>>>>>>>>>>>>>> the share groups
>>>>>>>>>>>>>>>>> part of this if it helps.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Andrew
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>>> From: Frank Yang <ya...@gmail.com>
>>>>>>>>>>>>>>>>> Sent: 14 November 2024 10:48
>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
>>> kafka-consumer-groups
>>>>>>>>>> command
>>>>>>>>>>>>>>>> line tool to support new consumer group
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi David,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for the review and suggestion! I would like to get
>>>> this
>>>>>>> in
>>>>>>>>>> AK
>>>>>>>>>>>>>> 4.0
>>>>>>>>>>>>>>>> as well. I will do my best.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> DJ1: Update KIP to put GROUP-EPOCH and
>>>> TARGET-ASSIGNMENT-EPOCH
>>>>>>>>>> before
>>>>>>>>>>>>>>>> #MEMBERS.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> DJ2: I prefer to follow current missing column value “-“.
>>>>>>>>>> (reference <
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>
>>>
>> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92
>>>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> DJ3: Update KIP to use CURRENT-EPOCH CURRENT-ASSIGNMENT
>>>>>>>>>> TARGET-EPOCH
>>>>>>>>>>>>>>>> TARGET-ASSIGNMENT.
>>>>>>>>>>>>>>>>> Remove GROUP-EPOCH.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> For assignment value, it follows current output
>> (reference
>>> <
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>
>>>
>> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418
>>>>>>>>>>>>>>> ).
>>>>>>>>>>>>>>>> I think the form of `topicid-partitionid` is more clear.
>>>>>>>>>>>>>>>>> If we would like to use this form, I will update both
>>> output
>>>> in
>>>>>>>>>>>>>>>> kafka-consumer-groups.sh and kafka-share-groups.sh.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> DJ4: It looks like DescribeGroupsResponseData only has
>>>> protocol
>>>>>>>>>> type at
>>>>>>>>>>>>>>>> group level.
>>>>>>>>>>>>>>>>> Both DescribeGroupsResponseData and
>>>>>>>>>> ConsumerGroupDescribeResponseData
>>>>>>>>>>>>>>>> don’t have protocol at member level.
>>>>>>>>>>>>>>>>> Could we use a followup to add it?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> DJ5: Update KIP to put LEADER-EPOCH before
>> CURRENT-OFFSET.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot
>>>>>>>>>> <djacot@confluent.io.INVALID
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks for the KIP! I have a few minor
>>> comments/suggestions:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> DJ1: In the output of `--describe --verbose`, I would
>>>> suggest
>>>>>>>>>> putting
>>>>>>>>>>>>>>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before
>>>> `#MEMBERS`.
>>>>>>>>>>>>>>>>>> DJ2: Continuing on the above, I assume that we will
>> print
>>>> out
>>>>>>> N/A
>>>>>>>>>> for
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> fields not supported by classic groups. Is this correct?
>>>>>>>>>>>>>>>>>> DJ3: In the output of `--describe --members --verbose`,
>> I
>>>>>>> wonder
>>>>>>>>>> if we
>>>>>>>>>>>>>>>>>> should use the following order and terms: CURRENT-EPOCH
>>>>>>>>>>>>>>>> CURRENT-ASSIGNMENT
>>>>>>>>>>>>>>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the
>>>> GROUP-EPOCH
>>>>>>>>>>>>>> because
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>> is already in the group description. The value `(0)`
>> used
>>>> for
>>>>>>> the
>>>>>>>>>>>>>>>>>> assignment is incorrect. Here I suppose that we will
>> print
>>>> out
>>>>>>> the
>>>>>>>>>>>>>> list
>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> partitions in the form of `topicid-partitionid`.
>>>>>>>>>>>>>>>>>> DJ4: Continuing on the above, I wonder if we should also
>>> add
>>>>>>> the
>>>>>>>>>>>>>>>> protocol
>>>>>>>>>>>>>>>>>> used `classic` or `consumer`. For context, it is
>> possible
>>> to
>>>>>>> have
>>>>>>>>>>>>>>>> `classic`
>>>>>>>>>>>>>>>>>> members and `consumer` members in a `consumer` group
>>> during
>>>> an
>>>>>>>>>> online
>>>>>>>>>>>>>>>>>> upgrade from the classic protocol to the consumer
>>> protocol.
>>>>>>> Having
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> information may be handy for administrators. What do you
>>>> think?
>>>>>>>>>>>>>>>>>> DJ5: In the output of `--describe --offsets --verbose`,
>> I
>>>> would
>>>>>>>>>>>>>> suggest
>>>>>>>>>>>>>>>>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> It would be great if we could get this one in AK 4.0 as
>> it
>>>>>>>>>> changes the
>>>>>>>>>>>>>>>>>> output of the command.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> DJ
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <
>>>> yangpoan@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Sean / Andrew / Lianet,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks for all your review and suggestions.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> AS1, LM1, LM4: Change to add KIP-848 information when
>>> users
>>>>>>> give
>>>>>>>>>>>>>>>> —verbose
>>>>>>>>>>>>>>>>>>> option.
>>>>>>>>>>>>>>>>>>> —describe —verbose: shows group epoch and target
>>> assignment
>>>>>>>>>> epoch.
>>>>>>>>>>>>>>>>>>> —describe —members —verbose: shows above information,
>>>> member
>>>>>>>>>> epoch,
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> target assignment.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> AS2: Change to use MEMBER-EPOCH to align with KIP-848
>>>>>>> definition.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> LM2: For classic group, it doesn’t have epoch, so I use
>>>>>>> Optional
>>>>>>>>>>>>>>>> fields in
>>>>>>>>>>>>>>>>>>> ConsumerGroupDescription.
>>>>>>>>>>>>>>>>>>> For share group, it relies on KIP-848. It must have
>>> epoch,
>>>> so
>>>>>>> I
>>>>>>>>>> use
>>>>>>>>>>>>>> int
>>>>>>>>>>>>>>>>>>> fields in ShareGroupDescription.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> LM3: Remove —state change. We can get group level
>>>> information
>>>>>>> by
>>>>>>>>>>>>>>>> —describe
>>>>>>>>>>>>>>>>>>> —verbose.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> SQ1: Add LEADER-EPOCH when users give —describe
>> —offsets
>>>>>>>>>> —verbose.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <
>>> lianetmr@gmail.com
>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hello Frank, thanks for the KIP! A few comments:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> LM1. I strongly agree with Andrew's suggestion of
>> moving
>>>> this
>>>>>>>>>> into a
>>>>>>>>>>>>>>>>>>>> --verbose option. I expect someone would only attempt
>> to
>>>> make
>>>>>>>>>> sense
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> epochs while debugging an issue, not while trying to
>>> view
>>>> the
>>>>>>>>>> group
>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>> member's info. (Also member-epoch makes more sense to
>>> me,
>>>>>>> even
>>>>>>>>>> more
>>>>>>>>>>>>>>>> if we
>>>>>>>>>>>>>>>>>>>> end up agreeing in a --verbose). Related to both
>> issues,
>>>> my
>>>>>>>>>> take is
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> whoever is close to the protocol will expect
>>> member-epoch,
>>>>>>>>>> whoever
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>> will probably won't even need to see the epochs at
>> all.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> LM2. Why are the epochs defined as Optional (don't we
>>>> expect
>>>>>>> to
>>>>>>>>>>>>>> always
>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>> them? with 0 initially, defined on the broker side for
>>> the
>>>>>>> group
>>>>>>>>>>>>>> ones,
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> client side for the member)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> LM3. Why is the KIP including the –state option in the
>>>>>>> proposed
>>>>>>>>>>>>>>>> changes?
>>>>>>>>>>>>>>>>>>> (
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>
>>>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
>>>>>>>>>>>>>>>>>>>> ).
>>>>>>>>>>>>>>>>>>>> I get that the output would change in that example,
>> but
>>>> it’s
>>>>>>> not
>>>>>>>>>>>>>>>> because
>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>> any change to the –state option itself. It's because
>> of
>>>> the
>>>>>>>>>> change
>>>>>>>>>>>>>>>>>>> proposed
>>>>>>>>>>>>>>>>>>>> to the –described (required with the --state), and the
>>>>>>> changes
>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> --describe are already explained above (seems
>> confusing,
>>>> got
>>>>>>> me
>>>>>>>>>>>>>>>> looking
>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>> a change to the state filter which seemed unrelated).
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> LM4. Group epoch and target assignment epoch are
>>>> conceptually
>>>>>>>>>> at the
>>>>>>>>>>>>>>>>>>> group
>>>>>>>>>>>>>>>>>>>> level. vs member epoch that lands at a member level.
>> So
>>>>>>> wonder
>>>>>>>>>> if we
>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>> show them accordingly (ex. using the --verbose option)
>>>>>>>>>>>>>>>>>>>> –describe –verbose => shows group epoch and target
>>>> assignment
>>>>>>>>>> epoch
>>>>>>>>>>>>>>>>>>>> –describe –members –verbose => shows member
>>> epoch
>>>>>>>>>> (along
>>>>>>>>>>>>>>>>>>>> with group epoch and target assignment epoch)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Lianet
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield <
>>>>>>>>>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I have a few comments.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> AS1: It seems to me that these new additions are most
>>>>>>> useful to
>>>>>>>>>>>>>>>> people
>>>>>>>>>>>>>>>>>>>>> trying to understand
>>>>>>>>>>>>>>>>>>>>> the progress of rebalancing in quite some detail. The
>>>>>>>>>> information
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> really not understandable
>>>>>>>>>>>>>>>>>>>>> for most users who do not have deep knowledge of
>>>>>>> KIP-848/932.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> As a result, I suggest for kafka-share-groups.sh that
>>> you
>>>>>>> add a
>>>>>>>>>>>>>>>>>>> --members
>>>>>>>>>>>>>>>>>>>>> --verbose option
>>>>>>>>>>>>>>>>>>>>> and only include the new information in the output
>> for
>>>> that
>>>>>>>>>> option,
>>>>>>>>>>>>>>>>>>> rather
>>>>>>>>>>>>>>>>>>>>> than changing the
>>>>>>>>>>>>>>>>>>>>> non-verbose --members output.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I also make a similar suggestion for
>>>>>>> kafka-consumer-groups.sh
>>>>>>>>>>>>>>>> --members
>>>>>>>>>>>>>>>>>>>>> and only add the
>>>>>>>>>>>>>>>>>>>>> new information for the --verbose output.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> AS2: I strongly suggest that you use MEMBER-EPOCH
>>>> instead of
>>>>>>>>>>>>>>>>>>>>> CONSUMER-EPOCH.
>>>>>>>>>>>>>>>>>>>>> I think it's more confusing not following the
>>>> terminology of
>>>>>>>>>> the
>>>>>>>>>>>>>>>> KIPs.
>>>>>>>>>>>>>>>>>>> For
>>>>>>>>>>>>>>>>>>>>> one thing,
>>>>>>>>>>>>>>>>>>>>> we already have "member" in the admin client such as
>>>>>>>>>>>>>>>> MemberDescription.
>>>>>>>>>>>>>>>>>>>>> It's not a
>>>>>>>>>>>>>>>>>>>>> new term.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> Andrew
>>>>>>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>>>>>>> From: PoAn Yang <pa...@apache.org>
>>>>>>>>>>>>>>>>>>>>> Sent: 25 October 2024 13:55
>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
>>>>>>> kafka-consumer-groups
>>>>>>>>>>>>>> command
>>>>>>>>>>>>>>>>>>> line
>>>>>>>>>>>>>>>>>>>>> tool to support new consumer group
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi Lucas,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks for the review!
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 1) Yes, I add related change for
>> kafka-share-groups.sh
>>> to
>>>>>>> the
>>>>>>>>>> KIP.
>>>>>>>>>>>>>>>> Could
>>>>>>>>>>>>>>>>>>>>> you take a look? Thanks for the suggestion.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use
>>>> MEMBER-EPOCH
>>>>>>>>>> here,
>>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>>>>>>> confuse what is different between CONSUMER and
>> MEMBER.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
>>>>>>>>>>>>>>>>>>>>>> Hi Frank,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> thanks for the KIP!
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 1) For consistency, should we do the same for
>>>>>>>>>>>>>>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription, etc. ?
>>>> Even
>>>>>>> if
>>>>>>>>>> we do
>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>> implement it right now if the share group
>>> implementation
>>>>>>> may
>>>>>>>>>> still
>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>> incomplete, it may make sense to include it in the
>>> KIP.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH?
>> That
>>>> would
>>>>>>>>>> seem
>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>>>> consistent.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>> Lucas
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <
>>>>>>>>>> yangpoan@gmail.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I would like to kick off the discussion of
>> KIP-1099.
>>>> This
>>>>>>> KIP
>>>>>>>>>>>>>>>> enhances
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups tools to include state which
>> is
>>>>>>>>>> introduced
>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>> KIP-848.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups command line
>>>> tool
>>>>>>> to
>>>>>>>>>>>>>> support
>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>> consumer group - Apache Kafka - Apache Software
>>>> Foundation
>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>
>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> cwiki.apache.org
>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>
>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> [image: favicon.ico]
>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>
>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>
>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>>
>>>
>>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by Chia-Ping Tsai <ch...@gmail.com>.
hi DJ
> How about we use another Jira to add “SHARE” to GroupProtocol and change
return type of ModernGroup#protocolType?
This is not my comment ... sorry for the incorrect quote :(
> A member in a classic group should report "classic".
I completely agree. However, in this case, ClassicGroupDescription#protocol
returns 'consumer', while ClassicGroupDescription#members[x].protocol
returns 'classic'. This inconsistency feels a bit strange to me.
Best,
Chia-Ping
David Jacot <dj...@confluent.io.invalid> 於 2024年11月22日 週五 下午6:51寫道：
> Hi Chia,
>
> > > LM4: I agree it’s better to use a type like GroupProtocol enum, but I
> would like to align the format with kafka-groups.sh.
> How about we use another Jira to add “SHARE” to GroupProtocol and change
> return type of ModernGroup#protocolType?
>
> My understanding is that `GroupProtocol` is used to drive the config of the
> `Consumer`. Hence adding `Share` seems wrong to me. Or are you referring to
> another GroupProtocol enum? Perhaps GroupType enum?
>
> > In fact, I'm wondering if we should even add 'protocol' to
> MemberDescription,
> since the field has different values between classic and consumer groups—it
> shows 'consumer' in classic groups, whereas it shows 'classic' in consumer
> groups. Yes, it seems like a typo, but it's true. This inconsistency feels
> confusing to me. For example, users call Admin#describeConsumerGroups and
> see that a member has the 'consumer' protocol when it's in a classic group.
> Then, if the group is upgraded to a consumer group, calling
> Admin#describeConsumerGroups again shows the protocol as 'classic' for the
> same member.
>
> I don't fully follow your comment so please excuse me if I say something
> wrong.
>
> I think that it should work as follows:
> * A member in a classic group should report "classic".
> * A member using the consumer protocol in a consumer group should report
> "consumer".
> * A member using the classic protocol in a consumer group should report
> "classic".
> * A member in a share group should report "share".
>
> > If all we want to do is distinguish classic members in the
> ConsumerGroupDescription, then we shouldn't modify the common
> MemberDescription structure. Instead, we could add a collection to
> ConsumerGroupDescription to trace members using the classic protocol, which
> would allow us to enrich the output of command-line tools.
>
> Personally, I believe that using different collections per protocol types
> is not necessary. A field in the member is more than enough.
>
> Best,
> DJ
>
> On Fri, Nov 22, 2024 at 12:01 PM Chia-Ping Tsai <ch...@gmail.com>
> wrote:
>
> > hi PoAn
> >
> > > LM4: I agree it’s better to use a type like GroupProtocol enum, but I
> > would like to align the format with kafka-groups.sh.
> > How about we use another Jira to add “SHARE” to GroupProtocol and change
> > return type of ModernGroup#protocolType?
> >
> > MemberDescription is also used by ClassicGroupDescription, and in classic
> > groups, the protocol can be an arbitrary string, so we can't use an enum.
> >
> > In fact, I'm wondering if we should even add 'protocol' to
> > MemberDescription,
> > since the field has different values between classic and consumer
> groups—it
> > shows 'consumer' in classic groups, whereas it shows 'classic' in
> consumer
> > groups. Yes, it seems like a typo, but it's true. This inconsistency
> feels
> > confusing to me. For example, users call Admin#describeConsumerGroups and
> > see that a member has the 'consumer' protocol when it's in a classic
> group.
> > Then, if the group is upgraded to a consumer group, calling
> > Admin#describeConsumerGroups again shows the protocol as 'classic' for
> the
> > same member.
> >
> > If all we want to do is distinguish classic members in the
> > ConsumerGroupDescription, then we shouldn't modify the common
> > MemberDescription structure. Instead, we could add a collection to
> > ConsumerGroupDescription to trace members using the classic protocol,
> which
> > would allow us to enrich the output of command-line tools.
> > Best,
> > Chia-Ping
> >
> > PoAn Yang <ya...@gmail.com> 於 2024年11月22日 週五 下午3:46寫道：
> >
> > > Hi Lianet / Jeff,
> > >
> > > Thanks for the review.
> > >
> > > LM5: The kafka-share-groups.sh also uses “—describe” to show offsets
> > > information.
> > > Change to use “—describe —state —verbose” to show group level
> information
> > > in kafka-share-groups.sh.
> > >
> > > LM6: Update the example.
> > >
> > > LM7: Add a description to mention “—verbose” is a new option in
> > > ShareGroupCommandOptions.
> > >
> > > JK2: Yes, add the statement about format change for ASSIGNMENT value.
> > >
> > > Thanks,
> > > PoAn
> > >
> > > > On Nov 22, 2024, at 7:35 AM, Jeff Kim <je...@apache.org> wrote:
> > > >
> > > > Hi PoAn,
> > > >
> > > > JK2: Can we state in the KIP that we will reformat the existing
> > > > classic group's `--members --verbose` as well, specifically the
> > > > "ASSIGNMENT" column to include the topic name?
> > > >
> > > > Thanks!
> > > >
> > > > On 2024/11/21 14:33:37 "Lianet M." wrote:
> > > >> Thanks for the updates PoAn!
> > > >>
> > > >> LM6. Just a nit, under the --describe --state --verbose the header
> is
> > > fine
> > > >> but the example is still missing the --state argument.
> > > >>
> > > >> LM7. Under the Proposed changes for kafka-share-groups.sh, should we
> > > >> clearly mention that the KIP adds a new --verbose option? I think
> it's
> > > >> confusing because it's presented as if the option already exists and
> > > we're
> > > >> only changing the output (which is the case for
> kafka-consumer-groups
> > > but
> > > >> not for kafka-share-groups)
> > > >>
> > > >> That's all on my side. Thanks!
> > > >>
> > > >> Lianet
> > > >>
> > > >> On Wed, Nov 20, 2024 at 11:12 PM PoAn Yang <ya...@gmail.com>
> > wrote:
> > > >>
> > > >>> Hi Lianet / Jeff,
> > > >>>
> > > >>> Thanks for review and suggestions.
> > > >>>
> > > >>> LM4: At group level, the protocol field is from
> > > ModernGroup#protocolType.
> > > >>> It’s a string.
> > > >>> The value is defined in different places like
> > ShareGroup#PROTOCOL_TYPE
> > > and
> > > >>> ConsumerProtocol#PROTOCOL_TYPE.
> > > >>> In KIP-1043 <
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=305171038#KIP1043:Administrationofgroups-kafka-groups.sh
> > > >,
> > > >>> the kafka-groups.sh also shows protocol like “classic” /
> “consumer” /
> > > >>> “share”.
> > > >>> I agree it’s better to use a type like GroupProtocol enum, but I
> > would
> > > >>> like to align the format with kafka-groups.sh.
> > > >>> How about we use another Jira to add “SHARE” to GroupProtocol and
> > > change
> > > >>> return type of ModernGroup#protocolType?
> > > >>>
> > > >>> LM5 / JK1: Sorry, I misunderstood LM3, so I deleted —state in the
> > > title.
> > > >>> You’re correct. I would like to use `—describe —state —verbose` to
> > show
> > > >>> group level information.
> > > >>> For `—describe —verbose`, it will show output as same as `—describe
> > > >>> —offsets —verbose`,
> > > >>> because `—describe` also shows output as same as `—describe
> > —offsets`.
> > > >>>
> > > >>> JK2: Yes, I only use consumer group in the sample output, but the
> > > classic
> > > >>> group will use the same format as well.
> > > >>>
> > > >>> Thanks,
> > > >>> PoAn
> > > >>>
> > > >>>> On Nov 21, 2024, at 8:28 AM, Jeff Kim <je...@apache.org>
> wrote:
> > > >>>>
> > > >>>> Hi PoAn,
> > > >>>>
> > > >>>> Thanks for the KIP. I have some questions:
> > > >>>>
> > > >>>> JK1: (Realized this is similar to LM5) On `--describe --verbose`
> > > >>> proposed changes - doesn't `--describe $group` default to printing
> > the
> > > >>> offsets? Perhaps you're referring to the `--state` argument? Also,
> > > would
> > > >>> that mean the default `--describe $group --verbose` command would
> not
> > > print
> > > >>> the added field to `--offsets verbose` (leader-epoch) or would it?
> > > >>>>
> > > >>>> JK2: Looking at ConsumerGroupCommand.java, the existing
> "ASSIGNMENT"
> > > >>> column under `--members --verbose` does group the topic partitions
> by
> > > topic
> > > >>> but does not prefix the grouping with the topic name like you're
> > > proposal:
> > > >>> "my_topic:0,1;new_topic:0,1". Should we do apply the same format
> for
> > > the
> > > >>> classic group as well?
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On 2024/11/20 21:32:41 "Lianet M." wrote:
> > > >>>>> Hello PoAn, just of couple more minor comments:
> > > >>>>>
> > > >>>>> LM4. Regarding the new “protocol” field added to
> MemberDescription.
> > > >>> Should
> > > >>>>> we consider reusing the existing GroupProtocol enum instead of
> > > String?
> > > >>>>> (It’s the one we use from the consumer side to refer to the
> > protocol
> > > in
> > > >>>>> use, just missing Share I notice).
> > > >>>>>
> > > >>>>> LM5. Regarding the change to the output for
> kafka-consumer-groups,
> > > the
> > > >>>>> command shown does not include the —state option, but the output
> > > shows
> > > >>>>> state info (state, #members, epochs). I would guess that we want
> to
> > > >>> modify
> > > >>>>> the output only when we describe a group with the —state —verbose
> > > >>> option,
> > > >>>>> is my understanding right? If my understanding is right we’re
> just
> > > >>> missing
> > > >>>>> adding the —state in the example, and the KIP does not introduce
> > any
> > > >>>>> changes to the —describe —verbose option. (If not, it would mean
> a
> > > >>> bigger
> > > >>>>> change to the output of —describe —verbose which I expect is not
> > the
> > > >>>>> intention?)
> > > >>>>>
> > > >>>>> Thanks!
> > > >>>>>
> > > >>>>> Lianet
> > > >>>>>
> > > >>>>>
> > > >>>>> On Wed, Nov 20, 2024 at 2:12 AM PoAn Yang <ya...@gmail.com>
> > > wrote:
> > > >>>>>
> > > >>>>>> Hi Chia-Ping,
> > > >>>>>>
> > > >>>>>> Thanks for the review and suggestions.
> > > >>>>>>
> > > >>>>>> CI0: Thanks for the reminder. Update validVersions in
> > > >>>>>> ConsumerGroupDescribeRequest to 0-1.
> > > >>>>>>
> > > >>>>>> CI1: Yes, I use `ConsumerGroupMember#useClassicProtocol` to
> check
> > > >>> whether
> > > >>>>>> a member in consumer group uses “classic” or “consumer”
> protocol.
> > > >>>>>>
> > > >>>>>> CI2: Yes, a member in share group always uses “share” protocol.
> > > >>>>>>
> > > >>>>>> CI3: Add a table to show meaning of “classic”, “consumer”, and
> > > “share”
> > > >>>>>> protocol.
> > > >>>>>>
> > > >>>>>> BTW, the vote thread is in
> > > >>>>>>
> https://lists.apache.org/thread/rb25tf75tzf4c7jqqldlo5jh9w8chsq6.
> > > >>>>>>
> > > >>>>>> Thanks,
> > > >>>>>> PoAn
> > > >>>>>>
> > > >>>>>>> On Nov 20, 2024, at 11:46 AM, Chia-Ping Tsai <
> > chia7712@apache.org>
> > > >>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>> hi PoAn
> > > >>>>>>>
> > > >>>>>>> CI0: We have to bump the version of
> ConsumerGroupDescribeRequest
> > > as
> > > >>>>>> well, so server can distinguish the new and old behavior.
> > > >>>>>>>
> > > >>>>>>> CI1: the type of new filed is string, so I guess you plan to
> use
> > > >>>>>> `ConsumerGroupMember#useClassicProtocol` [0] flag to return
> either
> > > >>>>>> "classic" or "consumer", right?
> > > >>>>>>>
> > > >>>>>>> CI2: `MemberDescription` is used by `ShareGroupDescription`
> too,
> > so
> > > >>> the
> > > >>>>>> new filed protocol in shared group is always "shared", right?
> > > >>>>>>>
> > > >>>>>>> CI3: Could you consider adding a table to show the value of the
> > > >>> protocol
> > > >>>>>> field in each case? Andrew has a beautiful table in KIP-1043
> that
> > > >>> lists all
> > > >>>>>> possible protocol names.
> > > >>>>>>>
> > > >>>>>>> [0]
> > > >>>>>>
> > > >>>
> > >
> >
> https://github.com/apache/kafka/blob/441a6d0b790f4a17b454caeea7588a6b90fbd9db/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/consumer/ConsumerGroupMember.java#L454
> > > >>>>>>
> > > >>>>>>>
> > > >>>>>>> Best,
> > > >>>>>>> Chia-Ping
> > > >>>>>>>
> > > >>>>>>> On 2024/11/19 15:45:16 PoAn Yang wrote:
> > > >>>>>>>> Hi David,
> > > >>>>>>>>
> > > >>>>>>>> Thanks for the reminder.
> > > >>>>>>>>
> > > >>>>>>>> DJ4: Add related content to the KIP.
> > > >>>>>>>> Bump validVersions in ConsumerGroupDescribeResponse to “0-1”.
> > > >>>>>>>> Add a new field “Protocol” to ConsumerGroupDescribeResponse.
> > > >>>>>>>>
> > > >>>>>>>> Thanks,
> > > >>>>>>>> PoAn
> > > >>>>>>>>
> > > >>>>>>>>> On Nov 19, 2024, at 3:57 PM, David Jacot
> > > >>> <dj...@confluent.io.INVALID>
> > > >>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>> Hi PoAn,
> > > >>>>>>>>>
> > > >>>>>>>>> DJ4: Could you please also include the required changes to
> the
> > > >>>>>>>>> ConsumerGroupDescribed API in the public interface section?
> We
> > > >>>>>> basically
> > > >>>>>>>>> need to bump the version, add the new field, etc.
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks,
> > > >>>>>>>>> David
> > > >>>>>>>>>
> > > >>>>>>>>> On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang <
> yangpoan@gmail.com>
> > > >>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> Hi David,
> > > >>>>>>>>>>
> > > >>>>>>>>>> DJ4: Add a new field protocol to MemberDescription,
> > > >>>>>>>>>> so the command line tool can show protocol information when
> > > users
> > > >>>>>> describe
> > > >>>>>>>>>> members.
> > > >>>>>>>>>>
> > > >>>>>>>>>> If there is no further suggestion, I will start a vote
> thread
> > > >>> today.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Thanks,
> > > >>>>>>>>>> PoAn
> > > >>>>>>>>>>
> > > >>>>>>>>>>> On Nov 15, 2024, at 11:50 PM, David Jacot
> > > >>>>>> <dj...@confluent.io.INVALID>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Hi,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> DJ2: Using "-" sounds good to me.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> DJ3: That seems reasonable to me.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> DJ4: Why not add it right now? I don't want to change the
> > > output
> > > >>> of
> > > >>>>>> the
> > > >>>>>>>>>>> tool too many times.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Best,
> > > >>>>>>>>>>> David
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang <
> > yangpoan@gmail.com>
> > > >>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Hi David / Andrew,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Thanks for review. Thanks Andrew for picking up
> > > >>>>>> kafka-share-groups.sh
> > > >>>>>>>>>>>> implementation.
> > > >>>>>>>>>>>> I will handle kafka-consumer-groups.sh.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> DJ3: After discussing with @Chia-Ping Tsai, we think that
> > > using
> > > >>> new
> > > >>>>>>>>>> format
> > > >>>>>>>>>>>> is more clear.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> The new format will be like
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>
> > > >>>
> > >
> >
> <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Using colon(:) to concat topic name and partition IDs.
> > > >>>>>>>>>>>> Using comma(,) to concat partition IDs within same topic
> > name.
> > > >>>>>>>>>>>> Using semicolon(;) to concat topic strings.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> AS3: Fix it with kafka-share-groups.sh. Thanks.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> If there is no further suggestion, I will start a vote
> > thread
> > > >>> next
> > > >>>>>>>>>> Monday.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>> PoAn
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield <
> > > >>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Hi PoAn,
> > > >>>>>>>>>>>>> DJ2: I was just going to comment that "-" would be a more
> > > >>>>>> appropriate
> > > >>>>>>>>>>>> missing value, but
> > > >>>>>>>>>>>>> you got there first.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> AS3: The examples for kafka-share-groups.sh include
> > > >>>>>>>>>>>> kafka-consumer-groups.sh in the
> > > >>>>>>>>>>>>> command line.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> If this is accepted in time, I'm happy to pick up the
> > > >>>>>> implementation of
> > > >>>>>>>>>>>> the share groups
> > > >>>>>>>>>>>>> part of this if it helps.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>> Andrew
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> ________________________________________
> > > >>>>>>>>>>>>> From: Frank Yang <ya...@gmail.com>
> > > >>>>>>>>>>>>> Sent: 14 November 2024 10:48
> > > >>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
> > > >>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
> > kafka-consumer-groups
> > > >>>>>> command
> > > >>>>>>>>>>>> line tool to support new consumer group
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Hi David,
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Thanks for the review and suggestion! I would like to get
> > > this
> > > >>> in
> > > >>>>>> AK
> > > >>>>>>>>>> 4.0
> > > >>>>>>>>>>>> as well. I will do my best.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> DJ1: Update KIP to put GROUP-EPOCH and
> > > TARGET-ASSIGNMENT-EPOCH
> > > >>>>>> before
> > > >>>>>>>>>>>> #MEMBERS.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> DJ2: I prefer to follow current missing column value “-“.
> > > >>>>>> (reference <
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>
> > > >>>
> > >
> >
> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92
> > > >>>>>>>>>>>>> )
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> DJ3: Update KIP to use CURRENT-EPOCH CURRENT-ASSIGNMENT
> > > >>>>>> TARGET-EPOCH
> > > >>>>>>>>>>>> TARGET-ASSIGNMENT.
> > > >>>>>>>>>>>>> Remove GROUP-EPOCH.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> For assignment value, it follows current output
> (reference
> > <
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>
> > > >>>
> > >
> >
> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418
> > > >>>>>>>>>>> ).
> > > >>>>>>>>>>>> I think the form of `topicid-partitionid` is more clear.
> > > >>>>>>>>>>>>> If we would like to use this form, I will update both
> > output
> > > in
> > > >>>>>>>>>>>> kafka-consumer-groups.sh and kafka-share-groups.sh.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> DJ4: It looks like DescribeGroupsResponseData only has
> > > protocol
> > > >>>>>> type at
> > > >>>>>>>>>>>> group level.
> > > >>>>>>>>>>>>> Both DescribeGroupsResponseData and
> > > >>>>>> ConsumerGroupDescribeResponseData
> > > >>>>>>>>>>>> don’t have protocol at member level.
> > > >>>>>>>>>>>>> Could we use a followup to add it?
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> DJ5: Update KIP to put LEADER-EPOCH before
> CURRENT-OFFSET.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>> PoAn
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot
> > > >>>>>> <djacot@confluent.io.INVALID
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Hi PoAn,
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Thanks for the KIP! I have a few minor
> > comments/suggestions:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> DJ1: In the output of `--describe --verbose`, I would
> > > suggest
> > > >>>>>> putting
> > > >>>>>>>>>>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before
> > > `#MEMBERS`.
> > > >>>>>>>>>>>>>> DJ2: Continuing on the above, I assume that we will
> print
> > > out
> > > >>> N/A
> > > >>>>>> for
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>> fields not supported by classic groups. Is this correct?
> > > >>>>>>>>>>>>>> DJ3: In the output of `--describe --members --verbose`,
> I
> > > >>> wonder
> > > >>>>>> if we
> > > >>>>>>>>>>>>>> should use the following order and terms: CURRENT-EPOCH
> > > >>>>>>>>>>>> CURRENT-ASSIGNMENT
> > > >>>>>>>>>>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the
> > > GROUP-EPOCH
> > > >>>>>>>>>> because
> > > >>>>>>>>>>>> it
> > > >>>>>>>>>>>>>> is already in the group description. The value `(0)`
> used
> > > for
> > > >>> the
> > > >>>>>>>>>>>>>> assignment is incorrect. Here I suppose that we will
> print
> > > out
> > > >>> the
> > > >>>>>>>>>> list
> > > >>>>>>>>>>>> of
> > > >>>>>>>>>>>>>> partitions in the form of `topicid-partitionid`.
> > > >>>>>>>>>>>>>> DJ4: Continuing on the above, I wonder if we should also
> > add
> > > >>> the
> > > >>>>>>>>>>>> protocol
> > > >>>>>>>>>>>>>> used `classic` or `consumer`. For context, it is
> possible
> > to
> > > >>> have
> > > >>>>>>>>>>>> `classic`
> > > >>>>>>>>>>>>>> members and `consumer` members in a `consumer` group
> > during
> > > an
> > > >>>>>> online
> > > >>>>>>>>>>>>>> upgrade from the classic protocol to the consumer
> > protocol.
> > > >>> Having
> > > >>>>>>>>>> this
> > > >>>>>>>>>>>>>> information may be handy for administrators. What do you
> > > think?
> > > >>>>>>>>>>>>>> DJ5: In the output of `--describe --offsets --verbose`,
> I
> > > would
> > > >>>>>>>>>> suggest
> > > >>>>>>>>>>>>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> It would be great if we could get this one in AK 4.0 as
> it
> > > >>>>>> changes the
> > > >>>>>>>>>>>>>> output of the command.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>> DJ
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <
> > > yangpoan@gmail.com>
> > > >>>>>> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Hi Sean / Andrew / Lianet,
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Thanks for all your review and suggestions.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> AS1, LM1, LM4: Change to add KIP-848 information when
> > users
> > > >>> give
> > > >>>>>>>>>>>> —verbose
> > > >>>>>>>>>>>>>>> option.
> > > >>>>>>>>>>>>>>> —describe —verbose: shows group epoch and target
> > assignment
> > > >>>>>> epoch.
> > > >>>>>>>>>>>>>>> —describe —members —verbose: shows above information,
> > > member
> > > >>>>>> epoch,
> > > >>>>>>>>>> and
> > > >>>>>>>>>>>>>>> target assignment.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> AS2: Change to use MEMBER-EPOCH to align with KIP-848
> > > >>> definition.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> LM2: For classic group, it doesn’t have epoch, so I use
> > > >>> Optional
> > > >>>>>>>>>>>> fields in
> > > >>>>>>>>>>>>>>> ConsumerGroupDescription.
> > > >>>>>>>>>>>>>>> For share group, it relies on KIP-848. It must have
> > epoch,
> > > so
> > > >>> I
> > > >>>>>> use
> > > >>>>>>>>>> int
> > > >>>>>>>>>>>>>>> fields in ShareGroupDescription.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> LM3: Remove —state change. We can get group level
> > > information
> > > >>> by
> > > >>>>>>>>>>>> —describe
> > > >>>>>>>>>>>>>>> —verbose.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> SQ1: Add LEADER-EPOCH when users give —describe
> —offsets
> > > >>>>>> —verbose.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Thanks.
> > > >>>>>>>>>>>>>>> PoAn
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <
> > lianetmr@gmail.com
> > > >
> > > >>>>>> wrote:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Hello Frank, thanks for the KIP! A few comments:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> LM1. I strongly agree with Andrew's suggestion of
> moving
> > > this
> > > >>>>>> into a
> > > >>>>>>>>>>>>>>>> --verbose option. I expect someone would only attempt
> to
> > > make
> > > >>>>>> sense
> > > >>>>>>>>>> of
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> epochs while debugging an issue, not while trying to
> > view
> > > the
> > > >>>>>> group
> > > >>>>>>>>>> or
> > > >>>>>>>>>>>>>>>> member's info. (Also member-epoch makes more sense to
> > me,
> > > >>> even
> > > >>>>>> more
> > > >>>>>>>>>>>> if we
> > > >>>>>>>>>>>>>>>> end up agreeing in a --verbose). Related to both
> issues,
> > > my
> > > >>>>>> take is
> > > >>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>> whoever is close to the protocol will expect
> > member-epoch,
> > > >>>>>> whoever
> > > >>>>>>>>>> is
> > > >>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>>> will probably won't even need to see the epochs at
> all.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> LM2. Why are the epochs defined as Optional (don't we
> > > expect
> > > >>> to
> > > >>>>>>>>>> always
> > > >>>>>>>>>>>>>>> have
> > > >>>>>>>>>>>>>>>> them? with 0 initially, defined on the broker side for
> > the
> > > >>> group
> > > >>>>>>>>>> ones,
> > > >>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>> client side for the member)
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> LM3. Why is the KIP including the –state option in the
> > > >>> proposed
> > > >>>>>>>>>>>> changes?
> > > >>>>>>>>>>>>>>> (
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
> > > >>>>>>>>>>>>>>>> ).
> > > >>>>>>>>>>>>>>>> I get that the output would change in that example,
> but
> > > it’s
> > > >>> not
> > > >>>>>>>>>>>> because
> > > >>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>> any change to the –state option itself. It's because
> of
> > > the
> > > >>>>>> change
> > > >>>>>>>>>>>>>>> proposed
> > > >>>>>>>>>>>>>>>> to the –described (required with the --state), and the
> > > >>> changes
> > > >>>>>> to
> > > >>>>>>>>>>>>>>>> --describe are already explained above (seems
> confusing,
> > > got
> > > >>> me
> > > >>>>>>>>>>>> looking
> > > >>>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>> a change to the state filter which seemed unrelated).
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> LM4. Group epoch and target assignment epoch are
> > > conceptually
> > > >>>>>> at the
> > > >>>>>>>>>>>>>>> group
> > > >>>>>>>>>>>>>>>> level. vs member epoch that lands at a member level.
> So
> > > >>> wonder
> > > >>>>>> if we
> > > >>>>>>>>>>>>>>> should
> > > >>>>>>>>>>>>>>>> show them accordingly (ex. using the --verbose option)
> > > >>>>>>>>>>>>>>>> –describe –verbose => shows group epoch and target
> > > assignment
> > > >>>>>> epoch
> > > >>>>>>>>>>>>>>>> –describe –members –verbose => shows member
> > epoch
> > > >>>>>> (along
> > > >>>>>>>>>>>>>>>> with group epoch and target assignment epoch)
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Thanks!
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Lianet
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield <
> > > >>>>>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Hi PoAn,
> > > >>>>>>>>>>>>>>>>> Thanks for the KIP. I have a few comments.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> AS1: It seems to me that these new additions are most
> > > >>> useful to
> > > >>>>>>>>>>>> people
> > > >>>>>>>>>>>>>>>>> trying to understand
> > > >>>>>>>>>>>>>>>>> the progress of rebalancing in quite some detail. The
> > > >>>>>> information
> > > >>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>> really not understandable
> > > >>>>>>>>>>>>>>>>> for most users who do not have deep knowledge of
> > > >>> KIP-848/932.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> As a result, I suggest for kafka-share-groups.sh that
> > you
> > > >>> add a
> > > >>>>>>>>>>>>>>> --members
> > > >>>>>>>>>>>>>>>>> --verbose option
> > > >>>>>>>>>>>>>>>>> and only include the new information in the output
> for
> > > that
> > > >>>>>> option,
> > > >>>>>>>>>>>>>>> rather
> > > >>>>>>>>>>>>>>>>> than changing the
> > > >>>>>>>>>>>>>>>>> non-verbose --members output.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> I also make a similar suggestion for
> > > >>> kafka-consumer-groups.sh
> > > >>>>>>>>>>>> --members
> > > >>>>>>>>>>>>>>>>> and only add the
> > > >>>>>>>>>>>>>>>>> new information for the --verbose output.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> AS2: I strongly suggest that you use MEMBER-EPOCH
> > > instead of
> > > >>>>>>>>>>>>>>>>> CONSUMER-EPOCH.
> > > >>>>>>>>>>>>>>>>> I think it's more confusing not following the
> > > terminology of
> > > >>>>>> the
> > > >>>>>>>>>>>> KIPs.
> > > >>>>>>>>>>>>>>> For
> > > >>>>>>>>>>>>>>>>> one thing,
> > > >>>>>>>>>>>>>>>>> we already have "member" in the admin client such as
> > > >>>>>>>>>>>> MemberDescription.
> > > >>>>>>>>>>>>>>>>> It's not a
> > > >>>>>>>>>>>>>>>>> new term.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>> Andrew
> > > >>>>>>>>>>>>>>>>> ________________________________________
> > > >>>>>>>>>>>>>>>>> From: PoAn Yang <pa...@apache.org>
> > > >>>>>>>>>>>>>>>>> Sent: 25 October 2024 13:55
> > > >>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
> > > >>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
> > > >>> kafka-consumer-groups
> > > >>>>>>>>>> command
> > > >>>>>>>>>>>>>>> line
> > > >>>>>>>>>>>>>>>>> tool to support new consumer group
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Hi Lucas,
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Thanks for the review!
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> 1) Yes, I add related change for
> kafka-share-groups.sh
> > to
> > > >>> the
> > > >>>>>> KIP.
> > > >>>>>>>>>>>> Could
> > > >>>>>>>>>>>>>>>>> you take a look? Thanks for the suggestion.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use
> > > MEMBER-EPOCH
> > > >>>>>> here,
> > > >>>>>>>>>>>> users
> > > >>>>>>>>>>>>>>> may
> > > >>>>>>>>>>>>>>>>> confuse what is different between CONSUMER and
> MEMBER.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>> PoAn
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
> > > >>>>>>>>>>>>>>>>>> Hi Frank,
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> thanks for the KIP!
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> 1) For consistency, should we do the same for
> > > >>>>>>>>>>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription, etc. ?
> > > Even
> > > >>> if
> > > >>>>>> we do
> > > >>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>>>>> implement it right now if the share group
> > implementation
> > > >>> may
> > > >>>>>> still
> > > >>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>> incomplete, it may make sense to include it in the
> > KIP.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH?
> That
> > > would
> > > >>>>>> seem
> > > >>>>>>>>>>>> more
> > > >>>>>>>>>>>>>>>>>> consistent.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>>> Lucas
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <
> > > >>>>>> yangpoan@gmail.com>
> > > >>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Hi all,
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> I would like to kick off the discussion of
> KIP-1099.
> > > This
> > > >>> KIP
> > > >>>>>>>>>>>> enhances
> > > >>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>> kafka-consumer-groups tools to include state which
> is
> > > >>>>>> introduced
> > > >>>>>>>>>> by
> > > >>>>>>>>>>>>>>>>> KIP-848.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups command line
> > > tool
> > > >>> to
> > > >>>>>>>>>> support
> > > >>>>>>>>>>>>>>> new
> > > >>>>>>>>>>>>>>>>>>> consumer group - Apache Kafka - Apache Software
> > > Foundation
> > > >>>>>>>>>>>>>>>>>>> <
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> cwiki.apache.org
> > > >>>>>>>>>>>>>>>>>>> <
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> [image: favicon.ico]
> > > >>>>>>>>>>>>>>>>>>> <
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> <
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Thank you,
> > > >>>>>>>>>>>>>>>>>>> PoAn
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>
> > > >>>
> > > >>
> > >
> > >
> >
>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by David Jacot <dj...@confluent.io.INVALID>.
Hi Chia,
> > LM4: I agree it’s better to use a type like GroupProtocol enum, but I
would like to align the format with kafka-groups.sh.
How about we use another Jira to add “SHARE” to GroupProtocol and change
return type of ModernGroup#protocolType?
My understanding is that `GroupProtocol` is used to drive the config of the
`Consumer`. Hence adding `Share` seems wrong to me. Or are you referring to
another GroupProtocol enum? Perhaps GroupType enum?
> In fact, I'm wondering if we should even add 'protocol' to
MemberDescription,
since the field has different values between classic and consumer groups—it
shows 'consumer' in classic groups, whereas it shows 'classic' in consumer
groups. Yes, it seems like a typo, but it's true. This inconsistency feels
confusing to me. For example, users call Admin#describeConsumerGroups and
see that a member has the 'consumer' protocol when it's in a classic group.
Then, if the group is upgraded to a consumer group, calling
Admin#describeConsumerGroups again shows the protocol as 'classic' for the
same member.
I don't fully follow your comment so please excuse me if I say something
wrong.
I think that it should work as follows:
* A member in a classic group should report "classic".
* A member using the consumer protocol in a consumer group should report
"consumer".
* A member using the classic protocol in a consumer group should report
"classic".
* A member in a share group should report "share".
> If all we want to do is distinguish classic members in the
ConsumerGroupDescription, then we shouldn't modify the common
MemberDescription structure. Instead, we could add a collection to
ConsumerGroupDescription to trace members using the classic protocol, which
would allow us to enrich the output of command-line tools.
Personally, I believe that using different collections per protocol types
is not necessary. A field in the member is more than enough.
Best,
DJ
On Fri, Nov 22, 2024 at 12:01 PM Chia-Ping Tsai <ch...@gmail.com> wrote:
> hi PoAn
>
> > LM4: I agree it’s better to use a type like GroupProtocol enum, but I
> would like to align the format with kafka-groups.sh.
> How about we use another Jira to add “SHARE” to GroupProtocol and change
> return type of ModernGroup#protocolType?
>
> MemberDescription is also used by ClassicGroupDescription, and in classic
> groups, the protocol can be an arbitrary string, so we can't use an enum.
>
> In fact, I'm wondering if we should even add 'protocol' to
> MemberDescription,
> since the field has different values between classic and consumer groups—it
> shows 'consumer' in classic groups, whereas it shows 'classic' in consumer
> groups. Yes, it seems like a typo, but it's true. This inconsistency feels
> confusing to me. For example, users call Admin#describeConsumerGroups and
> see that a member has the 'consumer' protocol when it's in a classic group.
> Then, if the group is upgraded to a consumer group, calling
> Admin#describeConsumerGroups again shows the protocol as 'classic' for the
> same member.
>
> If all we want to do is distinguish classic members in the
> ConsumerGroupDescription, then we shouldn't modify the common
> MemberDescription structure. Instead, we could add a collection to
> ConsumerGroupDescription to trace members using the classic protocol, which
> would allow us to enrich the output of command-line tools.
> Best,
> Chia-Ping
>
> PoAn Yang <ya...@gmail.com> 於 2024年11月22日 週五 下午3:46寫道：
>
> > Hi Lianet / Jeff,
> >
> > Thanks for the review.
> >
> > LM5: The kafka-share-groups.sh also uses “—describe” to show offsets
> > information.
> > Change to use “—describe —state —verbose” to show group level information
> > in kafka-share-groups.sh.
> >
> > LM6: Update the example.
> >
> > LM7: Add a description to mention “—verbose” is a new option in
> > ShareGroupCommandOptions.
> >
> > JK2: Yes, add the statement about format change for ASSIGNMENT value.
> >
> > Thanks,
> > PoAn
> >
> > > On Nov 22, 2024, at 7:35 AM, Jeff Kim <je...@apache.org> wrote:
> > >
> > > Hi PoAn,
> > >
> > > JK2: Can we state in the KIP that we will reformat the existing
> > > classic group's `--members --verbose` as well, specifically the
> > > "ASSIGNMENT" column to include the topic name?
> > >
> > > Thanks!
> > >
> > > On 2024/11/21 14:33:37 "Lianet M." wrote:
> > >> Thanks for the updates PoAn!
> > >>
> > >> LM6. Just a nit, under the --describe --state --verbose the header is
> > fine
> > >> but the example is still missing the --state argument.
> > >>
> > >> LM7. Under the Proposed changes for kafka-share-groups.sh, should we
> > >> clearly mention that the KIP adds a new --verbose option? I think it's
> > >> confusing because it's presented as if the option already exists and
> > we're
> > >> only changing the output (which is the case for kafka-consumer-groups
> > but
> > >> not for kafka-share-groups)
> > >>
> > >> That's all on my side. Thanks!
> > >>
> > >> Lianet
> > >>
> > >> On Wed, Nov 20, 2024 at 11:12 PM PoAn Yang <ya...@gmail.com>
> wrote:
> > >>
> > >>> Hi Lianet / Jeff,
> > >>>
> > >>> Thanks for review and suggestions.
> > >>>
> > >>> LM4: At group level, the protocol field is from
> > ModernGroup#protocolType.
> > >>> It’s a string.
> > >>> The value is defined in different places like
> ShareGroup#PROTOCOL_TYPE
> > and
> > >>> ConsumerProtocol#PROTOCOL_TYPE.
> > >>> In KIP-1043 <
> > >>>
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=305171038#KIP1043:Administrationofgroups-kafka-groups.sh
> > >,
> > >>> the kafka-groups.sh also shows protocol like “classic” / “consumer” /
> > >>> “share”.
> > >>> I agree it’s better to use a type like GroupProtocol enum, but I
> would
> > >>> like to align the format with kafka-groups.sh.
> > >>> How about we use another Jira to add “SHARE” to GroupProtocol and
> > change
> > >>> return type of ModernGroup#protocolType?
> > >>>
> > >>> LM5 / JK1: Sorry, I misunderstood LM3, so I deleted —state in the
> > title.
> > >>> You’re correct. I would like to use `—describe —state —verbose` to
> show
> > >>> group level information.
> > >>> For `—describe —verbose`, it will show output as same as `—describe
> > >>> —offsets —verbose`,
> > >>> because `—describe` also shows output as same as `—describe
> —offsets`.
> > >>>
> > >>> JK2: Yes, I only use consumer group in the sample output, but the
> > classic
> > >>> group will use the same format as well.
> > >>>
> > >>> Thanks,
> > >>> PoAn
> > >>>
> > >>>> On Nov 21, 2024, at 8:28 AM, Jeff Kim <je...@apache.org> wrote:
> > >>>>
> > >>>> Hi PoAn,
> > >>>>
> > >>>> Thanks for the KIP. I have some questions:
> > >>>>
> > >>>> JK1: (Realized this is similar to LM5) On `--describe --verbose`
> > >>> proposed changes - doesn't `--describe $group` default to printing
> the
> > >>> offsets? Perhaps you're referring to the `--state` argument? Also,
> > would
> > >>> that mean the default `--describe $group --verbose` command would not
> > print
> > >>> the added field to `--offsets verbose` (leader-epoch) or would it?
> > >>>>
> > >>>> JK2: Looking at ConsumerGroupCommand.java, the existing "ASSIGNMENT"
> > >>> column under `--members --verbose` does group the topic partitions by
> > topic
> > >>> but does not prefix the grouping with the topic name like you're
> > proposal:
> > >>> "my_topic:0,1;new_topic:0,1". Should we do apply the same format for
> > the
> > >>> classic group as well?
> > >>>>
> > >>>>
> > >>>>
> > >>>> On 2024/11/20 21:32:41 "Lianet M." wrote:
> > >>>>> Hello PoAn, just of couple more minor comments:
> > >>>>>
> > >>>>> LM4. Regarding the new “protocol” field added to MemberDescription.
> > >>> Should
> > >>>>> we consider reusing the existing GroupProtocol enum instead of
> > String?
> > >>>>> (It’s the one we use from the consumer side to refer to the
> protocol
> > in
> > >>>>> use, just missing Share I notice).
> > >>>>>
> > >>>>> LM5. Regarding the change to the output for kafka-consumer-groups,
> > the
> > >>>>> command shown does not include the —state option, but the output
> > shows
> > >>>>> state info (state, #members, epochs). I would guess that we want to
> > >>> modify
> > >>>>> the output only when we describe a group with the —state —verbose
> > >>> option,
> > >>>>> is my understanding right? If my understanding is right we’re just
> > >>> missing
> > >>>>> adding the —state in the example, and the KIP does not introduce
> any
> > >>>>> changes to the —describe —verbose option. (If not, it would mean a
> > >>> bigger
> > >>>>> change to the output of —describe —verbose which I expect is not
> the
> > >>>>> intention?)
> > >>>>>
> > >>>>> Thanks!
> > >>>>>
> > >>>>> Lianet
> > >>>>>
> > >>>>>
> > >>>>> On Wed, Nov 20, 2024 at 2:12 AM PoAn Yang <ya...@gmail.com>
> > wrote:
> > >>>>>
> > >>>>>> Hi Chia-Ping,
> > >>>>>>
> > >>>>>> Thanks for the review and suggestions.
> > >>>>>>
> > >>>>>> CI0: Thanks for the reminder. Update validVersions in
> > >>>>>> ConsumerGroupDescribeRequest to 0-1.
> > >>>>>>
> > >>>>>> CI1: Yes, I use `ConsumerGroupMember#useClassicProtocol` to check
> > >>> whether
> > >>>>>> a member in consumer group uses “classic” or “consumer” protocol.
> > >>>>>>
> > >>>>>> CI2: Yes, a member in share group always uses “share” protocol.
> > >>>>>>
> > >>>>>> CI3: Add a table to show meaning of “classic”, “consumer”, and
> > “share”
> > >>>>>> protocol.
> > >>>>>>
> > >>>>>> BTW, the vote thread is in
> > >>>>>> https://lists.apache.org/thread/rb25tf75tzf4c7jqqldlo5jh9w8chsq6.
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> PoAn
> > >>>>>>
> > >>>>>>> On Nov 20, 2024, at 11:46 AM, Chia-Ping Tsai <
> chia7712@apache.org>
> > >>>>>> wrote:
> > >>>>>>>
> > >>>>>>> hi PoAn
> > >>>>>>>
> > >>>>>>> CI0: We have to bump the version of ConsumerGroupDescribeRequest
> > as
> > >>>>>> well, so server can distinguish the new and old behavior.
> > >>>>>>>
> > >>>>>>> CI1: the type of new filed is string, so I guess you plan to use
> > >>>>>> `ConsumerGroupMember#useClassicProtocol` [0] flag to return either
> > >>>>>> "classic" or "consumer", right?
> > >>>>>>>
> > >>>>>>> CI2: `MemberDescription` is used by `ShareGroupDescription` too,
> so
> > >>> the
> > >>>>>> new filed protocol in shared group is always "shared", right?
> > >>>>>>>
> > >>>>>>> CI3: Could you consider adding a table to show the value of the
> > >>> protocol
> > >>>>>> field in each case? Andrew has a beautiful table in KIP-1043 that
> > >>> lists all
> > >>>>>> possible protocol names.
> > >>>>>>>
> > >>>>>>> [0]
> > >>>>>>
> > >>>
> >
> https://github.com/apache/kafka/blob/441a6d0b790f4a17b454caeea7588a6b90fbd9db/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/consumer/ConsumerGroupMember.java#L454
> > >>>>>>
> > >>>>>>>
> > >>>>>>> Best,
> > >>>>>>> Chia-Ping
> > >>>>>>>
> > >>>>>>> On 2024/11/19 15:45:16 PoAn Yang wrote:
> > >>>>>>>> Hi David,
> > >>>>>>>>
> > >>>>>>>> Thanks for the reminder.
> > >>>>>>>>
> > >>>>>>>> DJ4: Add related content to the KIP.
> > >>>>>>>> Bump validVersions in ConsumerGroupDescribeResponse to “0-1”.
> > >>>>>>>> Add a new field “Protocol” to ConsumerGroupDescribeResponse.
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>> PoAn
> > >>>>>>>>
> > >>>>>>>>> On Nov 19, 2024, at 3:57 PM, David Jacot
> > >>> <dj...@confluent.io.INVALID>
> > >>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>> Hi PoAn,
> > >>>>>>>>>
> > >>>>>>>>> DJ4: Could you please also include the required changes to the
> > >>>>>>>>> ConsumerGroupDescribed API in the public interface section? We
> > >>>>>> basically
> > >>>>>>>>> need to bump the version, add the new field, etc.
> > >>>>>>>>>
> > >>>>>>>>> Thanks,
> > >>>>>>>>> David
> > >>>>>>>>>
> > >>>>>>>>> On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang <ya...@gmail.com>
> > >>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Hi David,
> > >>>>>>>>>>
> > >>>>>>>>>> DJ4: Add a new field protocol to MemberDescription,
> > >>>>>>>>>> so the command line tool can show protocol information when
> > users
> > >>>>>> describe
> > >>>>>>>>>> members.
> > >>>>>>>>>>
> > >>>>>>>>>> If there is no further suggestion, I will start a vote thread
> > >>> today.
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks,
> > >>>>>>>>>> PoAn
> > >>>>>>>>>>
> > >>>>>>>>>>> On Nov 15, 2024, at 11:50 PM, David Jacot
> > >>>>>> <dj...@confluent.io.INVALID>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> Hi,
> > >>>>>>>>>>>
> > >>>>>>>>>>> DJ2: Using "-" sounds good to me.
> > >>>>>>>>>>>
> > >>>>>>>>>>> DJ3: That seems reasonable to me.
> > >>>>>>>>>>>
> > >>>>>>>>>>> DJ4: Why not add it right now? I don't want to change the
> > output
> > >>> of
> > >>>>>> the
> > >>>>>>>>>>> tool too many times.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Best,
> > >>>>>>>>>>> David
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang <
> yangpoan@gmail.com>
> > >>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hi David / Andrew,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks for review. Thanks Andrew for picking up
> > >>>>>> kafka-share-groups.sh
> > >>>>>>>>>>>> implementation.
> > >>>>>>>>>>>> I will handle kafka-consumer-groups.sh.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> DJ3: After discussing with @Chia-Ping Tsai, we think that
> > using
> > >>> new
> > >>>>>>>>>> format
> > >>>>>>>>>>>> is more clear.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> The new format will be like
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>
> > >>>
> >
> <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Using colon(:) to concat topic name and partition IDs.
> > >>>>>>>>>>>> Using comma(,) to concat partition IDs within same topic
> name.
> > >>>>>>>>>>>> Using semicolon(;) to concat topic strings.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> AS3: Fix it with kafka-share-groups.sh. Thanks.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> If there is no further suggestion, I will start a vote
> thread
> > >>> next
> > >>>>>>>>>> Monday.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>> PoAn
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield <
> > >>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi PoAn,
> > >>>>>>>>>>>>> DJ2: I was just going to comment that "-" would be a more
> > >>>>>> appropriate
> > >>>>>>>>>>>> missing value, but
> > >>>>>>>>>>>>> you got there first.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> AS3: The examples for kafka-share-groups.sh include
> > >>>>>>>>>>>> kafka-consumer-groups.sh in the
> > >>>>>>>>>>>>> command line.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> If this is accepted in time, I'm happy to pick up the
> > >>>>>> implementation of
> > >>>>>>>>>>>> the share groups
> > >>>>>>>>>>>>> part of this if it helps.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>> Andrew
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> ________________________________________
> > >>>>>>>>>>>>> From: Frank Yang <ya...@gmail.com>
> > >>>>>>>>>>>>> Sent: 14 November 2024 10:48
> > >>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
> > >>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
> kafka-consumer-groups
> > >>>>>> command
> > >>>>>>>>>>>> line tool to support new consumer group
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi David,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks for the review and suggestion! I would like to get
> > this
> > >>> in
> > >>>>>> AK
> > >>>>>>>>>> 4.0
> > >>>>>>>>>>>> as well. I will do my best.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> DJ1: Update KIP to put GROUP-EPOCH and
> > TARGET-ASSIGNMENT-EPOCH
> > >>>>>> before
> > >>>>>>>>>>>> #MEMBERS.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> DJ2: I prefer to follow current missing column value “-“.
> > >>>>>> (reference <
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>
> > >>>
> >
> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92
> > >>>>>>>>>>>>> )
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> DJ3: Update KIP to use CURRENT-EPOCH CURRENT-ASSIGNMENT
> > >>>>>> TARGET-EPOCH
> > >>>>>>>>>>>> TARGET-ASSIGNMENT.
> > >>>>>>>>>>>>> Remove GROUP-EPOCH.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> For assignment value, it follows current output (reference
> <
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>
> > >>>
> >
> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418
> > >>>>>>>>>>> ).
> > >>>>>>>>>>>> I think the form of `topicid-partitionid` is more clear.
> > >>>>>>>>>>>>> If we would like to use this form, I will update both
> output
> > in
> > >>>>>>>>>>>> kafka-consumer-groups.sh and kafka-share-groups.sh.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> DJ4: It looks like DescribeGroupsResponseData only has
> > protocol
> > >>>>>> type at
> > >>>>>>>>>>>> group level.
> > >>>>>>>>>>>>> Both DescribeGroupsResponseData and
> > >>>>>> ConsumerGroupDescribeResponseData
> > >>>>>>>>>>>> don’t have protocol at member level.
> > >>>>>>>>>>>>> Could we use a followup to add it?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> DJ5: Update KIP to put LEADER-EPOCH before CURRENT-OFFSET.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>> PoAn
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot
> > >>>>>> <djacot@confluent.io.INVALID
> > >>>>>>>>>>>
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hi PoAn,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks for the KIP! I have a few minor
> comments/suggestions:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> DJ1: In the output of `--describe --verbose`, I would
> > suggest
> > >>>>>> putting
> > >>>>>>>>>>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before
> > `#MEMBERS`.
> > >>>>>>>>>>>>>> DJ2: Continuing on the above, I assume that we will print
> > out
> > >>> N/A
> > >>>>>> for
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>> fields not supported by classic groups. Is this correct?
> > >>>>>>>>>>>>>> DJ3: In the output of `--describe --members --verbose`, I
> > >>> wonder
> > >>>>>> if we
> > >>>>>>>>>>>>>> should use the following order and terms: CURRENT-EPOCH
> > >>>>>>>>>>>> CURRENT-ASSIGNMENT
> > >>>>>>>>>>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the
> > GROUP-EPOCH
> > >>>>>>>>>> because
> > >>>>>>>>>>>> it
> > >>>>>>>>>>>>>> is already in the group description. The value `(0)` used
> > for
> > >>> the
> > >>>>>>>>>>>>>> assignment is incorrect. Here I suppose that we will print
> > out
> > >>> the
> > >>>>>>>>>> list
> > >>>>>>>>>>>> of
> > >>>>>>>>>>>>>> partitions in the form of `topicid-partitionid`.
> > >>>>>>>>>>>>>> DJ4: Continuing on the above, I wonder if we should also
> add
> > >>> the
> > >>>>>>>>>>>> protocol
> > >>>>>>>>>>>>>> used `classic` or `consumer`. For context, it is possible
> to
> > >>> have
> > >>>>>>>>>>>> `classic`
> > >>>>>>>>>>>>>> members and `consumer` members in a `consumer` group
> during
> > an
> > >>>>>> online
> > >>>>>>>>>>>>>> upgrade from the classic protocol to the consumer
> protocol.
> > >>> Having
> > >>>>>>>>>> this
> > >>>>>>>>>>>>>> information may be handy for administrators. What do you
> > think?
> > >>>>>>>>>>>>>> DJ5: In the output of `--describe --offsets --verbose`, I
> > would
> > >>>>>>>>>> suggest
> > >>>>>>>>>>>>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> It would be great if we could get this one in AK 4.0 as it
> > >>>>>> changes the
> > >>>>>>>>>>>>>> output of the command.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>> DJ
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <
> > yangpoan@gmail.com>
> > >>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Hi Sean / Andrew / Lianet,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks for all your review and suggestions.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> AS1, LM1, LM4: Change to add KIP-848 information when
> users
> > >>> give
> > >>>>>>>>>>>> —verbose
> > >>>>>>>>>>>>>>> option.
> > >>>>>>>>>>>>>>> —describe —verbose: shows group epoch and target
> assignment
> > >>>>>> epoch.
> > >>>>>>>>>>>>>>> —describe —members —verbose: shows above information,
> > member
> > >>>>>> epoch,
> > >>>>>>>>>> and
> > >>>>>>>>>>>>>>> target assignment.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> AS2: Change to use MEMBER-EPOCH to align with KIP-848
> > >>> definition.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> LM2: For classic group, it doesn’t have epoch, so I use
> > >>> Optional
> > >>>>>>>>>>>> fields in
> > >>>>>>>>>>>>>>> ConsumerGroupDescription.
> > >>>>>>>>>>>>>>> For share group, it relies on KIP-848. It must have
> epoch,
> > so
> > >>> I
> > >>>>>> use
> > >>>>>>>>>> int
> > >>>>>>>>>>>>>>> fields in ShareGroupDescription.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> LM3: Remove —state change. We can get group level
> > information
> > >>> by
> > >>>>>>>>>>>> —describe
> > >>>>>>>>>>>>>>> —verbose.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> SQ1: Add LEADER-EPOCH when users give —describe —offsets
> > >>>>>> —verbose.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks.
> > >>>>>>>>>>>>>>> PoAn
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <
> lianetmr@gmail.com
> > >
> > >>>>>> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Hello Frank, thanks for the KIP! A few comments:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> LM1. I strongly agree with Andrew's suggestion of moving
> > this
> > >>>>>> into a
> > >>>>>>>>>>>>>>>> --verbose option. I expect someone would only attempt to
> > make
> > >>>>>> sense
> > >>>>>>>>>> of
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> epochs while debugging an issue, not while trying to
> view
> > the
> > >>>>>> group
> > >>>>>>>>>> or
> > >>>>>>>>>>>>>>>> member's info. (Also member-epoch makes more sense to
> me,
> > >>> even
> > >>>>>> more
> > >>>>>>>>>>>> if we
> > >>>>>>>>>>>>>>>> end up agreeing in a --verbose). Related to both issues,
> > my
> > >>>>>> take is
> > >>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>> whoever is close to the protocol will expect
> member-epoch,
> > >>>>>> whoever
> > >>>>>>>>>> is
> > >>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>> will probably won't even need to see the epochs at all.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> LM2. Why are the epochs defined as Optional (don't we
> > expect
> > >>> to
> > >>>>>>>>>> always
> > >>>>>>>>>>>>>>> have
> > >>>>>>>>>>>>>>>> them? with 0 initially, defined on the broker side for
> the
> > >>> group
> > >>>>>>>>>> ones,
> > >>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>> client side for the member)
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> LM3. Why is the KIP including the –state option in the
> > >>> proposed
> > >>>>>>>>>>>> changes?
> > >>>>>>>>>>>>>>> (
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>
> > >>>
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
> > >>>>>>>>>>>>>>>> ).
> > >>>>>>>>>>>>>>>> I get that the output would change in that example, but
> > it’s
> > >>> not
> > >>>>>>>>>>>> because
> > >>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>> any change to the –state option itself. It's because of
> > the
> > >>>>>> change
> > >>>>>>>>>>>>>>> proposed
> > >>>>>>>>>>>>>>>> to the –described (required with the --state), and the
> > >>> changes
> > >>>>>> to
> > >>>>>>>>>>>>>>>> --describe are already explained above (seems confusing,
> > got
> > >>> me
> > >>>>>>>>>>>> looking
> > >>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>> a change to the state filter which seemed unrelated).
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> LM4. Group epoch and target assignment epoch are
> > conceptually
> > >>>>>> at the
> > >>>>>>>>>>>>>>> group
> > >>>>>>>>>>>>>>>> level. vs member epoch that lands at a member level. So
> > >>> wonder
> > >>>>>> if we
> > >>>>>>>>>>>>>>> should
> > >>>>>>>>>>>>>>>> show them accordingly (ex. using the --verbose option)
> > >>>>>>>>>>>>>>>> –describe –verbose => shows group epoch and target
> > assignment
> > >>>>>> epoch
> > >>>>>>>>>>>>>>>> –describe –members –verbose => shows member
> epoch
> > >>>>>> (along
> > >>>>>>>>>>>>>>>> with group epoch and target assignment epoch)
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thanks!
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Lianet
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield <
> > >>>>>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hi PoAn,
> > >>>>>>>>>>>>>>>>> Thanks for the KIP. I have a few comments.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> AS1: It seems to me that these new additions are most
> > >>> useful to
> > >>>>>>>>>>>> people
> > >>>>>>>>>>>>>>>>> trying to understand
> > >>>>>>>>>>>>>>>>> the progress of rebalancing in quite some detail. The
> > >>>>>> information
> > >>>>>>>>>> is
> > >>>>>>>>>>>>>>>>> really not understandable
> > >>>>>>>>>>>>>>>>> for most users who do not have deep knowledge of
> > >>> KIP-848/932.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> As a result, I suggest for kafka-share-groups.sh that
> you
> > >>> add a
> > >>>>>>>>>>>>>>> --members
> > >>>>>>>>>>>>>>>>> --verbose option
> > >>>>>>>>>>>>>>>>> and only include the new information in the output for
> > that
> > >>>>>> option,
> > >>>>>>>>>>>>>>> rather
> > >>>>>>>>>>>>>>>>> than changing the
> > >>>>>>>>>>>>>>>>> non-verbose --members output.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> I also make a similar suggestion for
> > >>> kafka-consumer-groups.sh
> > >>>>>>>>>>>> --members
> > >>>>>>>>>>>>>>>>> and only add the
> > >>>>>>>>>>>>>>>>> new information for the --verbose output.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> AS2: I strongly suggest that you use MEMBER-EPOCH
> > instead of
> > >>>>>>>>>>>>>>>>> CONSUMER-EPOCH.
> > >>>>>>>>>>>>>>>>> I think it's more confusing not following the
> > terminology of
> > >>>>>> the
> > >>>>>>>>>>>> KIPs.
> > >>>>>>>>>>>>>>> For
> > >>>>>>>>>>>>>>>>> one thing,
> > >>>>>>>>>>>>>>>>> we already have "member" in the admin client such as
> > >>>>>>>>>>>> MemberDescription.
> > >>>>>>>>>>>>>>>>> It's not a
> > >>>>>>>>>>>>>>>>> new term.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>> Andrew
> > >>>>>>>>>>>>>>>>> ________________________________________
> > >>>>>>>>>>>>>>>>> From: PoAn Yang <pa...@apache.org>
> > >>>>>>>>>>>>>>>>> Sent: 25 October 2024 13:55
> > >>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
> > >>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
> > >>> kafka-consumer-groups
> > >>>>>>>>>> command
> > >>>>>>>>>>>>>>> line
> > >>>>>>>>>>>>>>>>> tool to support new consumer group
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hi Lucas,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Thanks for the review!
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> 1) Yes, I add related change for kafka-share-groups.sh
> to
> > >>> the
> > >>>>>> KIP.
> > >>>>>>>>>>>> Could
> > >>>>>>>>>>>>>>>>> you take a look? Thanks for the suggestion.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use
> > MEMBER-EPOCH
> > >>>>>> here,
> > >>>>>>>>>>>> users
> > >>>>>>>>>>>>>>> may
> > >>>>>>>>>>>>>>>>> confuse what is different between CONSUMER and MEMBER.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>> PoAn
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
> > >>>>>>>>>>>>>>>>>> Hi Frank,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> thanks for the KIP!
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> 1) For consistency, should we do the same for
> > >>>>>>>>>>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription, etc. ?
> > Even
> > >>> if
> > >>>>>> we do
> > >>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>> implement it right now if the share group
> implementation
> > >>> may
> > >>>>>> still
> > >>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>> incomplete, it may make sense to include it in the
> KIP.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH? That
> > would
> > >>>>>> seem
> > >>>>>>>>>>>> more
> > >>>>>>>>>>>>>>>>>> consistent.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>> Lucas
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <
> > >>>>>> yangpoan@gmail.com>
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> I would like to kick off the discussion of KIP-1099.
> > This
> > >>> KIP
> > >>>>>>>>>>>> enhances
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>> kafka-consumer-groups tools to include state which is
> > >>>>>> introduced
> > >>>>>>>>>> by
> > >>>>>>>>>>>>>>>>> KIP-848.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups command line
> > tool
> > >>> to
> > >>>>>>>>>> support
> > >>>>>>>>>>>>>>> new
> > >>>>>>>>>>>>>>>>>>> consumer group - Apache Kafka - Apache Software
> > Foundation
> > >>>>>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>
> > >>>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> cwiki.apache.org
> > >>>>>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>
> > >>>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> [image: favicon.ico]
> > >>>>>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>
> > >>>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>
> > >>>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Thank you,
> > >>>>>>>>>>>>>>>>>>> PoAn
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>
> > >>>
> > >>
> >
> >
>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by Chia-Ping Tsai <ch...@gmail.com>.
hi PoAn
> LM4: I agree it’s better to use a type like GroupProtocol enum, but I
would like to align the format with kafka-groups.sh.
How about we use another Jira to add “SHARE” to GroupProtocol and change
return type of ModernGroup#protocolType?
MemberDescription is also used by ClassicGroupDescription, and in classic
groups, the protocol can be an arbitrary string, so we can't use an enum.
In fact, I'm wondering if we should even add 'protocol' to MemberDescription,
since the field has different values between classic and consumer groups—it
shows 'consumer' in classic groups, whereas it shows 'classic' in consumer
groups. Yes, it seems like a typo, but it's true. This inconsistency feels
confusing to me. For example, users call Admin#describeConsumerGroups and
see that a member has the 'consumer' protocol when it's in a classic group.
Then, if the group is upgraded to a consumer group, calling
Admin#describeConsumerGroups again shows the protocol as 'classic' for the
same member.
If all we want to do is distinguish classic members in the
ConsumerGroupDescription, then we shouldn't modify the common
MemberDescription structure. Instead, we could add a collection to
ConsumerGroupDescription to trace members using the classic protocol, which
would allow us to enrich the output of command-line tools.
Best,
Chia-Ping
PoAn Yang <ya...@gmail.com> 於 2024年11月22日 週五 下午3:46寫道：
> Hi Lianet / Jeff,
>
> Thanks for the review.
>
> LM5: The kafka-share-groups.sh also uses “—describe” to show offsets
> information.
> Change to use “—describe —state —verbose” to show group level information
> in kafka-share-groups.sh.
>
> LM6: Update the example.
>
> LM7: Add a description to mention “—verbose” is a new option in
> ShareGroupCommandOptions.
>
> JK2: Yes, add the statement about format change for ASSIGNMENT value.
>
> Thanks,
> PoAn
>
> > On Nov 22, 2024, at 7:35 AM, Jeff Kim <je...@apache.org> wrote:
> >
> > Hi PoAn,
> >
> > JK2: Can we state in the KIP that we will reformat the existing
> > classic group's `--members --verbose` as well, specifically the
> > "ASSIGNMENT" column to include the topic name?
> >
> > Thanks!
> >
> > On 2024/11/21 14:33:37 "Lianet M." wrote:
> >> Thanks for the updates PoAn!
> >>
> >> LM6. Just a nit, under the --describe --state --verbose the header is
> fine
> >> but the example is still missing the --state argument.
> >>
> >> LM7. Under the Proposed changes for kafka-share-groups.sh, should we
> >> clearly mention that the KIP adds a new --verbose option? I think it's
> >> confusing because it's presented as if the option already exists and
> we're
> >> only changing the output (which is the case for kafka-consumer-groups
> but
> >> not for kafka-share-groups)
> >>
> >> That's all on my side. Thanks!
> >>
> >> Lianet
> >>
> >> On Wed, Nov 20, 2024 at 11:12 PM PoAn Yang <ya...@gmail.com> wrote:
> >>
> >>> Hi Lianet / Jeff,
> >>>
> >>> Thanks for review and suggestions.
> >>>
> >>> LM4: At group level, the protocol field is from
> ModernGroup#protocolType.
> >>> It’s a string.
> >>> The value is defined in different places like ShareGroup#PROTOCOL_TYPE
> and
> >>> ConsumerProtocol#PROTOCOL_TYPE.
> >>> In KIP-1043 <
> >>>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=305171038#KIP1043:Administrationofgroups-kafka-groups.sh
> >,
> >>> the kafka-groups.sh also shows protocol like “classic” / “consumer” /
> >>> “share”.
> >>> I agree it’s better to use a type like GroupProtocol enum, but I would
> >>> like to align the format with kafka-groups.sh.
> >>> How about we use another Jira to add “SHARE” to GroupProtocol and
> change
> >>> return type of ModernGroup#protocolType?
> >>>
> >>> LM5 / JK1: Sorry, I misunderstood LM3, so I deleted —state in the
> title.
> >>> You’re correct. I would like to use `—describe —state —verbose` to show
> >>> group level information.
> >>> For `—describe —verbose`, it will show output as same as `—describe
> >>> —offsets —verbose`,
> >>> because `—describe` also shows output as same as `—describe —offsets`.
> >>>
> >>> JK2: Yes, I only use consumer group in the sample output, but the
> classic
> >>> group will use the same format as well.
> >>>
> >>> Thanks,
> >>> PoAn
> >>>
> >>>> On Nov 21, 2024, at 8:28 AM, Jeff Kim <je...@apache.org> wrote:
> >>>>
> >>>> Hi PoAn,
> >>>>
> >>>> Thanks for the KIP. I have some questions:
> >>>>
> >>>> JK1: (Realized this is similar to LM5) On `--describe --verbose`
> >>> proposed changes - doesn't `--describe $group` default to printing the
> >>> offsets? Perhaps you're referring to the `--state` argument? Also,
> would
> >>> that mean the default `--describe $group --verbose` command would not
> print
> >>> the added field to `--offsets verbose` (leader-epoch) or would it?
> >>>>
> >>>> JK2: Looking at ConsumerGroupCommand.java, the existing "ASSIGNMENT"
> >>> column under `--members --verbose` does group the topic partitions by
> topic
> >>> but does not prefix the grouping with the topic name like you're
> proposal:
> >>> "my_topic:0,1;new_topic:0,1". Should we do apply the same format for
> the
> >>> classic group as well?
> >>>>
> >>>>
> >>>>
> >>>> On 2024/11/20 21:32:41 "Lianet M." wrote:
> >>>>> Hello PoAn, just of couple more minor comments:
> >>>>>
> >>>>> LM4. Regarding the new “protocol” field added to MemberDescription.
> >>> Should
> >>>>> we consider reusing the existing GroupProtocol enum instead of
> String?
> >>>>> (It’s the one we use from the consumer side to refer to the protocol
> in
> >>>>> use, just missing Share I notice).
> >>>>>
> >>>>> LM5. Regarding the change to the output for kafka-consumer-groups,
> the
> >>>>> command shown does not include the —state option, but the output
> shows
> >>>>> state info (state, #members, epochs). I would guess that we want to
> >>> modify
> >>>>> the output only when we describe a group with the —state —verbose
> >>> option,
> >>>>> is my understanding right? If my understanding is right we’re just
> >>> missing
> >>>>> adding the —state in the example, and the KIP does not introduce any
> >>>>> changes to the —describe —verbose option. (If not, it would mean a
> >>> bigger
> >>>>> change to the output of —describe —verbose which I expect is not the
> >>>>> intention?)
> >>>>>
> >>>>> Thanks!
> >>>>>
> >>>>> Lianet
> >>>>>
> >>>>>
> >>>>> On Wed, Nov 20, 2024 at 2:12 AM PoAn Yang <ya...@gmail.com>
> wrote:
> >>>>>
> >>>>>> Hi Chia-Ping,
> >>>>>>
> >>>>>> Thanks for the review and suggestions.
> >>>>>>
> >>>>>> CI0: Thanks for the reminder. Update validVersions in
> >>>>>> ConsumerGroupDescribeRequest to 0-1.
> >>>>>>
> >>>>>> CI1: Yes, I use `ConsumerGroupMember#useClassicProtocol` to check
> >>> whether
> >>>>>> a member in consumer group uses “classic” or “consumer” protocol.
> >>>>>>
> >>>>>> CI2: Yes, a member in share group always uses “share” protocol.
> >>>>>>
> >>>>>> CI3: Add a table to show meaning of “classic”, “consumer”, and
> “share”
> >>>>>> protocol.
> >>>>>>
> >>>>>> BTW, the vote thread is in
> >>>>>> https://lists.apache.org/thread/rb25tf75tzf4c7jqqldlo5jh9w8chsq6.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> PoAn
> >>>>>>
> >>>>>>> On Nov 20, 2024, at 11:46 AM, Chia-Ping Tsai <ch...@apache.org>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> hi PoAn
> >>>>>>>
> >>>>>>> CI0: We have to bump the version of ConsumerGroupDescribeRequest
> as
> >>>>>> well, so server can distinguish the new and old behavior.
> >>>>>>>
> >>>>>>> CI1: the type of new filed is string, so I guess you plan to use
> >>>>>> `ConsumerGroupMember#useClassicProtocol` [0] flag to return either
> >>>>>> "classic" or "consumer", right?
> >>>>>>>
> >>>>>>> CI2: `MemberDescription` is used by `ShareGroupDescription` too, so
> >>> the
> >>>>>> new filed protocol in shared group is always "shared", right?
> >>>>>>>
> >>>>>>> CI3: Could you consider adding a table to show the value of the
> >>> protocol
> >>>>>> field in each case? Andrew has a beautiful table in KIP-1043 that
> >>> lists all
> >>>>>> possible protocol names.
> >>>>>>>
> >>>>>>> [0]
> >>>>>>
> >>>
> https://github.com/apache/kafka/blob/441a6d0b790f4a17b454caeea7588a6b90fbd9db/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/consumer/ConsumerGroupMember.java#L454
> >>>>>>
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Chia-Ping
> >>>>>>>
> >>>>>>> On 2024/11/19 15:45:16 PoAn Yang wrote:
> >>>>>>>> Hi David,
> >>>>>>>>
> >>>>>>>> Thanks for the reminder.
> >>>>>>>>
> >>>>>>>> DJ4: Add related content to the KIP.
> >>>>>>>> Bump validVersions in ConsumerGroupDescribeResponse to “0-1”.
> >>>>>>>> Add a new field “Protocol” to ConsumerGroupDescribeResponse.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> PoAn
> >>>>>>>>
> >>>>>>>>> On Nov 19, 2024, at 3:57 PM, David Jacot
> >>> <dj...@confluent.io.INVALID>
> >>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Hi PoAn,
> >>>>>>>>>
> >>>>>>>>> DJ4: Could you please also include the required changes to the
> >>>>>>>>> ConsumerGroupDescribed API in the public interface section? We
> >>>>>> basically
> >>>>>>>>> need to bump the version, add the new field, etc.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> David
> >>>>>>>>>
> >>>>>>>>> On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang <ya...@gmail.com>
> >>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi David,
> >>>>>>>>>>
> >>>>>>>>>> DJ4: Add a new field protocol to MemberDescription,
> >>>>>>>>>> so the command line tool can show protocol information when
> users
> >>>>>> describe
> >>>>>>>>>> members.
> >>>>>>>>>>
> >>>>>>>>>> If there is no further suggestion, I will start a vote thread
> >>> today.
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> PoAn
> >>>>>>>>>>
> >>>>>>>>>>> On Nov 15, 2024, at 11:50 PM, David Jacot
> >>>>>> <dj...@confluent.io.INVALID>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Hi,
> >>>>>>>>>>>
> >>>>>>>>>>> DJ2: Using "-" sounds good to me.
> >>>>>>>>>>>
> >>>>>>>>>>> DJ3: That seems reasonable to me.
> >>>>>>>>>>>
> >>>>>>>>>>> DJ4: Why not add it right now? I don't want to change the
> output
> >>> of
> >>>>>> the
> >>>>>>>>>>> tool too many times.
> >>>>>>>>>>>
> >>>>>>>>>>> Best,
> >>>>>>>>>>> David
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang <ya...@gmail.com>
> >>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi David / Andrew,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks for review. Thanks Andrew for picking up
> >>>>>> kafka-share-groups.sh
> >>>>>>>>>>>> implementation.
> >>>>>>>>>>>> I will handle kafka-consumer-groups.sh.
> >>>>>>>>>>>>
> >>>>>>>>>>>> DJ3: After discussing with @Chia-Ping Tsai, we think that
> using
> >>> new
> >>>>>>>>>> format
> >>>>>>>>>>>> is more clear.
> >>>>>>>>>>>>
> >>>>>>>>>>>> The new format will be like
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>
> >>>
> <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Using colon(:) to concat topic name and partition IDs.
> >>>>>>>>>>>> Using comma(,) to concat partition IDs within same topic name.
> >>>>>>>>>>>> Using semicolon(;) to concat topic strings.
> >>>>>>>>>>>>
> >>>>>>>>>>>> AS3: Fix it with kafka-share-groups.sh. Thanks.
> >>>>>>>>>>>>
> >>>>>>>>>>>> If there is no further suggestion, I will start a vote thread
> >>> next
> >>>>>>>>>> Monday.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>> PoAn
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield <
> >>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>>>> DJ2: I was just going to comment that "-" would be a more
> >>>>>> appropriate
> >>>>>>>>>>>> missing value, but
> >>>>>>>>>>>>> you got there first.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> AS3: The examples for kafka-share-groups.sh include
> >>>>>>>>>>>> kafka-consumer-groups.sh in the
> >>>>>>>>>>>>> command line.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> If this is accepted in time, I'm happy to pick up the
> >>>>>> implementation of
> >>>>>>>>>>>> the share groups
> >>>>>>>>>>>>> part of this if it helps.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>> Andrew
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>> From: Frank Yang <ya...@gmail.com>
> >>>>>>>>>>>>> Sent: 14 November 2024 10:48
> >>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
> >>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups
> >>>>>> command
> >>>>>>>>>>>> line tool to support new consumer group
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi David,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for the review and suggestion! I would like to get
> this
> >>> in
> >>>>>> AK
> >>>>>>>>>> 4.0
> >>>>>>>>>>>> as well. I will do my best.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> DJ1: Update KIP to put GROUP-EPOCH and
> TARGET-ASSIGNMENT-EPOCH
> >>>>>> before
> >>>>>>>>>>>> #MEMBERS.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> DJ2: I prefer to follow current missing column value “-“.
> >>>>>> (reference <
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>
> >>>
> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92
> >>>>>>>>>>>>> )
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> DJ3: Update KIP to use CURRENT-EPOCH CURRENT-ASSIGNMENT
> >>>>>> TARGET-EPOCH
> >>>>>>>>>>>> TARGET-ASSIGNMENT.
> >>>>>>>>>>>>> Remove GROUP-EPOCH.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> For assignment value, it follows current output (reference <
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>
> >>>
> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418
> >>>>>>>>>>> ).
> >>>>>>>>>>>> I think the form of `topicid-partitionid` is more clear.
> >>>>>>>>>>>>> If we would like to use this form, I will update both output
> in
> >>>>>>>>>>>> kafka-consumer-groups.sh and kafka-share-groups.sh.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> DJ4: It looks like DescribeGroupsResponseData only has
> protocol
> >>>>>> type at
> >>>>>>>>>>>> group level.
> >>>>>>>>>>>>> Both DescribeGroupsResponseData and
> >>>>>> ConsumerGroupDescribeResponseData
> >>>>>>>>>>>> don’t have protocol at member level.
> >>>>>>>>>>>>> Could we use a followup to add it?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> DJ5: Update KIP to put LEADER-EPOCH before CURRENT-OFFSET.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot
> >>>>>> <djacot@confluent.io.INVALID
> >>>>>>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks for the KIP! I have a few minor comments/suggestions:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> DJ1: In the output of `--describe --verbose`, I would
> suggest
> >>>>>> putting
> >>>>>>>>>>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before
> `#MEMBERS`.
> >>>>>>>>>>>>>> DJ2: Continuing on the above, I assume that we will print
> out
> >>> N/A
> >>>>>> for
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>> fields not supported by classic groups. Is this correct?
> >>>>>>>>>>>>>> DJ3: In the output of `--describe --members --verbose`, I
> >>> wonder
> >>>>>> if we
> >>>>>>>>>>>>>> should use the following order and terms: CURRENT-EPOCH
> >>>>>>>>>>>> CURRENT-ASSIGNMENT
> >>>>>>>>>>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the
> GROUP-EPOCH
> >>>>>>>>>> because
> >>>>>>>>>>>> it
> >>>>>>>>>>>>>> is already in the group description. The value `(0)` used
> for
> >>> the
> >>>>>>>>>>>>>> assignment is incorrect. Here I suppose that we will print
> out
> >>> the
> >>>>>>>>>> list
> >>>>>>>>>>>> of
> >>>>>>>>>>>>>> partitions in the form of `topicid-partitionid`.
> >>>>>>>>>>>>>> DJ4: Continuing on the above, I wonder if we should also add
> >>> the
> >>>>>>>>>>>> protocol
> >>>>>>>>>>>>>> used `classic` or `consumer`. For context, it is possible to
> >>> have
> >>>>>>>>>>>> `classic`
> >>>>>>>>>>>>>> members and `consumer` members in a `consumer` group during
> an
> >>>>>> online
> >>>>>>>>>>>>>> upgrade from the classic protocol to the consumer protocol.
> >>> Having
> >>>>>>>>>> this
> >>>>>>>>>>>>>> information may be handy for administrators. What do you
> think?
> >>>>>>>>>>>>>> DJ5: In the output of `--describe --offsets --verbose`, I
> would
> >>>>>>>>>> suggest
> >>>>>>>>>>>>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> It would be great if we could get this one in AK 4.0 as it
> >>>>>> changes the
> >>>>>>>>>>>>>> output of the command.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>> DJ
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <
> yangpoan@gmail.com>
> >>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi Sean / Andrew / Lianet,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks for all your review and suggestions.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> AS1, LM1, LM4: Change to add KIP-848 information when users
> >>> give
> >>>>>>>>>>>> —verbose
> >>>>>>>>>>>>>>> option.
> >>>>>>>>>>>>>>> —describe —verbose: shows group epoch and target assignment
> >>>>>> epoch.
> >>>>>>>>>>>>>>> —describe —members —verbose: shows above information,
> member
> >>>>>> epoch,
> >>>>>>>>>> and
> >>>>>>>>>>>>>>> target assignment.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> AS2: Change to use MEMBER-EPOCH to align with KIP-848
> >>> definition.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> LM2: For classic group, it doesn’t have epoch, so I use
> >>> Optional
> >>>>>>>>>>>> fields in
> >>>>>>>>>>>>>>> ConsumerGroupDescription.
> >>>>>>>>>>>>>>> For share group, it relies on KIP-848. It must have epoch,
> so
> >>> I
> >>>>>> use
> >>>>>>>>>> int
> >>>>>>>>>>>>>>> fields in ShareGroupDescription.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> LM3: Remove —state change. We can get group level
> information
> >>> by
> >>>>>>>>>>>> —describe
> >>>>>>>>>>>>>>> —verbose.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> SQ1: Add LEADER-EPOCH when users give —describe —offsets
> >>>>>> —verbose.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks.
> >>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <lianetmr@gmail.com
> >
> >>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hello Frank, thanks for the KIP! A few comments:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> LM1. I strongly agree with Andrew's suggestion of moving
> this
> >>>>>> into a
> >>>>>>>>>>>>>>>> --verbose option. I expect someone would only attempt to
> make
> >>>>>> sense
> >>>>>>>>>> of
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> epochs while debugging an issue, not while trying to view
> the
> >>>>>> group
> >>>>>>>>>> or
> >>>>>>>>>>>>>>>> member's info. (Also member-epoch makes more sense to me,
> >>> even
> >>>>>> more
> >>>>>>>>>>>> if we
> >>>>>>>>>>>>>>>> end up agreeing in a --verbose). Related to both issues,
> my
> >>>>>> take is
> >>>>>>>>>>>> that
> >>>>>>>>>>>>>>>> whoever is close to the protocol will expect member-epoch,
> >>>>>> whoever
> >>>>>>>>>> is
> >>>>>>>>>>>> not
> >>>>>>>>>>>>>>>> will probably won't even need to see the epochs at all.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> LM2. Why are the epochs defined as Optional (don't we
> expect
> >>> to
> >>>>>>>>>> always
> >>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>> them? with 0 initially, defined on the broker side for the
> >>> group
> >>>>>>>>>> ones,
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>> client side for the member)
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> LM3. Why is the KIP including the –state option in the
> >>> proposed
> >>>>>>>>>>>> changes?
> >>>>>>>>>>>>>>> (
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>
> >>>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
> >>>>>>>>>>>>>>>> ).
> >>>>>>>>>>>>>>>> I get that the output would change in that example, but
> it’s
> >>> not
> >>>>>>>>>>>> because
> >>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>> any change to the –state option itself. It's because of
> the
> >>>>>> change
> >>>>>>>>>>>>>>> proposed
> >>>>>>>>>>>>>>>> to the –described (required with the --state), and the
> >>> changes
> >>>>>> to
> >>>>>>>>>>>>>>>> --describe are already explained above (seems confusing,
> got
> >>> me
> >>>>>>>>>>>> looking
> >>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>> a change to the state filter which seemed unrelated).
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> LM4. Group epoch and target assignment epoch are
> conceptually
> >>>>>> at the
> >>>>>>>>>>>>>>> group
> >>>>>>>>>>>>>>>> level. vs member epoch that lands at a member level. So
> >>> wonder
> >>>>>> if we
> >>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>> show them accordingly (ex. using the --verbose option)
> >>>>>>>>>>>>>>>> –describe –verbose => shows group epoch and target
> assignment
> >>>>>> epoch
> >>>>>>>>>>>>>>>> –describe –members –verbose => shows member epoch
> >>>>>> (along
> >>>>>>>>>>>>>>>> with group epoch and target assignment epoch)
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Lianet
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield <
> >>>>>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>>>>>>>> Thanks for the KIP. I have a few comments.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> AS1: It seems to me that these new additions are most
> >>> useful to
> >>>>>>>>>>>> people
> >>>>>>>>>>>>>>>>> trying to understand
> >>>>>>>>>>>>>>>>> the progress of rebalancing in quite some detail. The
> >>>>>> information
> >>>>>>>>>> is
> >>>>>>>>>>>>>>>>> really not understandable
> >>>>>>>>>>>>>>>>> for most users who do not have deep knowledge of
> >>> KIP-848/932.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> As a result, I suggest for kafka-share-groups.sh that you
> >>> add a
> >>>>>>>>>>>>>>> --members
> >>>>>>>>>>>>>>>>> --verbose option
> >>>>>>>>>>>>>>>>> and only include the new information in the output for
> that
> >>>>>> option,
> >>>>>>>>>>>>>>> rather
> >>>>>>>>>>>>>>>>> than changing the
> >>>>>>>>>>>>>>>>> non-verbose --members output.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I also make a similar suggestion for
> >>> kafka-consumer-groups.sh
> >>>>>>>>>>>> --members
> >>>>>>>>>>>>>>>>> and only add the
> >>>>>>>>>>>>>>>>> new information for the --verbose output.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> AS2: I strongly suggest that you use MEMBER-EPOCH
> instead of
> >>>>>>>>>>>>>>>>> CONSUMER-EPOCH.
> >>>>>>>>>>>>>>>>> I think it's more confusing not following the
> terminology of
> >>>>>> the
> >>>>>>>>>>>> KIPs.
> >>>>>>>>>>>>>>> For
> >>>>>>>>>>>>>>>>> one thing,
> >>>>>>>>>>>>>>>>> we already have "member" in the admin client such as
> >>>>>>>>>>>> MemberDescription.
> >>>>>>>>>>>>>>>>> It's not a
> >>>>>>>>>>>>>>>>> new term.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>> Andrew
> >>>>>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>>>>> From: PoAn Yang <pa...@apache.org>
> >>>>>>>>>>>>>>>>> Sent: 25 October 2024 13:55
> >>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
> >>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
> >>> kafka-consumer-groups
> >>>>>>>>>> command
> >>>>>>>>>>>>>>> line
> >>>>>>>>>>>>>>>>> tool to support new consumer group
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi Lucas,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks for the review!
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> 1) Yes, I add related change for kafka-share-groups.sh to
> >>> the
> >>>>>> KIP.
> >>>>>>>>>>>> Could
> >>>>>>>>>>>>>>>>> you take a look? Thanks for the suggestion.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use
> MEMBER-EPOCH
> >>>>>> here,
> >>>>>>>>>>>> users
> >>>>>>>>>>>>>>> may
> >>>>>>>>>>>>>>>>> confuse what is different between CONSUMER and MEMBER.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
> >>>>>>>>>>>>>>>>>> Hi Frank,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> thanks for the KIP!
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> 1) For consistency, should we do the same for
> >>>>>>>>>>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription, etc. ?
> Even
> >>> if
> >>>>>> we do
> >>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>> implement it right now if the share group implementation
> >>> may
> >>>>>> still
> >>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>> incomplete, it may make sense to include it in the KIP.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH? That
> would
> >>>>>> seem
> >>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>> consistent.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>> Lucas
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <
> >>>>>> yangpoan@gmail.com>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I would like to kick off the discussion of KIP-1099.
> This
> >>> KIP
> >>>>>>>>>>>> enhances
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> kafka-consumer-groups tools to include state which is
> >>>>>> introduced
> >>>>>>>>>> by
> >>>>>>>>>>>>>>>>> KIP-848.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups command line
> tool
> >>> to
> >>>>>>>>>> support
> >>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>> consumer group - Apache Kafka - Apache Software
> Foundation
> >>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> cwiki.apache.org
> >>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> [image: favicon.ico]
> >>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thank you,
> >>>>>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>
> >>>
> >>
>
>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by PoAn Yang <ya...@gmail.com>.
Hi Lianet / Jeff,
Thanks for the review.
LM5: The kafka-share-groups.sh also uses “—describe” to show offsets information.
Change to use “—describe —state —verbose” to show group level information in kafka-share-groups.sh.
LM6: Update the example.
LM7: Add a description to mention “—verbose” is a new option in ShareGroupCommandOptions.
JK2: Yes, add the statement about format change for ASSIGNMENT value.
Thanks,
PoAn
> On Nov 22, 2024, at 7:35 AM, Jeff Kim <je...@apache.org> wrote:
>
> Hi PoAn,
>
> JK2: Can we state in the KIP that we will reformat the existing
> classic group's `--members --verbose` as well, specifically the
> "ASSIGNMENT" column to include the topic name?
>
> Thanks!
>
> On 2024/11/21 14:33:37 "Lianet M." wrote:
>> Thanks for the updates PoAn!
>>
>> LM6. Just a nit, under the --describe --state --verbose the header is fine
>> but the example is still missing the --state argument.
>>
>> LM7. Under the Proposed changes for kafka-share-groups.sh, should we
>> clearly mention that the KIP adds a new --verbose option? I think it's
>> confusing because it's presented as if the option already exists and we're
>> only changing the output (which is the case for kafka-consumer-groups but
>> not for kafka-share-groups)
>>
>> That's all on my side. Thanks!
>>
>> Lianet
>>
>> On Wed, Nov 20, 2024 at 11:12 PM PoAn Yang <ya...@gmail.com> wrote:
>>
>>> Hi Lianet / Jeff,
>>>
>>> Thanks for review and suggestions.
>>>
>>> LM4: At group level, the protocol field is from ModernGroup#protocolType.
>>> It’s a string.
>>> The value is defined in different places like ShareGroup#PROTOCOL_TYPE and
>>> ConsumerProtocol#PROTOCOL_TYPE.
>>> In KIP-1043 <
>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=305171038#KIP1043:Administrationofgroups-kafka-groups.sh>,
>>> the kafka-groups.sh also shows protocol like “classic” / “consumer” /
>>> “share”.
>>> I agree it’s better to use a type like GroupProtocol enum, but I would
>>> like to align the format with kafka-groups.sh.
>>> How about we use another Jira to add “SHARE” to GroupProtocol and change
>>> return type of ModernGroup#protocolType?
>>>
>>> LM5 / JK1: Sorry, I misunderstood LM3, so I deleted —state in the title.
>>> You’re correct. I would like to use `—describe —state —verbose` to show
>>> group level information.
>>> For `—describe —verbose`, it will show output as same as `—describe
>>> —offsets —verbose`,
>>> because `—describe` also shows output as same as `—describe —offsets`.
>>>
>>> JK2: Yes, I only use consumer group in the sample output, but the classic
>>> group will use the same format as well.
>>>
>>> Thanks,
>>> PoAn
>>>
>>>> On Nov 21, 2024, at 8:28 AM, Jeff Kim <je...@apache.org> wrote:
>>>>
>>>> Hi PoAn,
>>>>
>>>> Thanks for the KIP. I have some questions:
>>>>
>>>> JK1: (Realized this is similar to LM5) On `--describe --verbose`
>>> proposed changes - doesn't `--describe $group` default to printing the
>>> offsets? Perhaps you're referring to the `--state` argument? Also, would
>>> that mean the default `--describe $group --verbose` command would not print
>>> the added field to `--offsets verbose` (leader-epoch) or would it?
>>>>
>>>> JK2: Looking at ConsumerGroupCommand.java, the existing "ASSIGNMENT"
>>> column under `--members --verbose` does group the topic partitions by topic
>>> but does not prefix the grouping with the topic name like you're proposal:
>>> "my_topic:0,1;new_topic:0,1". Should we do apply the same format for the
>>> classic group as well?
>>>>
>>>>
>>>>
>>>> On 2024/11/20 21:32:41 "Lianet M." wrote:
>>>>> Hello PoAn, just of couple more minor comments:
>>>>>
>>>>> LM4. Regarding the new “protocol” field added to MemberDescription.
>>> Should
>>>>> we consider reusing the existing GroupProtocol enum instead of String?
>>>>> (It’s the one we use from the consumer side to refer to the protocol in
>>>>> use, just missing Share I notice).
>>>>>
>>>>> LM5. Regarding the change to the output for kafka-consumer-groups, the
>>>>> command shown does not include the —state option, but the output shows
>>>>> state info (state, #members, epochs). I would guess that we want to
>>> modify
>>>>> the output only when we describe a group with the —state —verbose
>>> option,
>>>>> is my understanding right? If my understanding is right we’re just
>>> missing
>>>>> adding the —state in the example, and the KIP does not introduce any
>>>>> changes to the —describe —verbose option. (If not, it would mean a
>>> bigger
>>>>> change to the output of —describe —verbose which I expect is not the
>>>>> intention?)
>>>>>
>>>>> Thanks!
>>>>>
>>>>> Lianet
>>>>>
>>>>>
>>>>> On Wed, Nov 20, 2024 at 2:12 AM PoAn Yang <ya...@gmail.com> wrote:
>>>>>
>>>>>> Hi Chia-Ping,
>>>>>>
>>>>>> Thanks for the review and suggestions.
>>>>>>
>>>>>> CI0: Thanks for the reminder. Update validVersions in
>>>>>> ConsumerGroupDescribeRequest to 0-1.
>>>>>>
>>>>>> CI1: Yes, I use `ConsumerGroupMember#useClassicProtocol` to check
>>> whether
>>>>>> a member in consumer group uses “classic” or “consumer” protocol.
>>>>>>
>>>>>> CI2: Yes, a member in share group always uses “share” protocol.
>>>>>>
>>>>>> CI3: Add a table to show meaning of “classic”, “consumer”, and “share”
>>>>>> protocol.
>>>>>>
>>>>>> BTW, the vote thread is in
>>>>>> https://lists.apache.org/thread/rb25tf75tzf4c7jqqldlo5jh9w8chsq6.
>>>>>>
>>>>>> Thanks,
>>>>>> PoAn
>>>>>>
>>>>>>> On Nov 20, 2024, at 11:46 AM, Chia-Ping Tsai <ch...@apache.org>
>>>>>> wrote:
>>>>>>>
>>>>>>> hi PoAn
>>>>>>>
>>>>>>> CI0: We have to bump the version of ConsumerGroupDescribeRequest as
>>>>>> well, so server can distinguish the new and old behavior.
>>>>>>>
>>>>>>> CI1: the type of new filed is string, so I guess you plan to use
>>>>>> `ConsumerGroupMember#useClassicProtocol` [0] flag to return either
>>>>>> "classic" or "consumer", right?
>>>>>>>
>>>>>>> CI2: `MemberDescription` is used by `ShareGroupDescription` too, so
>>> the
>>>>>> new filed protocol in shared group is always "shared", right?
>>>>>>>
>>>>>>> CI3: Could you consider adding a table to show the value of the
>>> protocol
>>>>>> field in each case? Andrew has a beautiful table in KIP-1043 that
>>> lists all
>>>>>> possible protocol names.
>>>>>>>
>>>>>>> [0]
>>>>>>
>>> https://github.com/apache/kafka/blob/441a6d0b790f4a17b454caeea7588a6b90fbd9db/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/consumer/ConsumerGroupMember.java#L454
>>>>>>
>>>>>>>
>>>>>>> Best,
>>>>>>> Chia-Ping
>>>>>>>
>>>>>>> On 2024/11/19 15:45:16 PoAn Yang wrote:
>>>>>>>> Hi David,
>>>>>>>>
>>>>>>>> Thanks for the reminder.
>>>>>>>>
>>>>>>>> DJ4: Add related content to the KIP.
>>>>>>>> Bump validVersions in ConsumerGroupDescribeResponse to “0-1”.
>>>>>>>> Add a new field “Protocol” to ConsumerGroupDescribeResponse.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> PoAn
>>>>>>>>
>>>>>>>>> On Nov 19, 2024, at 3:57 PM, David Jacot
>>> <dj...@confluent.io.INVALID>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi PoAn,
>>>>>>>>>
>>>>>>>>> DJ4: Could you please also include the required changes to the
>>>>>>>>> ConsumerGroupDescribed API in the public interface section? We
>>>>>> basically
>>>>>>>>> need to bump the version, add the new field, etc.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> David
>>>>>>>>>
>>>>>>>>> On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang <ya...@gmail.com>
>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi David,
>>>>>>>>>>
>>>>>>>>>> DJ4: Add a new field protocol to MemberDescription,
>>>>>>>>>> so the command line tool can show protocol information when users
>>>>>> describe
>>>>>>>>>> members.
>>>>>>>>>>
>>>>>>>>>> If there is no further suggestion, I will start a vote thread
>>> today.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> PoAn
>>>>>>>>>>
>>>>>>>>>>> On Nov 15, 2024, at 11:50 PM, David Jacot
>>>>>> <dj...@confluent.io.INVALID>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> DJ2: Using "-" sounds good to me.
>>>>>>>>>>>
>>>>>>>>>>> DJ3: That seems reasonable to me.
>>>>>>>>>>>
>>>>>>>>>>> DJ4: Why not add it right now? I don't want to change the output
>>> of
>>>>>> the
>>>>>>>>>>> tool too many times.
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> David
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang <ya...@gmail.com>
>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi David / Andrew,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for review. Thanks Andrew for picking up
>>>>>> kafka-share-groups.sh
>>>>>>>>>>>> implementation.
>>>>>>>>>>>> I will handle kafka-consumer-groups.sh.
>>>>>>>>>>>>
>>>>>>>>>>>> DJ3: After discussing with @Chia-Ping Tsai, we think that using
>>> new
>>>>>>>>>> format
>>>>>>>>>>>> is more clear.
>>>>>>>>>>>>
>>>>>>>>>>>> The new format will be like
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>
>>> <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2>
>>>>>>>>>>>>
>>>>>>>>>>>> Using colon(:) to concat topic name and partition IDs.
>>>>>>>>>>>> Using comma(,) to concat partition IDs within same topic name.
>>>>>>>>>>>> Using semicolon(;) to concat topic strings.
>>>>>>>>>>>>
>>>>>>>>>>>> AS3: Fix it with kafka-share-groups.sh. Thanks.
>>>>>>>>>>>>
>>>>>>>>>>>> If there is no further suggestion, I will start a vote thread
>>> next
>>>>>>>>>> Monday.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> PoAn
>>>>>>>>>>>>
>>>>>>>>>>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield <
>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>> DJ2: I was just going to comment that "-" would be a more
>>>>>> appropriate
>>>>>>>>>>>> missing value, but
>>>>>>>>>>>>> you got there first.
>>>>>>>>>>>>>
>>>>>>>>>>>>> AS3: The examples for kafka-share-groups.sh include
>>>>>>>>>>>> kafka-consumer-groups.sh in the
>>>>>>>>>>>>> command line.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If this is accepted in time, I'm happy to pick up the
>>>>>> implementation of
>>>>>>>>>>>> the share groups
>>>>>>>>>>>>> part of this if it helps.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Andrew
>>>>>>>>>>>>>
>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>> From: Frank Yang <ya...@gmail.com>
>>>>>>>>>>>>> Sent: 14 November 2024 10:48
>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups
>>>>>> command
>>>>>>>>>>>> line tool to support new consumer group
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi David,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the review and suggestion! I would like to get this
>>> in
>>>>>> AK
>>>>>>>>>> 4.0
>>>>>>>>>>>> as well. I will do my best.
>>>>>>>>>>>>>
>>>>>>>>>>>>> DJ1: Update KIP to put GROUP-EPOCH and TARGET-ASSIGNMENT-EPOCH
>>>>>> before
>>>>>>>>>>>> #MEMBERS.
>>>>>>>>>>>>>
>>>>>>>>>>>>> DJ2: I prefer to follow current missing column value “-“.
>>>>>> (reference <
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>
>>> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92
>>>>>>>>>>>>> )
>>>>>>>>>>>>>
>>>>>>>>>>>>> DJ3: Update KIP to use CURRENT-EPOCH CURRENT-ASSIGNMENT
>>>>>> TARGET-EPOCH
>>>>>>>>>>>> TARGET-ASSIGNMENT.
>>>>>>>>>>>>> Remove GROUP-EPOCH.
>>>>>>>>>>>>>
>>>>>>>>>>>>> For assignment value, it follows current output (reference <
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>
>>> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418
>>>>>>>>>>> ).
>>>>>>>>>>>> I think the form of `topicid-partitionid` is more clear.
>>>>>>>>>>>>> If we would like to use this form, I will update both output in
>>>>>>>>>>>> kafka-consumer-groups.sh and kafka-share-groups.sh.
>>>>>>>>>>>>>
>>>>>>>>>>>>> DJ4: It looks like DescribeGroupsResponseData only has protocol
>>>>>> type at
>>>>>>>>>>>> group level.
>>>>>>>>>>>>> Both DescribeGroupsResponseData and
>>>>>> ConsumerGroupDescribeResponseData
>>>>>>>>>>>> don’t have protocol at member level.
>>>>>>>>>>>>> Could we use a followup to add it?
>>>>>>>>>>>>>
>>>>>>>>>>>>> DJ5: Update KIP to put LEADER-EPOCH before CURRENT-OFFSET.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot
>>>>>> <djacot@confluent.io.INVALID
>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for the KIP! I have a few minor comments/suggestions:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> DJ1: In the output of `--describe --verbose`, I would suggest
>>>>>> putting
>>>>>>>>>>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before `#MEMBERS`.
>>>>>>>>>>>>>> DJ2: Continuing on the above, I assume that we will print out
>>> N/A
>>>>>> for
>>>>>>>>>>>> the
>>>>>>>>>>>>>> fields not supported by classic groups. Is this correct?
>>>>>>>>>>>>>> DJ3: In the output of `--describe --members --verbose`, I
>>> wonder
>>>>>> if we
>>>>>>>>>>>>>> should use the following order and terms: CURRENT-EPOCH
>>>>>>>>>>>> CURRENT-ASSIGNMENT
>>>>>>>>>>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the GROUP-EPOCH
>>>>>>>>>> because
>>>>>>>>>>>> it
>>>>>>>>>>>>>> is already in the group description. The value `(0)` used for
>>> the
>>>>>>>>>>>>>> assignment is incorrect. Here I suppose that we will print out
>>> the
>>>>>>>>>> list
>>>>>>>>>>>> of
>>>>>>>>>>>>>> partitions in the form of `topicid-partitionid`.
>>>>>>>>>>>>>> DJ4: Continuing on the above, I wonder if we should also add
>>> the
>>>>>>>>>>>> protocol
>>>>>>>>>>>>>> used `classic` or `consumer`. For context, it is possible to
>>> have
>>>>>>>>>>>> `classic`
>>>>>>>>>>>>>> members and `consumer` members in a `consumer` group during an
>>>>>> online
>>>>>>>>>>>>>> upgrade from the classic protocol to the consumer protocol.
>>> Having
>>>>>>>>>> this
>>>>>>>>>>>>>> information may be handy for administrators. What do you think?
>>>>>>>>>>>>>> DJ5: In the output of `--describe --offsets --verbose`, I would
>>>>>>>>>> suggest
>>>>>>>>>>>>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It would be great if we could get this one in AK 4.0 as it
>>>>>> changes the
>>>>>>>>>>>>>> output of the command.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> DJ
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <ya...@gmail.com>
>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Sean / Andrew / Lianet,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for all your review and suggestions.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> AS1, LM1, LM4: Change to add KIP-848 information when users
>>> give
>>>>>>>>>>>> —verbose
>>>>>>>>>>>>>>> option.
>>>>>>>>>>>>>>> —describe —verbose: shows group epoch and target assignment
>>>>>> epoch.
>>>>>>>>>>>>>>> —describe —members —verbose: shows above information, member
>>>>>> epoch,
>>>>>>>>>> and
>>>>>>>>>>>>>>> target assignment.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> AS2: Change to use MEMBER-EPOCH to align with KIP-848
>>> definition.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> LM2: For classic group, it doesn’t have epoch, so I use
>>> Optional
>>>>>>>>>>>> fields in
>>>>>>>>>>>>>>> ConsumerGroupDescription.
>>>>>>>>>>>>>>> For share group, it relies on KIP-848. It must have epoch, so
>>> I
>>>>>> use
>>>>>>>>>> int
>>>>>>>>>>>>>>> fields in ShareGroupDescription.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> LM3: Remove —state change. We can get group level information
>>> by
>>>>>>>>>>>> —describe
>>>>>>>>>>>>>>> —verbose.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> SQ1: Add LEADER-EPOCH when users give —describe —offsets
>>>>>> —verbose.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <li...@gmail.com>
>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hello Frank, thanks for the KIP! A few comments:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> LM1. I strongly agree with Andrew's suggestion of moving this
>>>>>> into a
>>>>>>>>>>>>>>>> --verbose option. I expect someone would only attempt to make
>>>>>> sense
>>>>>>>>>> of
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> epochs while debugging an issue, not while trying to view the
>>>>>> group
>>>>>>>>>> or
>>>>>>>>>>>>>>>> member's info. (Also member-epoch makes more sense to me,
>>> even
>>>>>> more
>>>>>>>>>>>> if we
>>>>>>>>>>>>>>>> end up agreeing in a --verbose). Related to both issues, my
>>>>>> take is
>>>>>>>>>>>> that
>>>>>>>>>>>>>>>> whoever is close to the protocol will expect member-epoch,
>>>>>> whoever
>>>>>>>>>> is
>>>>>>>>>>>> not
>>>>>>>>>>>>>>>> will probably won't even need to see the epochs at all.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> LM2. Why are the epochs defined as Optional (don't we expect
>>> to
>>>>>>>>>> always
>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>> them? with 0 initially, defined on the broker side for the
>>> group
>>>>>>>>>> ones,
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> client side for the member)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> LM3. Why is the KIP including the –state option in the
>>> proposed
>>>>>>>>>>>> changes?
>>>>>>>>>>>>>>> (
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>
>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
>>>>>>>>>>>>>>>> ).
>>>>>>>>>>>>>>>> I get that the output would change in that example, but it’s
>>> not
>>>>>>>>>>>> because
>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> any change to the –state option itself. It's because of the
>>>>>> change
>>>>>>>>>>>>>>> proposed
>>>>>>>>>>>>>>>> to the –described (required with the --state), and the
>>> changes
>>>>>> to
>>>>>>>>>>>>>>>> --describe are already explained above (seems confusing, got
>>> me
>>>>>>>>>>>> looking
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>> a change to the state filter which seemed unrelated).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> LM4. Group epoch and target assignment epoch are conceptually
>>>>>> at the
>>>>>>>>>>>>>>> group
>>>>>>>>>>>>>>>> level. vs member epoch that lands at a member level. So
>>> wonder
>>>>>> if we
>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>> show them accordingly (ex. using the --verbose option)
>>>>>>>>>>>>>>>> –describe –verbose => shows group epoch and target assignment
>>>>>> epoch
>>>>>>>>>>>>>>>> –describe –members –verbose => shows member epoch
>>>>>> (along
>>>>>>>>>>>>>>>> with group epoch and target assignment epoch)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Lianet
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield <
>>>>>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>>>>> Thanks for the KIP. I have a few comments.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> AS1: It seems to me that these new additions are most
>>> useful to
>>>>>>>>>>>> people
>>>>>>>>>>>>>>>>> trying to understand
>>>>>>>>>>>>>>>>> the progress of rebalancing in quite some detail. The
>>>>>> information
>>>>>>>>>> is
>>>>>>>>>>>>>>>>> really not understandable
>>>>>>>>>>>>>>>>> for most users who do not have deep knowledge of
>>> KIP-848/932.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> As a result, I suggest for kafka-share-groups.sh that you
>>> add a
>>>>>>>>>>>>>>> --members
>>>>>>>>>>>>>>>>> --verbose option
>>>>>>>>>>>>>>>>> and only include the new information in the output for that
>>>>>> option,
>>>>>>>>>>>>>>> rather
>>>>>>>>>>>>>>>>> than changing the
>>>>>>>>>>>>>>>>> non-verbose --members output.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I also make a similar suggestion for
>>> kafka-consumer-groups.sh
>>>>>>>>>>>> --members
>>>>>>>>>>>>>>>>> and only add the
>>>>>>>>>>>>>>>>> new information for the --verbose output.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> AS2: I strongly suggest that you use MEMBER-EPOCH instead of
>>>>>>>>>>>>>>>>> CONSUMER-EPOCH.
>>>>>>>>>>>>>>>>> I think it's more confusing not following the terminology of
>>>>>> the
>>>>>>>>>>>> KIPs.
>>>>>>>>>>>>>>> For
>>>>>>>>>>>>>>>>> one thing,
>>>>>>>>>>>>>>>>> we already have "member" in the admin client such as
>>>>>>>>>>>> MemberDescription.
>>>>>>>>>>>>>>>>> It's not a
>>>>>>>>>>>>>>>>> new term.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Andrew
>>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>>> From: PoAn Yang <pa...@apache.org>
>>>>>>>>>>>>>>>>> Sent: 25 October 2024 13:55
>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
>>> kafka-consumer-groups
>>>>>>>>>> command
>>>>>>>>>>>>>>> line
>>>>>>>>>>>>>>>>> tool to support new consumer group
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Lucas,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for the review!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 1) Yes, I add related change for kafka-share-groups.sh to
>>> the
>>>>>> KIP.
>>>>>>>>>>>> Could
>>>>>>>>>>>>>>>>> you take a look? Thanks for the suggestion.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use MEMBER-EPOCH
>>>>>> here,
>>>>>>>>>>>> users
>>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>>> confuse what is different between CONSUMER and MEMBER.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
>>>>>>>>>>>>>>>>>> Hi Frank,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> thanks for the KIP!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 1) For consistency, should we do the same for
>>>>>>>>>>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription, etc. ? Even
>>> if
>>>>>> we do
>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>> implement it right now if the share group implementation
>>> may
>>>>>> still
>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>> incomplete, it may make sense to include it in the KIP.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH? That would
>>>>>> seem
>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>> consistent.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>> Lucas
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <
>>>>>> yangpoan@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I would like to kick off the discussion of KIP-1099. This
>>> KIP
>>>>>>>>>>>> enhances
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> kafka-consumer-groups tools to include state which is
>>>>>> introduced
>>>>>>>>>> by
>>>>>>>>>>>>>>>>> KIP-848.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups command line tool
>>> to
>>>>>>>>>> support
>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>> consumer group - Apache Kafka - Apache Software Foundation
>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> cwiki.apache.org
>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [image: favicon.ico]
>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
>>>
>>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by Jeff Kim <je...@apache.org>.
Hi PoAn,
JK2: Can we state in the KIP that we will reformat the existing
classic group's `--members --verbose` as well, specifically the
"ASSIGNMENT" column to include the topic name?
Thanks!
On 2024/11/21 14:33:37 "Lianet M." wrote:
> Thanks for the updates PoAn!
>
> LM6. Just a nit, under the --describe --state --verbose the header is fine
> but the example is still missing the --state argument.
>
> LM7. Under the Proposed changes for kafka-share-groups.sh, should we
> clearly mention that the KIP adds a new --verbose option? I think it's
> confusing because it's presented as if the option already exists and we're
> only changing the output (which is the case for kafka-consumer-groups but
> not for kafka-share-groups)
>
> That's all on my side. Thanks!
>
> Lianet
>
> On Wed, Nov 20, 2024 at 11:12 PM PoAn Yang <ya...@gmail.com> wrote:
>
> > Hi Lianet / Jeff,
> >
> > Thanks for review and suggestions.
> >
> > LM4: At group level, the protocol field is from ModernGroup#protocolType.
> > It’s a string.
> > The value is defined in different places like ShareGroup#PROTOCOL_TYPE and
> > ConsumerProtocol#PROTOCOL_TYPE.
> > In KIP-1043 <
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=305171038#KIP1043:Administrationofgroups-kafka-groups.sh>,
> > the kafka-groups.sh also shows protocol like “classic” / “consumer” /
> > “share”.
> > I agree it’s better to use a type like GroupProtocol enum, but I would
> > like to align the format with kafka-groups.sh.
> > How about we use another Jira to add “SHARE” to GroupProtocol and change
> > return type of ModernGroup#protocolType?
> >
> > LM5 / JK1: Sorry, I misunderstood LM3, so I deleted —state in the title.
> > You’re correct. I would like to use `—describe —state —verbose` to show
> > group level information.
> > For `—describe —verbose`, it will show output as same as `—describe
> > —offsets —verbose`,
> > because `—describe` also shows output as same as `—describe —offsets`.
> >
> > JK2: Yes, I only use consumer group in the sample output, but the classic
> > group will use the same format as well.
> >
> > Thanks,
> > PoAn
> >
> > > On Nov 21, 2024, at 8:28 AM, Jeff Kim <je...@apache.org> wrote:
> > >
> > > Hi PoAn,
> > >
> > > Thanks for the KIP. I have some questions:
> > >
> > > JK1: (Realized this is similar to LM5) On `--describe --verbose`
> > proposed changes - doesn't `--describe $group` default to printing the
> > offsets? Perhaps you're referring to the `--state` argument? Also, would
> > that mean the default `--describe $group --verbose` command would not print
> > the added field to `--offsets verbose` (leader-epoch) or would it?
> > >
> > > JK2: Looking at ConsumerGroupCommand.java, the existing "ASSIGNMENT"
> > column under `--members --verbose` does group the topic partitions by topic
> > but does not prefix the grouping with the topic name like you're proposal:
> > "my_topic:0,1;new_topic:0,1". Should we do apply the same format for the
> > classic group as well?
> > >
> > >
> > >
> > > On 2024/11/20 21:32:41 "Lianet M." wrote:
> > >> Hello PoAn, just of couple more minor comments:
> > >>
> > >> LM4. Regarding the new “protocol” field added to MemberDescription.
> > Should
> > >> we consider reusing the existing GroupProtocol enum instead of String?
> > >> (It’s the one we use from the consumer side to refer to the protocol in
> > >> use, just missing Share I notice).
> > >>
> > >> LM5. Regarding the change to the output for kafka-consumer-groups, the
> > >> command shown does not include the —state option, but the output shows
> > >> state info (state, #members, epochs). I would guess that we want to
> > modify
> > >> the output only when we describe a group with the —state —verbose
> > option,
> > >> is my understanding right? If my understanding is right we’re just
> > missing
> > >> adding the —state in the example, and the KIP does not introduce any
> > >> changes to the —describe —verbose option. (If not, it would mean a
> > bigger
> > >> change to the output of —describe —verbose which I expect is not the
> > >> intention?)
> > >>
> > >> Thanks!
> > >>
> > >> Lianet
> > >>
> > >>
> > >> On Wed, Nov 20, 2024 at 2:12 AM PoAn Yang <ya...@gmail.com> wrote:
> > >>
> > >>> Hi Chia-Ping,
> > >>>
> > >>> Thanks for the review and suggestions.
> > >>>
> > >>> CI0: Thanks for the reminder. Update validVersions in
> > >>> ConsumerGroupDescribeRequest to 0-1.
> > >>>
> > >>> CI1: Yes, I use `ConsumerGroupMember#useClassicProtocol` to check
> > whether
> > >>> a member in consumer group uses “classic” or “consumer” protocol.
> > >>>
> > >>> CI2: Yes, a member in share group always uses “share” protocol.
> > >>>
> > >>> CI3: Add a table to show meaning of “classic”, “consumer”, and “share”
> > >>> protocol.
> > >>>
> > >>> BTW, the vote thread is in
> > >>> https://lists.apache.org/thread/rb25tf75tzf4c7jqqldlo5jh9w8chsq6.
> > >>>
> > >>> Thanks,
> > >>> PoAn
> > >>>
> > >>>> On Nov 20, 2024, at 11:46 AM, Chia-Ping Tsai <ch...@apache.org>
> > >>> wrote:
> > >>>>
> > >>>> hi PoAn
> > >>>>
> > >>>> CI0: We have to bump the version of ConsumerGroupDescribeRequest as
> > >>> well, so server can distinguish the new and old behavior.
> > >>>>
> > >>>> CI1: the type of new filed is string, so I guess you plan to use
> > >>> `ConsumerGroupMember#useClassicProtocol` [0] flag to return either
> > >>> "classic" or "consumer", right?
> > >>>>
> > >>>> CI2: `MemberDescription` is used by `ShareGroupDescription` too, so
> > the
> > >>> new filed protocol in shared group is always "shared", right?
> > >>>>
> > >>>> CI3: Could you consider adding a table to show the value of the
> > protocol
> > >>> field in each case? Andrew has a beautiful table in KIP-1043 that
> > lists all
> > >>> possible protocol names.
> > >>>>
> > >>>> [0]
> > >>>
> > https://github.com/apache/kafka/blob/441a6d0b790f4a17b454caeea7588a6b90fbd9db/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/consumer/ConsumerGroupMember.java#L454
> > >>>
> > >>>>
> > >>>> Best,
> > >>>> Chia-Ping
> > >>>>
> > >>>> On 2024/11/19 15:45:16 PoAn Yang wrote:
> > >>>>> Hi David,
> > >>>>>
> > >>>>> Thanks for the reminder.
> > >>>>>
> > >>>>> DJ4: Add related content to the KIP.
> > >>>>> Bump validVersions in ConsumerGroupDescribeResponse to “0-1”.
> > >>>>> Add a new field “Protocol” to ConsumerGroupDescribeResponse.
> > >>>>>
> > >>>>> Thanks,
> > >>>>> PoAn
> > >>>>>
> > >>>>>> On Nov 19, 2024, at 3:57 PM, David Jacot
> > <dj...@confluent.io.INVALID>
> > >>> wrote:
> > >>>>>>
> > >>>>>> Hi PoAn,
> > >>>>>>
> > >>>>>> DJ4: Could you please also include the required changes to the
> > >>>>>> ConsumerGroupDescribed API in the public interface section? We
> > >>> basically
> > >>>>>> need to bump the version, add the new field, etc.
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> David
> > >>>>>>
> > >>>>>> On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang <ya...@gmail.com>
> > wrote:
> > >>>>>>
> > >>>>>>> Hi David,
> > >>>>>>>
> > >>>>>>> DJ4: Add a new field protocol to MemberDescription,
> > >>>>>>> so the command line tool can show protocol information when users
> > >>> describe
> > >>>>>>> members.
> > >>>>>>>
> > >>>>>>> If there is no further suggestion, I will start a vote thread
> > today.
> > >>>>>>>
> > >>>>>>> Thanks,
> > >>>>>>> PoAn
> > >>>>>>>
> > >>>>>>>> On Nov 15, 2024, at 11:50 PM, David Jacot
> > >>> <dj...@confluent.io.INVALID>
> > >>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> Hi,
> > >>>>>>>>
> > >>>>>>>> DJ2: Using "-" sounds good to me.
> > >>>>>>>>
> > >>>>>>>> DJ3: That seems reasonable to me.
> > >>>>>>>>
> > >>>>>>>> DJ4: Why not add it right now? I don't want to change the output
> > of
> > >>> the
> > >>>>>>>> tool too many times.
> > >>>>>>>>
> > >>>>>>>> Best,
> > >>>>>>>> David
> > >>>>>>>>
> > >>>>>>>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang <ya...@gmail.com>
> > >>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Hi David / Andrew,
> > >>>>>>>>>
> > >>>>>>>>> Thanks for review. Thanks Andrew for picking up
> > >>> kafka-share-groups.sh
> > >>>>>>>>> implementation.
> > >>>>>>>>> I will handle kafka-consumer-groups.sh.
> > >>>>>>>>>
> > >>>>>>>>> DJ3: After discussing with @Chia-Ping Tsai, we think that using
> > new
> > >>>>>>> format
> > >>>>>>>>> is more clear.
> > >>>>>>>>>
> > >>>>>>>>> The new format will be like
> > >>>>>>>>>
> > >>>>>>>
> > >>>
> > <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2>
> > >>>>>>>>>
> > >>>>>>>>> Using colon(:) to concat topic name and partition IDs.
> > >>>>>>>>> Using comma(,) to concat partition IDs within same topic name.
> > >>>>>>>>> Using semicolon(;) to concat topic strings.
> > >>>>>>>>>
> > >>>>>>>>> AS3: Fix it with kafka-share-groups.sh. Thanks.
> > >>>>>>>>>
> > >>>>>>>>> If there is no further suggestion, I will start a vote thread
> > next
> > >>>>>>> Monday.
> > >>>>>>>>>
> > >>>>>>>>> Thanks,
> > >>>>>>>>> PoAn
> > >>>>>>>>>
> > >>>>>>>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield <
> > >>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> Hi PoAn,
> > >>>>>>>>>> DJ2: I was just going to comment that "-" would be a more
> > >>> appropriate
> > >>>>>>>>> missing value, but
> > >>>>>>>>>> you got there first.
> > >>>>>>>>>>
> > >>>>>>>>>> AS3: The examples for kafka-share-groups.sh include
> > >>>>>>>>> kafka-consumer-groups.sh in the
> > >>>>>>>>>> command line.
> > >>>>>>>>>>
> > >>>>>>>>>> If this is accepted in time, I'm happy to pick up the
> > >>> implementation of
> > >>>>>>>>> the share groups
> > >>>>>>>>>> part of this if it helps.
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks,
> > >>>>>>>>>> Andrew
> > >>>>>>>>>>
> > >>>>>>>>>> ________________________________________
> > >>>>>>>>>> From: Frank Yang <ya...@gmail.com>
> > >>>>>>>>>> Sent: 14 November 2024 10:48
> > >>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
> > >>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups
> > >>> command
> > >>>>>>>>> line tool to support new consumer group
> > >>>>>>>>>>
> > >>>>>>>>>> Hi David,
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks for the review and suggestion! I would like to get this
> > in
> > >>> AK
> > >>>>>>> 4.0
> > >>>>>>>>> as well. I will do my best.
> > >>>>>>>>>>
> > >>>>>>>>>> DJ1: Update KIP to put GROUP-EPOCH and TARGET-ASSIGNMENT-EPOCH
> > >>> before
> > >>>>>>>>> #MEMBERS.
> > >>>>>>>>>>
> > >>>>>>>>>> DJ2: I prefer to follow current missing column value “-“.
> > >>> (reference <
> > >>>>>>>>>
> > >>>>>>>
> > >>>
> > https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92
> > >>>>>>>>>> )
> > >>>>>>>>>>
> > >>>>>>>>>> DJ3: Update KIP to use CURRENT-EPOCH CURRENT-ASSIGNMENT
> > >>> TARGET-EPOCH
> > >>>>>>>>> TARGET-ASSIGNMENT.
> > >>>>>>>>>> Remove GROUP-EPOCH.
> > >>>>>>>>>>
> > >>>>>>>>>> For assignment value, it follows current output (reference <
> > >>>>>>>>>
> > >>>>>>>
> > >>>
> > https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418
> > >>>>>>>> ).
> > >>>>>>>>> I think the form of `topicid-partitionid` is more clear.
> > >>>>>>>>>> If we would like to use this form, I will update both output in
> > >>>>>>>>> kafka-consumer-groups.sh and kafka-share-groups.sh.
> > >>>>>>>>>>
> > >>>>>>>>>> DJ4: It looks like DescribeGroupsResponseData only has protocol
> > >>> type at
> > >>>>>>>>> group level.
> > >>>>>>>>>> Both DescribeGroupsResponseData and
> > >>> ConsumerGroupDescribeResponseData
> > >>>>>>>>> don’t have protocol at member level.
> > >>>>>>>>>> Could we use a followup to add it?
> > >>>>>>>>>>
> > >>>>>>>>>> DJ5: Update KIP to put LEADER-EPOCH before CURRENT-OFFSET.
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks,
> > >>>>>>>>>> PoAn
> > >>>>>>>>>>
> > >>>>>>>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot
> > >>> <djacot@confluent.io.INVALID
> > >>>>>>>>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> Hi PoAn,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks for the KIP! I have a few minor comments/suggestions:
> > >>>>>>>>>>>
> > >>>>>>>>>>> DJ1: In the output of `--describe --verbose`, I would suggest
> > >>> putting
> > >>>>>>>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before `#MEMBERS`.
> > >>>>>>>>>>> DJ2: Continuing on the above, I assume that we will print out
> > N/A
> > >>> for
> > >>>>>>>>> the
> > >>>>>>>>>>> fields not supported by classic groups. Is this correct?
> > >>>>>>>>>>> DJ3: In the output of `--describe --members --verbose`, I
> > wonder
> > >>> if we
> > >>>>>>>>>>> should use the following order and terms: CURRENT-EPOCH
> > >>>>>>>>> CURRENT-ASSIGNMENT
> > >>>>>>>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the GROUP-EPOCH
> > >>>>>>> because
> > >>>>>>>>> it
> > >>>>>>>>>>> is already in the group description. The value `(0)` used for
> > the
> > >>>>>>>>>>> assignment is incorrect. Here I suppose that we will print out
> > the
> > >>>>>>> list
> > >>>>>>>>> of
> > >>>>>>>>>>> partitions in the form of `topicid-partitionid`.
> > >>>>>>>>>>> DJ4: Continuing on the above, I wonder if we should also add
> > the
> > >>>>>>>>> protocol
> > >>>>>>>>>>> used `classic` or `consumer`. For context, it is possible to
> > have
> > >>>>>>>>> `classic`
> > >>>>>>>>>>> members and `consumer` members in a `consumer` group during an
> > >>> online
> > >>>>>>>>>>> upgrade from the classic protocol to the consumer protocol.
> > Having
> > >>>>>>> this
> > >>>>>>>>>>> information may be handy for administrators. What do you think?
> > >>>>>>>>>>> DJ5: In the output of `--describe --offsets --verbose`, I would
> > >>>>>>> suggest
> > >>>>>>>>>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`.
> > >>>>>>>>>>>
> > >>>>>>>>>>> It would be great if we could get this one in AK 4.0 as it
> > >>> changes the
> > >>>>>>>>>>> output of the command.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks,
> > >>>>>>>>>>> DJ
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <ya...@gmail.com>
> > >>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hi Sean / Andrew / Lianet,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks for all your review and suggestions.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> AS1, LM1, LM4: Change to add KIP-848 information when users
> > give
> > >>>>>>>>> —verbose
> > >>>>>>>>>>>> option.
> > >>>>>>>>>>>> —describe —verbose: shows group epoch and target assignment
> > >>> epoch.
> > >>>>>>>>>>>> —describe —members —verbose: shows above information, member
> > >>> epoch,
> > >>>>>>> and
> > >>>>>>>>>>>> target assignment.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> AS2: Change to use MEMBER-EPOCH to align with KIP-848
> > definition.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> LM2: For classic group, it doesn’t have epoch, so I use
> > Optional
> > >>>>>>>>> fields in
> > >>>>>>>>>>>> ConsumerGroupDescription.
> > >>>>>>>>>>>> For share group, it relies on KIP-848. It must have epoch, so
> > I
> > >>> use
> > >>>>>>> int
> > >>>>>>>>>>>> fields in ShareGroupDescription.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> LM3: Remove —state change. We can get group level information
> > by
> > >>>>>>>>> —describe
> > >>>>>>>>>>>> —verbose.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> SQ1: Add LEADER-EPOCH when users give —describe —offsets
> > >>> —verbose.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks.
> > >>>>>>>>>>>> PoAn
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <li...@gmail.com>
> > >>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Hello Frank, thanks for the KIP! A few comments:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> LM1. I strongly agree with Andrew's suggestion of moving this
> > >>> into a
> > >>>>>>>>>>>>> --verbose option. I expect someone would only attempt to make
> > >>> sense
> > >>>>>>> of
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>> epochs while debugging an issue, not while trying to view the
> > >>> group
> > >>>>>>> or
> > >>>>>>>>>>>>> member's info. (Also member-epoch makes more sense to me,
> > even
> > >>> more
> > >>>>>>>>> if we
> > >>>>>>>>>>>>> end up agreeing in a --verbose). Related to both issues, my
> > >>> take is
> > >>>>>>>>> that
> > >>>>>>>>>>>>> whoever is close to the protocol will expect member-epoch,
> > >>> whoever
> > >>>>>>> is
> > >>>>>>>>> not
> > >>>>>>>>>>>>> will probably won't even need to see the epochs at all.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> LM2. Why are the epochs defined as Optional (don't we expect
> > to
> > >>>>>>> always
> > >>>>>>>>>>>> have
> > >>>>>>>>>>>>> them? with 0 initially, defined on the broker side for the
> > group
> > >>>>>>> ones,
> > >>>>>>>>>>>> and
> > >>>>>>>>>>>>> client side for the member)
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> LM3. Why is the KIP including the –state option in the
> > proposed
> > >>>>>>>>> changes?
> > >>>>>>>>>>>> (
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
> > >>>>>>>>>>>>> ).
> > >>>>>>>>>>>>> I get that the output would change in that example, but it’s
> > not
> > >>>>>>>>> because
> > >>>>>>>>>>>> of
> > >>>>>>>>>>>>> any change to the –state option itself. It's because of the
> > >>> change
> > >>>>>>>>>>>> proposed
> > >>>>>>>>>>>>> to the –described (required with the --state), and the
> > changes
> > >>> to
> > >>>>>>>>>>>>> --describe are already explained above (seems confusing, got
> > me
> > >>>>>>>>> looking
> > >>>>>>>>>>>> for
> > >>>>>>>>>>>>> a change to the state filter which seemed unrelated).
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> LM4. Group epoch and target assignment epoch are conceptually
> > >>> at the
> > >>>>>>>>>>>> group
> > >>>>>>>>>>>>> level. vs member epoch that lands at a member level. So
> > wonder
> > >>> if we
> > >>>>>>>>>>>> should
> > >>>>>>>>>>>>> show them accordingly (ex. using the --verbose option)
> > >>>>>>>>>>>>> –describe –verbose => shows group epoch and target assignment
> > >>> epoch
> > >>>>>>>>>>>>> –describe –members –verbose => shows member epoch
> > >>> (along
> > >>>>>>>>>>>>> with group epoch and target assignment epoch)
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks!
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Lianet
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield <
> > >>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hi PoAn,
> > >>>>>>>>>>>>>> Thanks for the KIP. I have a few comments.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> AS1: It seems to me that these new additions are most
> > useful to
> > >>>>>>>>> people
> > >>>>>>>>>>>>>> trying to understand
> > >>>>>>>>>>>>>> the progress of rebalancing in quite some detail. The
> > >>> information
> > >>>>>>> is
> > >>>>>>>>>>>>>> really not understandable
> > >>>>>>>>>>>>>> for most users who do not have deep knowledge of
> > KIP-848/932.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> As a result, I suggest for kafka-share-groups.sh that you
> > add a
> > >>>>>>>>>>>> --members
> > >>>>>>>>>>>>>> --verbose option
> > >>>>>>>>>>>>>> and only include the new information in the output for that
> > >>> option,
> > >>>>>>>>>>>> rather
> > >>>>>>>>>>>>>> than changing the
> > >>>>>>>>>>>>>> non-verbose --members output.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I also make a similar suggestion for
> > kafka-consumer-groups.sh
> > >>>>>>>>> --members
> > >>>>>>>>>>>>>> and only add the
> > >>>>>>>>>>>>>> new information for the --verbose output.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> AS2: I strongly suggest that you use MEMBER-EPOCH instead of
> > >>>>>>>>>>>>>> CONSUMER-EPOCH.
> > >>>>>>>>>>>>>> I think it's more confusing not following the terminology of
> > >>> the
> > >>>>>>>>> KIPs.
> > >>>>>>>>>>>> For
> > >>>>>>>>>>>>>> one thing,
> > >>>>>>>>>>>>>> we already have "member" in the admin client such as
> > >>>>>>>>> MemberDescription.
> > >>>>>>>>>>>>>> It's not a
> > >>>>>>>>>>>>>> new term.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>> Andrew
> > >>>>>>>>>>>>>> ________________________________________
> > >>>>>>>>>>>>>> From: PoAn Yang <pa...@apache.org>
> > >>>>>>>>>>>>>> Sent: 25 October 2024 13:55
> > >>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
> > >>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
> > kafka-consumer-groups
> > >>>>>>> command
> > >>>>>>>>>>>> line
> > >>>>>>>>>>>>>> tool to support new consumer group
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hi Lucas,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks for the review!
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> 1) Yes, I add related change for kafka-share-groups.sh to
> > the
> > >>> KIP.
> > >>>>>>>>> Could
> > >>>>>>>>>>>>>> you take a look? Thanks for the suggestion.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use MEMBER-EPOCH
> > >>> here,
> > >>>>>>>>> users
> > >>>>>>>>>>>> may
> > >>>>>>>>>>>>>> confuse what is different between CONSUMER and MEMBER.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>> PoAn
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
> > >>>>>>>>>>>>>>> Hi Frank,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> thanks for the KIP!
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> 1) For consistency, should we do the same for
> > >>>>>>>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription, etc. ? Even
> > if
> > >>> we do
> > >>>>>>>>> not
> > >>>>>>>>>>>>>>> implement it right now if the share group implementation
> > may
> > >>> still
> > >>>>>>>>> be
> > >>>>>>>>>>>>>>> incomplete, it may make sense to include it in the KIP.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH? That would
> > >>> seem
> > >>>>>>>>> more
> > >>>>>>>>>>>>>>> consistent.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>> Lucas
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <
> > >>> yangpoan@gmail.com>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> I would like to kick off the discussion of KIP-1099. This
> > KIP
> > >>>>>>>>> enhances
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> kafka-consumer-groups tools to include state which is
> > >>> introduced
> > >>>>>>> by
> > >>>>>>>>>>>>>> KIP-848.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups command line tool
> > to
> > >>>>>>> support
> > >>>>>>>>>>>> new
> > >>>>>>>>>>>>>>>> consumer group - Apache Kafka - Apache Software Foundation
> > >>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> cwiki.apache.org
> > >>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> [image: favicon.ico]
> > >>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thank you,
> > >>>>>>>>>>>>>>>> PoAn
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>>
> > >>>
> > >>>
> > >>
> >
> >
>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by "Lianet M." <li...@gmail.com>.
Thanks for the updates PoAn!
LM6. Just a nit, under the --describe --state --verbose the header is fine
but the example is still missing the --state argument.
LM7. Under the Proposed changes for kafka-share-groups.sh, should we
clearly mention that the KIP adds a new --verbose option? I think it's
confusing because it's presented as if the option already exists and we're
only changing the output (which is the case for kafka-consumer-groups but
not for kafka-share-groups)
That's all on my side. Thanks!
Lianet
On Wed, Nov 20, 2024 at 11:12 PM PoAn Yang <ya...@gmail.com> wrote:
> Hi Lianet / Jeff,
>
> Thanks for review and suggestions.
>
> LM4: At group level, the protocol field is from ModernGroup#protocolType.
> It’s a string.
> The value is defined in different places like ShareGroup#PROTOCOL_TYPE and
> ConsumerProtocol#PROTOCOL_TYPE.
> In KIP-1043 <
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=305171038#KIP1043:Administrationofgroups-kafka-groups.sh>,
> the kafka-groups.sh also shows protocol like “classic” / “consumer” /
> “share”.
> I agree it’s better to use a type like GroupProtocol enum, but I would
> like to align the format with kafka-groups.sh.
> How about we use another Jira to add “SHARE” to GroupProtocol and change
> return type of ModernGroup#protocolType?
>
> LM5 / JK1: Sorry, I misunderstood LM3, so I deleted —state in the title.
> You’re correct. I would like to use `—describe —state —verbose` to show
> group level information.
> For `—describe —verbose`, it will show output as same as `—describe
> —offsets —verbose`,
> because `—describe` also shows output as same as `—describe —offsets`.
>
> JK2: Yes, I only use consumer group in the sample output, but the classic
> group will use the same format as well.
>
> Thanks,
> PoAn
>
> > On Nov 21, 2024, at 8:28 AM, Jeff Kim <je...@apache.org> wrote:
> >
> > Hi PoAn,
> >
> > Thanks for the KIP. I have some questions:
> >
> > JK1: (Realized this is similar to LM5) On `--describe --verbose`
> proposed changes - doesn't `--describe $group` default to printing the
> offsets? Perhaps you're referring to the `--state` argument? Also, would
> that mean the default `--describe $group --verbose` command would not print
> the added field to `--offsets verbose` (leader-epoch) or would it?
> >
> > JK2: Looking at ConsumerGroupCommand.java, the existing "ASSIGNMENT"
> column under `--members --verbose` does group the topic partitions by topic
> but does not prefix the grouping with the topic name like you're proposal:
> "my_topic:0,1;new_topic:0,1". Should we do apply the same format for the
> classic group as well?
> >
> >
> >
> > On 2024/11/20 21:32:41 "Lianet M." wrote:
> >> Hello PoAn, just of couple more minor comments:
> >>
> >> LM4. Regarding the new “protocol” field added to MemberDescription.
> Should
> >> we consider reusing the existing GroupProtocol enum instead of String?
> >> (It’s the one we use from the consumer side to refer to the protocol in
> >> use, just missing Share I notice).
> >>
> >> LM5. Regarding the change to the output for kafka-consumer-groups, the
> >> command shown does not include the —state option, but the output shows
> >> state info (state, #members, epochs). I would guess that we want to
> modify
> >> the output only when we describe a group with the —state —verbose
> option,
> >> is my understanding right? If my understanding is right we’re just
> missing
> >> adding the —state in the example, and the KIP does not introduce any
> >> changes to the —describe —verbose option. (If not, it would mean a
> bigger
> >> change to the output of —describe —verbose which I expect is not the
> >> intention?)
> >>
> >> Thanks!
> >>
> >> Lianet
> >>
> >>
> >> On Wed, Nov 20, 2024 at 2:12 AM PoAn Yang <ya...@gmail.com> wrote:
> >>
> >>> Hi Chia-Ping,
> >>>
> >>> Thanks for the review and suggestions.
> >>>
> >>> CI0: Thanks for the reminder. Update validVersions in
> >>> ConsumerGroupDescribeRequest to 0-1.
> >>>
> >>> CI1: Yes, I use `ConsumerGroupMember#useClassicProtocol` to check
> whether
> >>> a member in consumer group uses “classic” or “consumer” protocol.
> >>>
> >>> CI2: Yes, a member in share group always uses “share” protocol.
> >>>
> >>> CI3: Add a table to show meaning of “classic”, “consumer”, and “share”
> >>> protocol.
> >>>
> >>> BTW, the vote thread is in
> >>> https://lists.apache.org/thread/rb25tf75tzf4c7jqqldlo5jh9w8chsq6.
> >>>
> >>> Thanks,
> >>> PoAn
> >>>
> >>>> On Nov 20, 2024, at 11:46 AM, Chia-Ping Tsai <ch...@apache.org>
> >>> wrote:
> >>>>
> >>>> hi PoAn
> >>>>
> >>>> CI0: We have to bump the version of ConsumerGroupDescribeRequest as
> >>> well, so server can distinguish the new and old behavior.
> >>>>
> >>>> CI1: the type of new filed is string, so I guess you plan to use
> >>> `ConsumerGroupMember#useClassicProtocol` [0] flag to return either
> >>> "classic" or "consumer", right?
> >>>>
> >>>> CI2: `MemberDescription` is used by `ShareGroupDescription` too, so
> the
> >>> new filed protocol in shared group is always "shared", right?
> >>>>
> >>>> CI3: Could you consider adding a table to show the value of the
> protocol
> >>> field in each case? Andrew has a beautiful table in KIP-1043 that
> lists all
> >>> possible protocol names.
> >>>>
> >>>> [0]
> >>>
> https://github.com/apache/kafka/blob/441a6d0b790f4a17b454caeea7588a6b90fbd9db/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/consumer/ConsumerGroupMember.java#L454
> >>>
> >>>>
> >>>> Best,
> >>>> Chia-Ping
> >>>>
> >>>> On 2024/11/19 15:45:16 PoAn Yang wrote:
> >>>>> Hi David,
> >>>>>
> >>>>> Thanks for the reminder.
> >>>>>
> >>>>> DJ4: Add related content to the KIP.
> >>>>> Bump validVersions in ConsumerGroupDescribeResponse to “0-1”.
> >>>>> Add a new field “Protocol” to ConsumerGroupDescribeResponse.
> >>>>>
> >>>>> Thanks,
> >>>>> PoAn
> >>>>>
> >>>>>> On Nov 19, 2024, at 3:57 PM, David Jacot
> <dj...@confluent.io.INVALID>
> >>> wrote:
> >>>>>>
> >>>>>> Hi PoAn,
> >>>>>>
> >>>>>> DJ4: Could you please also include the required changes to the
> >>>>>> ConsumerGroupDescribed API in the public interface section? We
> >>> basically
> >>>>>> need to bump the version, add the new field, etc.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> David
> >>>>>>
> >>>>>> On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang <ya...@gmail.com>
> wrote:
> >>>>>>
> >>>>>>> Hi David,
> >>>>>>>
> >>>>>>> DJ4: Add a new field protocol to MemberDescription,
> >>>>>>> so the command line tool can show protocol information when users
> >>> describe
> >>>>>>> members.
> >>>>>>>
> >>>>>>> If there is no further suggestion, I will start a vote thread
> today.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> PoAn
> >>>>>>>
> >>>>>>>> On Nov 15, 2024, at 11:50 PM, David Jacot
> >>> <dj...@confluent.io.INVALID>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>> DJ2: Using "-" sounds good to me.
> >>>>>>>>
> >>>>>>>> DJ3: That seems reasonable to me.
> >>>>>>>>
> >>>>>>>> DJ4: Why not add it right now? I don't want to change the output
> of
> >>> the
> >>>>>>>> tool too many times.
> >>>>>>>>
> >>>>>>>> Best,
> >>>>>>>> David
> >>>>>>>>
> >>>>>>>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang <ya...@gmail.com>
> >>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi David / Andrew,
> >>>>>>>>>
> >>>>>>>>> Thanks for review. Thanks Andrew for picking up
> >>> kafka-share-groups.sh
> >>>>>>>>> implementation.
> >>>>>>>>> I will handle kafka-consumer-groups.sh.
> >>>>>>>>>
> >>>>>>>>> DJ3: After discussing with @Chia-Ping Tsai, we think that using
> new
> >>>>>>> format
> >>>>>>>>> is more clear.
> >>>>>>>>>
> >>>>>>>>> The new format will be like
> >>>>>>>>>
> >>>>>>>
> >>>
> <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2>
> >>>>>>>>>
> >>>>>>>>> Using colon(:) to concat topic name and partition IDs.
> >>>>>>>>> Using comma(,) to concat partition IDs within same topic name.
> >>>>>>>>> Using semicolon(;) to concat topic strings.
> >>>>>>>>>
> >>>>>>>>> AS3: Fix it with kafka-share-groups.sh. Thanks.
> >>>>>>>>>
> >>>>>>>>> If there is no further suggestion, I will start a vote thread
> next
> >>>>>>> Monday.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> PoAn
> >>>>>>>>>
> >>>>>>>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield <
> >>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hi PoAn,
> >>>>>>>>>> DJ2: I was just going to comment that "-" would be a more
> >>> appropriate
> >>>>>>>>> missing value, but
> >>>>>>>>>> you got there first.
> >>>>>>>>>>
> >>>>>>>>>> AS3: The examples for kafka-share-groups.sh include
> >>>>>>>>> kafka-consumer-groups.sh in the
> >>>>>>>>>> command line.
> >>>>>>>>>>
> >>>>>>>>>> If this is accepted in time, I'm happy to pick up the
> >>> implementation of
> >>>>>>>>> the share groups
> >>>>>>>>>> part of this if it helps.
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> Andrew
> >>>>>>>>>>
> >>>>>>>>>> ________________________________________
> >>>>>>>>>> From: Frank Yang <ya...@gmail.com>
> >>>>>>>>>> Sent: 14 November 2024 10:48
> >>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
> >>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups
> >>> command
> >>>>>>>>> line tool to support new consumer group
> >>>>>>>>>>
> >>>>>>>>>> Hi David,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for the review and suggestion! I would like to get this
> in
> >>> AK
> >>>>>>> 4.0
> >>>>>>>>> as well. I will do my best.
> >>>>>>>>>>
> >>>>>>>>>> DJ1: Update KIP to put GROUP-EPOCH and TARGET-ASSIGNMENT-EPOCH
> >>> before
> >>>>>>>>> #MEMBERS.
> >>>>>>>>>>
> >>>>>>>>>> DJ2: I prefer to follow current missing column value “-“.
> >>> (reference <
> >>>>>>>>>
> >>>>>>>
> >>>
> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92
> >>>>>>>>>> )
> >>>>>>>>>>
> >>>>>>>>>> DJ3: Update KIP to use CURRENT-EPOCH CURRENT-ASSIGNMENT
> >>> TARGET-EPOCH
> >>>>>>>>> TARGET-ASSIGNMENT.
> >>>>>>>>>> Remove GROUP-EPOCH.
> >>>>>>>>>>
> >>>>>>>>>> For assignment value, it follows current output (reference <
> >>>>>>>>>
> >>>>>>>
> >>>
> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418
> >>>>>>>> ).
> >>>>>>>>> I think the form of `topicid-partitionid` is more clear.
> >>>>>>>>>> If we would like to use this form, I will update both output in
> >>>>>>>>> kafka-consumer-groups.sh and kafka-share-groups.sh.
> >>>>>>>>>>
> >>>>>>>>>> DJ4: It looks like DescribeGroupsResponseData only has protocol
> >>> type at
> >>>>>>>>> group level.
> >>>>>>>>>> Both DescribeGroupsResponseData and
> >>> ConsumerGroupDescribeResponseData
> >>>>>>>>> don’t have protocol at member level.
> >>>>>>>>>> Could we use a followup to add it?
> >>>>>>>>>>
> >>>>>>>>>> DJ5: Update KIP to put LEADER-EPOCH before CURRENT-OFFSET.
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> PoAn
> >>>>>>>>>>
> >>>>>>>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot
> >>> <djacot@confluent.io.INVALID
> >>>>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for the KIP! I have a few minor comments/suggestions:
> >>>>>>>>>>>
> >>>>>>>>>>> DJ1: In the output of `--describe --verbose`, I would suggest
> >>> putting
> >>>>>>>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before `#MEMBERS`.
> >>>>>>>>>>> DJ2: Continuing on the above, I assume that we will print out
> N/A
> >>> for
> >>>>>>>>> the
> >>>>>>>>>>> fields not supported by classic groups. Is this correct?
> >>>>>>>>>>> DJ3: In the output of `--describe --members --verbose`, I
> wonder
> >>> if we
> >>>>>>>>>>> should use the following order and terms: CURRENT-EPOCH
> >>>>>>>>> CURRENT-ASSIGNMENT
> >>>>>>>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the GROUP-EPOCH
> >>>>>>> because
> >>>>>>>>> it
> >>>>>>>>>>> is already in the group description. The value `(0)` used for
> the
> >>>>>>>>>>> assignment is incorrect. Here I suppose that we will print out
> the
> >>>>>>> list
> >>>>>>>>> of
> >>>>>>>>>>> partitions in the form of `topicid-partitionid`.
> >>>>>>>>>>> DJ4: Continuing on the above, I wonder if we should also add
> the
> >>>>>>>>> protocol
> >>>>>>>>>>> used `classic` or `consumer`. For context, it is possible to
> have
> >>>>>>>>> `classic`
> >>>>>>>>>>> members and `consumer` members in a `consumer` group during an
> >>> online
> >>>>>>>>>>> upgrade from the classic protocol to the consumer protocol.
> Having
> >>>>>>> this
> >>>>>>>>>>> information may be handy for administrators. What do you think?
> >>>>>>>>>>> DJ5: In the output of `--describe --offsets --verbose`, I would
> >>>>>>> suggest
> >>>>>>>>>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`.
> >>>>>>>>>>>
> >>>>>>>>>>> It would be great if we could get this one in AK 4.0 as it
> >>> changes the
> >>>>>>>>>>> output of the command.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>> DJ
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <ya...@gmail.com>
> >>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi Sean / Andrew / Lianet,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks for all your review and suggestions.
> >>>>>>>>>>>>
> >>>>>>>>>>>> AS1, LM1, LM4: Change to add KIP-848 information when users
> give
> >>>>>>>>> —verbose
> >>>>>>>>>>>> option.
> >>>>>>>>>>>> —describe —verbose: shows group epoch and target assignment
> >>> epoch.
> >>>>>>>>>>>> —describe —members —verbose: shows above information, member
> >>> epoch,
> >>>>>>> and
> >>>>>>>>>>>> target assignment.
> >>>>>>>>>>>>
> >>>>>>>>>>>> AS2: Change to use MEMBER-EPOCH to align with KIP-848
> definition.
> >>>>>>>>>>>>
> >>>>>>>>>>>> LM2: For classic group, it doesn’t have epoch, so I use
> Optional
> >>>>>>>>> fields in
> >>>>>>>>>>>> ConsumerGroupDescription.
> >>>>>>>>>>>> For share group, it relies on KIP-848. It must have epoch, so
> I
> >>> use
> >>>>>>> int
> >>>>>>>>>>>> fields in ShareGroupDescription.
> >>>>>>>>>>>>
> >>>>>>>>>>>> LM3: Remove —state change. We can get group level information
> by
> >>>>>>>>> —describe
> >>>>>>>>>>>> —verbose.
> >>>>>>>>>>>>
> >>>>>>>>>>>> SQ1: Add LEADER-EPOCH when users give —describe —offsets
> >>> —verbose.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks.
> >>>>>>>>>>>> PoAn
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <li...@gmail.com>
> >>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hello Frank, thanks for the KIP! A few comments:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> LM1. I strongly agree with Andrew's suggestion of moving this
> >>> into a
> >>>>>>>>>>>>> --verbose option. I expect someone would only attempt to make
> >>> sense
> >>>>>>> of
> >>>>>>>>>>>> the
> >>>>>>>>>>>>> epochs while debugging an issue, not while trying to view the
> >>> group
> >>>>>>> or
> >>>>>>>>>>>>> member's info. (Also member-epoch makes more sense to me,
> even
> >>> more
> >>>>>>>>> if we
> >>>>>>>>>>>>> end up agreeing in a --verbose). Related to both issues, my
> >>> take is
> >>>>>>>>> that
> >>>>>>>>>>>>> whoever is close to the protocol will expect member-epoch,
> >>> whoever
> >>>>>>> is
> >>>>>>>>> not
> >>>>>>>>>>>>> will probably won't even need to see the epochs at all.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> LM2. Why are the epochs defined as Optional (don't we expect
> to
> >>>>>>> always
> >>>>>>>>>>>> have
> >>>>>>>>>>>>> them? with 0 initially, defined on the broker side for the
> group
> >>>>>>> ones,
> >>>>>>>>>>>> and
> >>>>>>>>>>>>> client side for the member)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> LM3. Why is the KIP including the –state option in the
> proposed
> >>>>>>>>> changes?
> >>>>>>>>>>>> (
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
> >>>>>>>>>>>>> ).
> >>>>>>>>>>>>> I get that the output would change in that example, but it’s
> not
> >>>>>>>>> because
> >>>>>>>>>>>> of
> >>>>>>>>>>>>> any change to the –state option itself. It's because of the
> >>> change
> >>>>>>>>>>>> proposed
> >>>>>>>>>>>>> to the –described (required with the --state), and the
> changes
> >>> to
> >>>>>>>>>>>>> --describe are already explained above (seems confusing, got
> me
> >>>>>>>>> looking
> >>>>>>>>>>>> for
> >>>>>>>>>>>>> a change to the state filter which seemed unrelated).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> LM4. Group epoch and target assignment epoch are conceptually
> >>> at the
> >>>>>>>>>>>> group
> >>>>>>>>>>>>> level. vs member epoch that lands at a member level. So
> wonder
> >>> if we
> >>>>>>>>>>>> should
> >>>>>>>>>>>>> show them accordingly (ex. using the --verbose option)
> >>>>>>>>>>>>> –describe –verbose => shows group epoch and target assignment
> >>> epoch
> >>>>>>>>>>>>> –describe –members –verbose => shows member epoch
> >>> (along
> >>>>>>>>>>>>> with group epoch and target assignment epoch)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Lianet
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield <
> >>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>>>>> Thanks for the KIP. I have a few comments.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> AS1: It seems to me that these new additions are most
> useful to
> >>>>>>>>> people
> >>>>>>>>>>>>>> trying to understand
> >>>>>>>>>>>>>> the progress of rebalancing in quite some detail. The
> >>> information
> >>>>>>> is
> >>>>>>>>>>>>>> really not understandable
> >>>>>>>>>>>>>> for most users who do not have deep knowledge of
> KIP-848/932.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> As a result, I suggest for kafka-share-groups.sh that you
> add a
> >>>>>>>>>>>> --members
> >>>>>>>>>>>>>> --verbose option
> >>>>>>>>>>>>>> and only include the new information in the output for that
> >>> option,
> >>>>>>>>>>>> rather
> >>>>>>>>>>>>>> than changing the
> >>>>>>>>>>>>>> non-verbose --members output.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I also make a similar suggestion for
> kafka-consumer-groups.sh
> >>>>>>>>> --members
> >>>>>>>>>>>>>> and only add the
> >>>>>>>>>>>>>> new information for the --verbose output.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> AS2: I strongly suggest that you use MEMBER-EPOCH instead of
> >>>>>>>>>>>>>> CONSUMER-EPOCH.
> >>>>>>>>>>>>>> I think it's more confusing not following the terminology of
> >>> the
> >>>>>>>>> KIPs.
> >>>>>>>>>>>> For
> >>>>>>>>>>>>>> one thing,
> >>>>>>>>>>>>>> we already have "member" in the admin client such as
> >>>>>>>>> MemberDescription.
> >>>>>>>>>>>>>> It's not a
> >>>>>>>>>>>>>> new term.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>> Andrew
> >>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>> From: PoAn Yang <pa...@apache.org>
> >>>>>>>>>>>>>> Sent: 25 October 2024 13:55
> >>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
> >>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend
> kafka-consumer-groups
> >>>>>>> command
> >>>>>>>>>>>> line
> >>>>>>>>>>>>>> tool to support new consumer group
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi Lucas,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks for the review!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 1) Yes, I add related change for kafka-share-groups.sh to
> the
> >>> KIP.
> >>>>>>>>> Could
> >>>>>>>>>>>>>> you take a look? Thanks for the suggestion.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use MEMBER-EPOCH
> >>> here,
> >>>>>>>>> users
> >>>>>>>>>>>> may
> >>>>>>>>>>>>>> confuse what is different between CONSUMER and MEMBER.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
> >>>>>>>>>>>>>>> Hi Frank,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> thanks for the KIP!
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 1) For consistency, should we do the same for
> >>>>>>>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription, etc. ? Even
> if
> >>> we do
> >>>>>>>>> not
> >>>>>>>>>>>>>>> implement it right now if the share group implementation
> may
> >>> still
> >>>>>>>>> be
> >>>>>>>>>>>>>>> incomplete, it may make sense to include it in the KIP.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH? That would
> >>> seem
> >>>>>>>>> more
> >>>>>>>>>>>>>>> consistent.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>> Lucas
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <
> >>> yangpoan@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I would like to kick off the discussion of KIP-1099. This
> KIP
> >>>>>>>>> enhances
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> kafka-consumer-groups tools to include state which is
> >>> introduced
> >>>>>>> by
> >>>>>>>>>>>>>> KIP-848.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups command line tool
> to
> >>>>>>> support
> >>>>>>>>>>>> new
> >>>>>>>>>>>>>>>> consumer group - Apache Kafka - Apache Software Foundation
> >>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> cwiki.apache.org
> >>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> [image: favicon.ico]
> >>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thank you,
> >>>>>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >>
>
>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by PoAn Yang <ya...@gmail.com>.
Hi Lianet / Jeff,
Thanks for review and suggestions.
LM4: At group level, the protocol field is from ModernGroup#protocolType. It’s a string.
The value is defined in different places like ShareGroup#PROTOCOL_TYPE and ConsumerProtocol#PROTOCOL_TYPE.
In KIP-1043 <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=305171038#KIP1043:Administrationofgroups-kafka-groups.sh>, the kafka-groups.sh also shows protocol like “classic” / “consumer” / “share”.
I agree it’s better to use a type like GroupProtocol enum, but I would like to align the format with kafka-groups.sh.
How about we use another Jira to add “SHARE” to GroupProtocol and change return type of ModernGroup#protocolType?
LM5 / JK1: Sorry, I misunderstood LM3, so I deleted —state in the title.
You’re correct. I would like to use `—describe —state —verbose` to show group level information.
For `—describe —verbose`, it will show output as same as `—describe —offsets —verbose`,
because `—describe` also shows output as same as `—describe —offsets`.
JK2: Yes, I only use consumer group in the sample output, but the classic group will use the same format as well.
Thanks,
PoAn
> On Nov 21, 2024, at 8:28 AM, Jeff Kim <je...@apache.org> wrote:
>
> Hi PoAn,
>
> Thanks for the KIP. I have some questions:
>
> JK1: (Realized this is similar to LM5) On `--describe --verbose` proposed changes - doesn't `--describe $group` default to printing the offsets? Perhaps you're referring to the `--state` argument? Also, would that mean the default `--describe $group --verbose` command would not print the added field to `--offsets verbose` (leader-epoch) or would it?
>
> JK2: Looking at ConsumerGroupCommand.java, the existing "ASSIGNMENT" column under `--members --verbose` does group the topic partitions by topic but does not prefix the grouping with the topic name like you're proposal: "my_topic:0,1;new_topic:0,1". Should we do apply the same format for the classic group as well?
>
>
>
> On 2024/11/20 21:32:41 "Lianet M." wrote:
>> Hello PoAn, just of couple more minor comments:
>>
>> LM4. Regarding the new “protocol” field added to MemberDescription. Should
>> we consider reusing the existing GroupProtocol enum instead of String?
>> (It’s the one we use from the consumer side to refer to the protocol in
>> use, just missing Share I notice).
>>
>> LM5. Regarding the change to the output for kafka-consumer-groups, the
>> command shown does not include the —state option, but the output shows
>> state info (state, #members, epochs). I would guess that we want to modify
>> the output only when we describe a group with the —state —verbose option,
>> is my understanding right? If my understanding is right we’re just missing
>> adding the —state in the example, and the KIP does not introduce any
>> changes to the —describe —verbose option. (If not, it would mean a bigger
>> change to the output of —describe —verbose which I expect is not the
>> intention?)
>>
>> Thanks!
>>
>> Lianet
>>
>>
>> On Wed, Nov 20, 2024 at 2:12 AM PoAn Yang <ya...@gmail.com> wrote:
>>
>>> Hi Chia-Ping,
>>>
>>> Thanks for the review and suggestions.
>>>
>>> CI0: Thanks for the reminder. Update validVersions in
>>> ConsumerGroupDescribeRequest to 0-1.
>>>
>>> CI1: Yes, I use `ConsumerGroupMember#useClassicProtocol` to check whether
>>> a member in consumer group uses “classic” or “consumer” protocol.
>>>
>>> CI2: Yes, a member in share group always uses “share” protocol.
>>>
>>> CI3: Add a table to show meaning of “classic”, “consumer”, and “share”
>>> protocol.
>>>
>>> BTW, the vote thread is in
>>> https://lists.apache.org/thread/rb25tf75tzf4c7jqqldlo5jh9w8chsq6.
>>>
>>> Thanks,
>>> PoAn
>>>
>>>> On Nov 20, 2024, at 11:46 AM, Chia-Ping Tsai <ch...@apache.org>
>>> wrote:
>>>>
>>>> hi PoAn
>>>>
>>>> CI0: We have to bump the version of ConsumerGroupDescribeRequest as
>>> well, so server can distinguish the new and old behavior.
>>>>
>>>> CI1: the type of new filed is string, so I guess you plan to use
>>> `ConsumerGroupMember#useClassicProtocol` [0] flag to return either
>>> "classic" or "consumer", right?
>>>>
>>>> CI2: `MemberDescription` is used by `ShareGroupDescription` too, so the
>>> new filed protocol in shared group is always "shared", right?
>>>>
>>>> CI3: Could you consider adding a table to show the value of the protocol
>>> field in each case? Andrew has a beautiful table in KIP-1043 that lists all
>>> possible protocol names.
>>>>
>>>> [0]
>>> https://github.com/apache/kafka/blob/441a6d0b790f4a17b454caeea7588a6b90fbd9db/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/consumer/ConsumerGroupMember.java#L454
>>>
>>>>
>>>> Best,
>>>> Chia-Ping
>>>>
>>>> On 2024/11/19 15:45:16 PoAn Yang wrote:
>>>>> Hi David,
>>>>>
>>>>> Thanks for the reminder.
>>>>>
>>>>> DJ4: Add related content to the KIP.
>>>>> Bump validVersions in ConsumerGroupDescribeResponse to “0-1”.
>>>>> Add a new field “Protocol” to ConsumerGroupDescribeResponse.
>>>>>
>>>>> Thanks,
>>>>> PoAn
>>>>>
>>>>>> On Nov 19, 2024, at 3:57 PM, David Jacot <dj...@confluent.io.INVALID>
>>> wrote:
>>>>>>
>>>>>> Hi PoAn,
>>>>>>
>>>>>> DJ4: Could you please also include the required changes to the
>>>>>> ConsumerGroupDescribed API in the public interface section? We
>>> basically
>>>>>> need to bump the version, add the new field, etc.
>>>>>>
>>>>>> Thanks,
>>>>>> David
>>>>>>
>>>>>> On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang <ya...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi David,
>>>>>>>
>>>>>>> DJ4: Add a new field protocol to MemberDescription,
>>>>>>> so the command line tool can show protocol information when users
>>> describe
>>>>>>> members.
>>>>>>>
>>>>>>> If there is no further suggestion, I will start a vote thread today.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> PoAn
>>>>>>>
>>>>>>>> On Nov 15, 2024, at 11:50 PM, David Jacot
>>> <dj...@confluent.io.INVALID>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> DJ2: Using "-" sounds good to me.
>>>>>>>>
>>>>>>>> DJ3: That seems reasonable to me.
>>>>>>>>
>>>>>>>> DJ4: Why not add it right now? I don't want to change the output of
>>> the
>>>>>>>> tool too many times.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> David
>>>>>>>>
>>>>>>>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang <ya...@gmail.com>
>>> wrote:
>>>>>>>>
>>>>>>>>> Hi David / Andrew,
>>>>>>>>>
>>>>>>>>> Thanks for review. Thanks Andrew for picking up
>>> kafka-share-groups.sh
>>>>>>>>> implementation.
>>>>>>>>> I will handle kafka-consumer-groups.sh.
>>>>>>>>>
>>>>>>>>> DJ3: After discussing with @Chia-Ping Tsai, we think that using new
>>>>>>> format
>>>>>>>>> is more clear.
>>>>>>>>>
>>>>>>>>> The new format will be like
>>>>>>>>>
>>>>>>>
>>> <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2>
>>>>>>>>>
>>>>>>>>> Using colon(:) to concat topic name and partition IDs.
>>>>>>>>> Using comma(,) to concat partition IDs within same topic name.
>>>>>>>>> Using semicolon(;) to concat topic strings.
>>>>>>>>>
>>>>>>>>> AS3: Fix it with kafka-share-groups.sh. Thanks.
>>>>>>>>>
>>>>>>>>> If there is no further suggestion, I will start a vote thread next
>>>>>>> Monday.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> PoAn
>>>>>>>>>
>>>>>>>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield <
>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi PoAn,
>>>>>>>>>> DJ2: I was just going to comment that "-" would be a more
>>> appropriate
>>>>>>>>> missing value, but
>>>>>>>>>> you got there first.
>>>>>>>>>>
>>>>>>>>>> AS3: The examples for kafka-share-groups.sh include
>>>>>>>>> kafka-consumer-groups.sh in the
>>>>>>>>>> command line.
>>>>>>>>>>
>>>>>>>>>> If this is accepted in time, I'm happy to pick up the
>>> implementation of
>>>>>>>>> the share groups
>>>>>>>>>> part of this if it helps.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Andrew
>>>>>>>>>>
>>>>>>>>>> ________________________________________
>>>>>>>>>> From: Frank Yang <ya...@gmail.com>
>>>>>>>>>> Sent: 14 November 2024 10:48
>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups
>>> command
>>>>>>>>> line tool to support new consumer group
>>>>>>>>>>
>>>>>>>>>> Hi David,
>>>>>>>>>>
>>>>>>>>>> Thanks for the review and suggestion! I would like to get this in
>>> AK
>>>>>>> 4.0
>>>>>>>>> as well. I will do my best.
>>>>>>>>>>
>>>>>>>>>> DJ1: Update KIP to put GROUP-EPOCH and TARGET-ASSIGNMENT-EPOCH
>>> before
>>>>>>>>> #MEMBERS.
>>>>>>>>>>
>>>>>>>>>> DJ2: I prefer to follow current missing column value “-“.
>>> (reference <
>>>>>>>>>
>>>>>>>
>>> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92
>>>>>>>>>> )
>>>>>>>>>>
>>>>>>>>>> DJ3: Update KIP to use CURRENT-EPOCH CURRENT-ASSIGNMENT
>>> TARGET-EPOCH
>>>>>>>>> TARGET-ASSIGNMENT.
>>>>>>>>>> Remove GROUP-EPOCH.
>>>>>>>>>>
>>>>>>>>>> For assignment value, it follows current output (reference <
>>>>>>>>>
>>>>>>>
>>> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418
>>>>>>>> ).
>>>>>>>>> I think the form of `topicid-partitionid` is more clear.
>>>>>>>>>> If we would like to use this form, I will update both output in
>>>>>>>>> kafka-consumer-groups.sh and kafka-share-groups.sh.
>>>>>>>>>>
>>>>>>>>>> DJ4: It looks like DescribeGroupsResponseData only has protocol
>>> type at
>>>>>>>>> group level.
>>>>>>>>>> Both DescribeGroupsResponseData and
>>> ConsumerGroupDescribeResponseData
>>>>>>>>> don’t have protocol at member level.
>>>>>>>>>> Could we use a followup to add it?
>>>>>>>>>>
>>>>>>>>>> DJ5: Update KIP to put LEADER-EPOCH before CURRENT-OFFSET.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> PoAn
>>>>>>>>>>
>>>>>>>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot
>>> <djacot@confluent.io.INVALID
>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>
>>>>>>>>>>> Thanks for the KIP! I have a few minor comments/suggestions:
>>>>>>>>>>>
>>>>>>>>>>> DJ1: In the output of `--describe --verbose`, I would suggest
>>> putting
>>>>>>>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before `#MEMBERS`.
>>>>>>>>>>> DJ2: Continuing on the above, I assume that we will print out N/A
>>> for
>>>>>>>>> the
>>>>>>>>>>> fields not supported by classic groups. Is this correct?
>>>>>>>>>>> DJ3: In the output of `--describe --members --verbose`, I wonder
>>> if we
>>>>>>>>>>> should use the following order and terms: CURRENT-EPOCH
>>>>>>>>> CURRENT-ASSIGNMENT
>>>>>>>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the GROUP-EPOCH
>>>>>>> because
>>>>>>>>> it
>>>>>>>>>>> is already in the group description. The value `(0)` used for the
>>>>>>>>>>> assignment is incorrect. Here I suppose that we will print out the
>>>>>>> list
>>>>>>>>> of
>>>>>>>>>>> partitions in the form of `topicid-partitionid`.
>>>>>>>>>>> DJ4: Continuing on the above, I wonder if we should also add the
>>>>>>>>> protocol
>>>>>>>>>>> used `classic` or `consumer`. For context, it is possible to have
>>>>>>>>> `classic`
>>>>>>>>>>> members and `consumer` members in a `consumer` group during an
>>> online
>>>>>>>>>>> upgrade from the classic protocol to the consumer protocol. Having
>>>>>>> this
>>>>>>>>>>> information may be handy for administrators. What do you think?
>>>>>>>>>>> DJ5: In the output of `--describe --offsets --verbose`, I would
>>>>>>> suggest
>>>>>>>>>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`.
>>>>>>>>>>>
>>>>>>>>>>> It would be great if we could get this one in AK 4.0 as it
>>> changes the
>>>>>>>>>>> output of the command.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> DJ
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <ya...@gmail.com>
>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Sean / Andrew / Lianet,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for all your review and suggestions.
>>>>>>>>>>>>
>>>>>>>>>>>> AS1, LM1, LM4: Change to add KIP-848 information when users give
>>>>>>>>> —verbose
>>>>>>>>>>>> option.
>>>>>>>>>>>> —describe —verbose: shows group epoch and target assignment
>>> epoch.
>>>>>>>>>>>> —describe —members —verbose: shows above information, member
>>> epoch,
>>>>>>> and
>>>>>>>>>>>> target assignment.
>>>>>>>>>>>>
>>>>>>>>>>>> AS2: Change to use MEMBER-EPOCH to align with KIP-848 definition.
>>>>>>>>>>>>
>>>>>>>>>>>> LM2: For classic group, it doesn’t have epoch, so I use Optional
>>>>>>>>> fields in
>>>>>>>>>>>> ConsumerGroupDescription.
>>>>>>>>>>>> For share group, it relies on KIP-848. It must have epoch, so I
>>> use
>>>>>>> int
>>>>>>>>>>>> fields in ShareGroupDescription.
>>>>>>>>>>>>
>>>>>>>>>>>> LM3: Remove —state change. We can get group level information by
>>>>>>>>> —describe
>>>>>>>>>>>> —verbose.
>>>>>>>>>>>>
>>>>>>>>>>>> SQ1: Add LEADER-EPOCH when users give —describe —offsets
>>> —verbose.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks.
>>>>>>>>>>>> PoAn
>>>>>>>>>>>>
>>>>>>>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <li...@gmail.com>
>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hello Frank, thanks for the KIP! A few comments:
>>>>>>>>>>>>>
>>>>>>>>>>>>> LM1. I strongly agree with Andrew's suggestion of moving this
>>> into a
>>>>>>>>>>>>> --verbose option. I expect someone would only attempt to make
>>> sense
>>>>>>> of
>>>>>>>>>>>> the
>>>>>>>>>>>>> epochs while debugging an issue, not while trying to view the
>>> group
>>>>>>> or
>>>>>>>>>>>>> member's info. (Also member-epoch makes more sense to me, even
>>> more
>>>>>>>>> if we
>>>>>>>>>>>>> end up agreeing in a --verbose). Related to both issues, my
>>> take is
>>>>>>>>> that
>>>>>>>>>>>>> whoever is close to the protocol will expect member-epoch,
>>> whoever
>>>>>>> is
>>>>>>>>> not
>>>>>>>>>>>>> will probably won't even need to see the epochs at all.
>>>>>>>>>>>>>
>>>>>>>>>>>>> LM2. Why are the epochs defined as Optional (don't we expect to
>>>>>>> always
>>>>>>>>>>>> have
>>>>>>>>>>>>> them? with 0 initially, defined on the broker side for the group
>>>>>>> ones,
>>>>>>>>>>>> and
>>>>>>>>>>>>> client side for the member)
>>>>>>>>>>>>>
>>>>>>>>>>>>> LM3. Why is the KIP including the –state option in the proposed
>>>>>>>>> changes?
>>>>>>>>>>>> (
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
>>>>>>>>>>>>> ).
>>>>>>>>>>>>> I get that the output would change in that example, but it’s not
>>>>>>>>> because
>>>>>>>>>>>> of
>>>>>>>>>>>>> any change to the –state option itself. It's because of the
>>> change
>>>>>>>>>>>> proposed
>>>>>>>>>>>>> to the –described (required with the --state), and the changes
>>> to
>>>>>>>>>>>>> --describe are already explained above (seems confusing, got me
>>>>>>>>> looking
>>>>>>>>>>>> for
>>>>>>>>>>>>> a change to the state filter which seemed unrelated).
>>>>>>>>>>>>>
>>>>>>>>>>>>> LM4. Group epoch and target assignment epoch are conceptually
>>> at the
>>>>>>>>>>>> group
>>>>>>>>>>>>> level. vs member epoch that lands at a member level. So wonder
>>> if we
>>>>>>>>>>>> should
>>>>>>>>>>>>> show them accordingly (ex. using the --verbose option)
>>>>>>>>>>>>> –describe –verbose => shows group epoch and target assignment
>>> epoch
>>>>>>>>>>>>> –describe –members –verbose => shows member epoch
>>> (along
>>>>>>>>>>>>> with group epoch and target assignment epoch)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>
>>>>>>>>>>>>> Lianet
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield <
>>>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>>>>> Thanks for the KIP. I have a few comments.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> AS1: It seems to me that these new additions are most useful to
>>>>>>>>> people
>>>>>>>>>>>>>> trying to understand
>>>>>>>>>>>>>> the progress of rebalancing in quite some detail. The
>>> information
>>>>>>> is
>>>>>>>>>>>>>> really not understandable
>>>>>>>>>>>>>> for most users who do not have deep knowledge of KIP-848/932.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> As a result, I suggest for kafka-share-groups.sh that you add a
>>>>>>>>>>>> --members
>>>>>>>>>>>>>> --verbose option
>>>>>>>>>>>>>> and only include the new information in the output for that
>>> option,
>>>>>>>>>>>> rather
>>>>>>>>>>>>>> than changing the
>>>>>>>>>>>>>> non-verbose --members output.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I also make a similar suggestion for kafka-consumer-groups.sh
>>>>>>>>> --members
>>>>>>>>>>>>>> and only add the
>>>>>>>>>>>>>> new information for the --verbose output.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> AS2: I strongly suggest that you use MEMBER-EPOCH instead of
>>>>>>>>>>>>>> CONSUMER-EPOCH.
>>>>>>>>>>>>>> I think it's more confusing not following the terminology of
>>> the
>>>>>>>>> KIPs.
>>>>>>>>>>>> For
>>>>>>>>>>>>>> one thing,
>>>>>>>>>>>>>> we already have "member" in the admin client such as
>>>>>>>>> MemberDescription.
>>>>>>>>>>>>>> It's not a
>>>>>>>>>>>>>> new term.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Andrew
>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>> From: PoAn Yang <pa...@apache.org>
>>>>>>>>>>>>>> Sent: 25 October 2024 13:55
>>>>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups
>>>>>>> command
>>>>>>>>>>>> line
>>>>>>>>>>>>>> tool to support new consumer group
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Lucas,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for the review!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1) Yes, I add related change for kafka-share-groups.sh to the
>>> KIP.
>>>>>>>>> Could
>>>>>>>>>>>>>> you take a look? Thanks for the suggestion.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use MEMBER-EPOCH
>>> here,
>>>>>>>>> users
>>>>>>>>>>>> may
>>>>>>>>>>>>>> confuse what is different between CONSUMER and MEMBER.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
>>>>>>>>>>>>>>> Hi Frank,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> thanks for the KIP!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 1) For consistency, should we do the same for
>>>>>>>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription, etc. ? Even if
>>> we do
>>>>>>>>> not
>>>>>>>>>>>>>>> implement it right now if the share group implementation may
>>> still
>>>>>>>>> be
>>>>>>>>>>>>>>> incomplete, it may make sense to include it in the KIP.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH? That would
>>> seem
>>>>>>>>> more
>>>>>>>>>>>>>>> consistent.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>> Lucas
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <
>>> yangpoan@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I would like to kick off the discussion of KIP-1099. This KIP
>>>>>>>>> enhances
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> kafka-consumer-groups tools to include state which is
>>> introduced
>>>>>>> by
>>>>>>>>>>>>>> KIP-848.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups command line tool to
>>>>>>> support
>>>>>>>>>>>> new
>>>>>>>>>>>>>>>> consumer group - Apache Kafka - Apache Software Foundation
>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> cwiki.apache.org
>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [image: favicon.ico]
>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>>
>>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by Jeff Kim <je...@apache.org>.
Hi PoAn,
Thanks for the KIP. I have some questions:
JK1: (Realized this is similar to LM5) On `--describe --verbose` proposed changes - doesn't `--describe $group` default to printing the offsets? Perhaps you're referring to the `--state` argument? Also, would that mean the default `--describe $group --verbose` command would not print the added field to `--offsets verbose` (leader-epoch) or would it?
JK2: Looking at ConsumerGroupCommand.java, the existing "ASSIGNMENT" column under `--members --verbose` does group the topic partitions by topic but does not prefix the grouping with the topic name like you're proposal: "my_topic:0,1;new_topic:0,1". Should we do apply the same format for the classic group as well?
On 2024/11/20 21:32:41 "Lianet M." wrote:
> Hello PoAn, just of couple more minor comments:
>
> LM4. Regarding the new “protocol” field added to MemberDescription. Should
> we consider reusing the existing GroupProtocol enum instead of String?
> (It’s the one we use from the consumer side to refer to the protocol in
> use, just missing Share I notice).
>
> LM5. Regarding the change to the output for kafka-consumer-groups, the
> command shown does not include the —state option, but the output shows
> state info (state, #members, epochs). I would guess that we want to modify
> the output only when we describe a group with the —state —verbose option,
> is my understanding right? If my understanding is right we’re just missing
> adding the —state in the example, and the KIP does not introduce any
> changes to the —describe —verbose option. (If not, it would mean a bigger
> change to the output of —describe —verbose which I expect is not the
> intention?)
>
> Thanks!
>
> Lianet
>
>
> On Wed, Nov 20, 2024 at 2:12 AM PoAn Yang <ya...@gmail.com> wrote:
>
> > Hi Chia-Ping,
> >
> > Thanks for the review and suggestions.
> >
> > CI0: Thanks for the reminder. Update validVersions in
> > ConsumerGroupDescribeRequest to 0-1.
> >
> > CI1: Yes, I use `ConsumerGroupMember#useClassicProtocol` to check whether
> > a member in consumer group uses “classic” or “consumer” protocol.
> >
> > CI2: Yes, a member in share group always uses “share” protocol.
> >
> > CI3: Add a table to show meaning of “classic”, “consumer”, and “share”
> > protocol.
> >
> > BTW, the vote thread is in
> > https://lists.apache.org/thread/rb25tf75tzf4c7jqqldlo5jh9w8chsq6.
> >
> > Thanks,
> > PoAn
> >
> > > On Nov 20, 2024, at 11:46 AM, Chia-Ping Tsai <ch...@apache.org>
> > wrote:
> > >
> > > hi PoAn
> > >
> > > CI0: We have to bump the version of ConsumerGroupDescribeRequest as
> > well, so server can distinguish the new and old behavior.
> > >
> > > CI1: the type of new filed is string, so I guess you plan to use
> > `ConsumerGroupMember#useClassicProtocol` [0] flag to return either
> > "classic" or "consumer", right?
> > >
> > > CI2: `MemberDescription` is used by `ShareGroupDescription` too, so the
> > new filed protocol in shared group is always "shared", right?
> > >
> > > CI3: Could you consider adding a table to show the value of the protocol
> > field in each case? Andrew has a beautiful table in KIP-1043 that lists all
> > possible protocol names.
> > >
> > > [0]
> > https://github.com/apache/kafka/blob/441a6d0b790f4a17b454caeea7588a6b90fbd9db/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/consumer/ConsumerGroupMember.java#L454
> >
> > >
> > > Best,
> > > Chia-Ping
> > >
> > > On 2024/11/19 15:45:16 PoAn Yang wrote:
> > >> Hi David,
> > >>
> > >> Thanks for the reminder.
> > >>
> > >> DJ4: Add related content to the KIP.
> > >> Bump validVersions in ConsumerGroupDescribeResponse to “0-1”.
> > >> Add a new field “Protocol” to ConsumerGroupDescribeResponse.
> > >>
> > >> Thanks,
> > >> PoAn
> > >>
> > >>> On Nov 19, 2024, at 3:57 PM, David Jacot <dj...@confluent.io.INVALID>
> > wrote:
> > >>>
> > >>> Hi PoAn,
> > >>>
> > >>> DJ4: Could you please also include the required changes to the
> > >>> ConsumerGroupDescribed API in the public interface section? We
> > basically
> > >>> need to bump the version, add the new field, etc.
> > >>>
> > >>> Thanks,
> > >>> David
> > >>>
> > >>> On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang <ya...@gmail.com> wrote:
> > >>>
> > >>>> Hi David,
> > >>>>
> > >>>> DJ4: Add a new field protocol to MemberDescription,
> > >>>> so the command line tool can show protocol information when users
> > describe
> > >>>> members.
> > >>>>
> > >>>> If there is no further suggestion, I will start a vote thread today.
> > >>>>
> > >>>> Thanks,
> > >>>> PoAn
> > >>>>
> > >>>>> On Nov 15, 2024, at 11:50 PM, David Jacot
> > <dj...@confluent.io.INVALID>
> > >>>> wrote:
> > >>>>>
> > >>>>> Hi,
> > >>>>>
> > >>>>> DJ2: Using "-" sounds good to me.
> > >>>>>
> > >>>>> DJ3: That seems reasonable to me.
> > >>>>>
> > >>>>> DJ4: Why not add it right now? I don't want to change the output of
> > the
> > >>>>> tool too many times.
> > >>>>>
> > >>>>> Best,
> > >>>>> David
> > >>>>>
> > >>>>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang <ya...@gmail.com>
> > wrote:
> > >>>>>
> > >>>>>> Hi David / Andrew,
> > >>>>>>
> > >>>>>> Thanks for review. Thanks Andrew for picking up
> > kafka-share-groups.sh
> > >>>>>> implementation.
> > >>>>>> I will handle kafka-consumer-groups.sh.
> > >>>>>>
> > >>>>>> DJ3: After discussing with @Chia-Ping Tsai, we think that using new
> > >>>> format
> > >>>>>> is more clear.
> > >>>>>>
> > >>>>>> The new format will be like
> > >>>>>>
> > >>>>
> > <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2>
> > >>>>>>
> > >>>>>> Using colon(:) to concat topic name and partition IDs.
> > >>>>>> Using comma(,) to concat partition IDs within same topic name.
> > >>>>>> Using semicolon(;) to concat topic strings.
> > >>>>>>
> > >>>>>> AS3: Fix it with kafka-share-groups.sh. Thanks.
> > >>>>>>
> > >>>>>> If there is no further suggestion, I will start a vote thread next
> > >>>> Monday.
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> PoAn
> > >>>>>>
> > >>>>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield <
> > >>>>>> andrew_schofield_jira@outlook.com> wrote:
> > >>>>>>>
> > >>>>>>> Hi PoAn,
> > >>>>>>> DJ2: I was just going to comment that "-" would be a more
> > appropriate
> > >>>>>> missing value, but
> > >>>>>>> you got there first.
> > >>>>>>>
> > >>>>>>> AS3: The examples for kafka-share-groups.sh include
> > >>>>>> kafka-consumer-groups.sh in the
> > >>>>>>> command line.
> > >>>>>>>
> > >>>>>>> If this is accepted in time, I'm happy to pick up the
> > implementation of
> > >>>>>> the share groups
> > >>>>>>> part of this if it helps.
> > >>>>>>>
> > >>>>>>> Thanks,
> > >>>>>>> Andrew
> > >>>>>>>
> > >>>>>>> ________________________________________
> > >>>>>>> From: Frank Yang <ya...@gmail.com>
> > >>>>>>> Sent: 14 November 2024 10:48
> > >>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
> > >>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups
> > command
> > >>>>>> line tool to support new consumer group
> > >>>>>>>
> > >>>>>>> Hi David,
> > >>>>>>>
> > >>>>>>> Thanks for the review and suggestion! I would like to get this in
> > AK
> > >>>> 4.0
> > >>>>>> as well. I will do my best.
> > >>>>>>>
> > >>>>>>> DJ1: Update KIP to put GROUP-EPOCH and TARGET-ASSIGNMENT-EPOCH
> > before
> > >>>>>> #MEMBERS.
> > >>>>>>>
> > >>>>>>> DJ2: I prefer to follow current missing column value “-“.
> > (reference <
> > >>>>>>
> > >>>>
> > https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92
> > >>>>>>> )
> > >>>>>>>
> > >>>>>>> DJ3: Update KIP to use CURRENT-EPOCH CURRENT-ASSIGNMENT
> > TARGET-EPOCH
> > >>>>>> TARGET-ASSIGNMENT.
> > >>>>>>> Remove GROUP-EPOCH.
> > >>>>>>>
> > >>>>>>> For assignment value, it follows current output (reference <
> > >>>>>>
> > >>>>
> > https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418
> > >>>>> ).
> > >>>>>> I think the form of `topicid-partitionid` is more clear.
> > >>>>>>> If we would like to use this form, I will update both output in
> > >>>>>> kafka-consumer-groups.sh and kafka-share-groups.sh.
> > >>>>>>>
> > >>>>>>> DJ4: It looks like DescribeGroupsResponseData only has protocol
> > type at
> > >>>>>> group level.
> > >>>>>>> Both DescribeGroupsResponseData and
> > ConsumerGroupDescribeResponseData
> > >>>>>> don’t have protocol at member level.
> > >>>>>>> Could we use a followup to add it?
> > >>>>>>>
> > >>>>>>> DJ5: Update KIP to put LEADER-EPOCH before CURRENT-OFFSET.
> > >>>>>>>
> > >>>>>>> Thanks,
> > >>>>>>> PoAn
> > >>>>>>>
> > >>>>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot
> > <djacot@confluent.io.INVALID
> > >>>>>
> > >>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> Hi PoAn,
> > >>>>>>>>
> > >>>>>>>> Thanks for the KIP! I have a few minor comments/suggestions:
> > >>>>>>>>
> > >>>>>>>> DJ1: In the output of `--describe --verbose`, I would suggest
> > putting
> > >>>>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before `#MEMBERS`.
> > >>>>>>>> DJ2: Continuing on the above, I assume that we will print out N/A
> > for
> > >>>>>> the
> > >>>>>>>> fields not supported by classic groups. Is this correct?
> > >>>>>>>> DJ3: In the output of `--describe --members --verbose`, I wonder
> > if we
> > >>>>>>>> should use the following order and terms: CURRENT-EPOCH
> > >>>>>> CURRENT-ASSIGNMENT
> > >>>>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the GROUP-EPOCH
> > >>>> because
> > >>>>>> it
> > >>>>>>>> is already in the group description. The value `(0)` used for the
> > >>>>>>>> assignment is incorrect. Here I suppose that we will print out the
> > >>>> list
> > >>>>>> of
> > >>>>>>>> partitions in the form of `topicid-partitionid`.
> > >>>>>>>> DJ4: Continuing on the above, I wonder if we should also add the
> > >>>>>> protocol
> > >>>>>>>> used `classic` or `consumer`. For context, it is possible to have
> > >>>>>> `classic`
> > >>>>>>>> members and `consumer` members in a `consumer` group during an
> > online
> > >>>>>>>> upgrade from the classic protocol to the consumer protocol. Having
> > >>>> this
> > >>>>>>>> information may be handy for administrators. What do you think?
> > >>>>>>>> DJ5: In the output of `--describe --offsets --verbose`, I would
> > >>>> suggest
> > >>>>>>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`.
> > >>>>>>>>
> > >>>>>>>> It would be great if we could get this one in AK 4.0 as it
> > changes the
> > >>>>>>>> output of the command.
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>> DJ
> > >>>>>>>>
> > >>>>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <ya...@gmail.com>
> > wrote:
> > >>>>>>>>
> > >>>>>>>>> Hi Sean / Andrew / Lianet,
> > >>>>>>>>>
> > >>>>>>>>> Thanks for all your review and suggestions.
> > >>>>>>>>>
> > >>>>>>>>> AS1, LM1, LM4: Change to add KIP-848 information when users give
> > >>>>>> —verbose
> > >>>>>>>>> option.
> > >>>>>>>>> —describe —verbose: shows group epoch and target assignment
> > epoch.
> > >>>>>>>>> —describe —members —verbose: shows above information, member
> > epoch,
> > >>>> and
> > >>>>>>>>> target assignment.
> > >>>>>>>>>
> > >>>>>>>>> AS2: Change to use MEMBER-EPOCH to align with KIP-848 definition.
> > >>>>>>>>>
> > >>>>>>>>> LM2: For classic group, it doesn’t have epoch, so I use Optional
> > >>>>>> fields in
> > >>>>>>>>> ConsumerGroupDescription.
> > >>>>>>>>> For share group, it relies on KIP-848. It must have epoch, so I
> > use
> > >>>> int
> > >>>>>>>>> fields in ShareGroupDescription.
> > >>>>>>>>>
> > >>>>>>>>> LM3: Remove —state change. We can get group level information by
> > >>>>>> —describe
> > >>>>>>>>> —verbose.
> > >>>>>>>>>
> > >>>>>>>>> SQ1: Add LEADER-EPOCH when users give —describe —offsets
> > —verbose.
> > >>>>>>>>>
> > >>>>>>>>> Thanks.
> > >>>>>>>>> PoAn
> > >>>>>>>>>
> > >>>>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <li...@gmail.com>
> > wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> Hello Frank, thanks for the KIP! A few comments:
> > >>>>>>>>>>
> > >>>>>>>>>> LM1. I strongly agree with Andrew's suggestion of moving this
> > into a
> > >>>>>>>>>> --verbose option. I expect someone would only attempt to make
> > sense
> > >>>> of
> > >>>>>>>>> the
> > >>>>>>>>>> epochs while debugging an issue, not while trying to view the
> > group
> > >>>> or
> > >>>>>>>>>> member's info. (Also member-epoch makes more sense to me, even
> > more
> > >>>>>> if we
> > >>>>>>>>>> end up agreeing in a --verbose). Related to both issues, my
> > take is
> > >>>>>> that
> > >>>>>>>>>> whoever is close to the protocol will expect member-epoch,
> > whoever
> > >>>> is
> > >>>>>> not
> > >>>>>>>>>> will probably won't even need to see the epochs at all.
> > >>>>>>>>>>
> > >>>>>>>>>> LM2. Why are the epochs defined as Optional (don't we expect to
> > >>>> always
> > >>>>>>>>> have
> > >>>>>>>>>> them? with 0 initially, defined on the broker side for the group
> > >>>> ones,
> > >>>>>>>>> and
> > >>>>>>>>>> client side for the member)
> > >>>>>>>>>>
> > >>>>>>>>>> LM3. Why is the KIP including the –state option in the proposed
> > >>>>>> changes?
> > >>>>>>>>> (
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>>
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
> > >>>>>>>>>> ).
> > >>>>>>>>>> I get that the output would change in that example, but it’s not
> > >>>>>> because
> > >>>>>>>>> of
> > >>>>>>>>>> any change to the –state option itself. It's because of the
> > change
> > >>>>>>>>> proposed
> > >>>>>>>>>> to the –described (required with the --state), and the changes
> > to
> > >>>>>>>>>> --describe are already explained above (seems confusing, got me
> > >>>>>> looking
> > >>>>>>>>> for
> > >>>>>>>>>> a change to the state filter which seemed unrelated).
> > >>>>>>>>>>
> > >>>>>>>>>> LM4. Group epoch and target assignment epoch are conceptually
> > at the
> > >>>>>>>>> group
> > >>>>>>>>>> level. vs member epoch that lands at a member level. So wonder
> > if we
> > >>>>>>>>> should
> > >>>>>>>>>> show them accordingly (ex. using the --verbose option)
> > >>>>>>>>>> –describe –verbose => shows group epoch and target assignment
> > epoch
> > >>>>>>>>>> –describe –members –verbose => shows member epoch
> > (along
> > >>>>>>>>>> with group epoch and target assignment epoch)
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks!
> > >>>>>>>>>>
> > >>>>>>>>>> Lianet
> > >>>>>>>>>>
> > >>>>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield <
> > >>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Hi PoAn,
> > >>>>>>>>>>> Thanks for the KIP. I have a few comments.
> > >>>>>>>>>>>
> > >>>>>>>>>>> AS1: It seems to me that these new additions are most useful to
> > >>>>>> people
> > >>>>>>>>>>> trying to understand
> > >>>>>>>>>>> the progress of rebalancing in quite some detail. The
> > information
> > >>>> is
> > >>>>>>>>>>> really not understandable
> > >>>>>>>>>>> for most users who do not have deep knowledge of KIP-848/932.
> > >>>>>>>>>>>
> > >>>>>>>>>>> As a result, I suggest for kafka-share-groups.sh that you add a
> > >>>>>>>>> --members
> > >>>>>>>>>>> --verbose option
> > >>>>>>>>>>> and only include the new information in the output for that
> > option,
> > >>>>>>>>> rather
> > >>>>>>>>>>> than changing the
> > >>>>>>>>>>> non-verbose --members output.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I also make a similar suggestion for kafka-consumer-groups.sh
> > >>>>>> --members
> > >>>>>>>>>>> and only add the
> > >>>>>>>>>>> new information for the --verbose output.
> > >>>>>>>>>>>
> > >>>>>>>>>>> AS2: I strongly suggest that you use MEMBER-EPOCH instead of
> > >>>>>>>>>>> CONSUMER-EPOCH.
> > >>>>>>>>>>> I think it's more confusing not following the terminology of
> > the
> > >>>>>> KIPs.
> > >>>>>>>>> For
> > >>>>>>>>>>> one thing,
> > >>>>>>>>>>> we already have "member" in the admin client such as
> > >>>>>> MemberDescription.
> > >>>>>>>>>>> It's not a
> > >>>>>>>>>>> new term.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks,
> > >>>>>>>>>>> Andrew
> > >>>>>>>>>>> ________________________________________
> > >>>>>>>>>>> From: PoAn Yang <pa...@apache.org>
> > >>>>>>>>>>> Sent: 25 October 2024 13:55
> > >>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
> > >>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups
> > >>>> command
> > >>>>>>>>> line
> > >>>>>>>>>>> tool to support new consumer group
> > >>>>>>>>>>>
> > >>>>>>>>>>> Hi Lucas,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks for the review!
> > >>>>>>>>>>>
> > >>>>>>>>>>> 1) Yes, I add related change for kafka-share-groups.sh to the
> > KIP.
> > >>>>>> Could
> > >>>>>>>>>>> you take a look? Thanks for the suggestion.
> > >>>>>>>>>>>
> > >>>>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use MEMBER-EPOCH
> > here,
> > >>>>>> users
> > >>>>>>>>> may
> > >>>>>>>>>>> confuse what is different between CONSUMER and MEMBER.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks,
> > >>>>>>>>>>> PoAn
> > >>>>>>>>>>>
> > >>>>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
> > >>>>>>>>>>>> Hi Frank,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> thanks for the KIP!
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 1) For consistency, should we do the same for
> > >>>>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription, etc. ? Even if
> > we do
> > >>>>>> not
> > >>>>>>>>>>>> implement it right now if the share group implementation may
> > still
> > >>>>>> be
> > >>>>>>>>>>>> incomplete, it may make sense to include it in the KIP.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH? That would
> > seem
> > >>>>>> more
> > >>>>>>>>>>>> consistent.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>> Lucas
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <
> > yangpoan@gmail.com>
> > >>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I would like to kick off the discussion of KIP-1099. This KIP
> > >>>>>> enhances
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>> kafka-consumer-groups tools to include state which is
> > introduced
> > >>>> by
> > >>>>>>>>>>> KIP-848.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups command line tool to
> > >>>> support
> > >>>>>>>>> new
> > >>>>>>>>>>>>> consumer group - Apache Kafka - Apache Software Foundation
> > >>>>>>>>>>>>> <
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> cwiki.apache.org
> > >>>>>>>>>>>>> <
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> [image: favicon.ico]
> > >>>>>>>>>>>>> <
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> <
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thank you,
> > >>>>>>>>>>>>> PoAn
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>
> > >>>>
> > >>
> > >>
> >
> >
>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by "Lianet M." <li...@gmail.com>.
Hello PoAn, just of couple more minor comments:
LM4. Regarding the new “protocol” field added to MemberDescription. Should
we consider reusing the existing GroupProtocol enum instead of String?
(It’s the one we use from the consumer side to refer to the protocol in
use, just missing Share I notice).
LM5. Regarding the change to the output for kafka-consumer-groups, the
command shown does not include the —state option, but the output shows
state info (state, #members, epochs). I would guess that we want to modify
the output only when we describe a group with the —state —verbose option,
is my understanding right? If my understanding is right we’re just missing
adding the —state in the example, and the KIP does not introduce any
changes to the —describe —verbose option. (If not, it would mean a bigger
change to the output of —describe —verbose which I expect is not the
intention?)
Thanks!
Lianet
On Wed, Nov 20, 2024 at 2:12 AM PoAn Yang <ya...@gmail.com> wrote:
> Hi Chia-Ping,
>
> Thanks for the review and suggestions.
>
> CI0: Thanks for the reminder. Update validVersions in
> ConsumerGroupDescribeRequest to 0-1.
>
> CI1: Yes, I use `ConsumerGroupMember#useClassicProtocol` to check whether
> a member in consumer group uses “classic” or “consumer” protocol.
>
> CI2: Yes, a member in share group always uses “share” protocol.
>
> CI3: Add a table to show meaning of “classic”, “consumer”, and “share”
> protocol.
>
> BTW, the vote thread is in
> https://lists.apache.org/thread/rb25tf75tzf4c7jqqldlo5jh9w8chsq6.
>
> Thanks,
> PoAn
>
> > On Nov 20, 2024, at 11:46 AM, Chia-Ping Tsai <ch...@apache.org>
> wrote:
> >
> > hi PoAn
> >
> > CI0: We have to bump the version of ConsumerGroupDescribeRequest as
> well, so server can distinguish the new and old behavior.
> >
> > CI1: the type of new filed is string, so I guess you plan to use
> `ConsumerGroupMember#useClassicProtocol` [0] flag to return either
> "classic" or "consumer", right?
> >
> > CI2: `MemberDescription` is used by `ShareGroupDescription` too, so the
> new filed protocol in shared group is always "shared", right?
> >
> > CI3: Could you consider adding a table to show the value of the protocol
> field in each case? Andrew has a beautiful table in KIP-1043 that lists all
> possible protocol names.
> >
> > [0]
> https://github.com/apache/kafka/blob/441a6d0b790f4a17b454caeea7588a6b90fbd9db/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/consumer/ConsumerGroupMember.java#L454
>
> >
> > Best,
> > Chia-Ping
> >
> > On 2024/11/19 15:45:16 PoAn Yang wrote:
> >> Hi David,
> >>
> >> Thanks for the reminder.
> >>
> >> DJ4: Add related content to the KIP.
> >> Bump validVersions in ConsumerGroupDescribeResponse to “0-1”.
> >> Add a new field “Protocol” to ConsumerGroupDescribeResponse.
> >>
> >> Thanks,
> >> PoAn
> >>
> >>> On Nov 19, 2024, at 3:57 PM, David Jacot <dj...@confluent.io.INVALID>
> wrote:
> >>>
> >>> Hi PoAn,
> >>>
> >>> DJ4: Could you please also include the required changes to the
> >>> ConsumerGroupDescribed API in the public interface section? We
> basically
> >>> need to bump the version, add the new field, etc.
> >>>
> >>> Thanks,
> >>> David
> >>>
> >>> On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang <ya...@gmail.com> wrote:
> >>>
> >>>> Hi David,
> >>>>
> >>>> DJ4: Add a new field protocol to MemberDescription,
> >>>> so the command line tool can show protocol information when users
> describe
> >>>> members.
> >>>>
> >>>> If there is no further suggestion, I will start a vote thread today.
> >>>>
> >>>> Thanks,
> >>>> PoAn
> >>>>
> >>>>> On Nov 15, 2024, at 11:50 PM, David Jacot
> <dj...@confluent.io.INVALID>
> >>>> wrote:
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> DJ2: Using "-" sounds good to me.
> >>>>>
> >>>>> DJ3: That seems reasonable to me.
> >>>>>
> >>>>> DJ4: Why not add it right now? I don't want to change the output of
> the
> >>>>> tool too many times.
> >>>>>
> >>>>> Best,
> >>>>> David
> >>>>>
> >>>>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang <ya...@gmail.com>
> wrote:
> >>>>>
> >>>>>> Hi David / Andrew,
> >>>>>>
> >>>>>> Thanks for review. Thanks Andrew for picking up
> kafka-share-groups.sh
> >>>>>> implementation.
> >>>>>> I will handle kafka-consumer-groups.sh.
> >>>>>>
> >>>>>> DJ3: After discussing with @Chia-Ping Tsai, we think that using new
> >>>> format
> >>>>>> is more clear.
> >>>>>>
> >>>>>> The new format will be like
> >>>>>>
> >>>>
> <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2>
> >>>>>>
> >>>>>> Using colon(:) to concat topic name and partition IDs.
> >>>>>> Using comma(,) to concat partition IDs within same topic name.
> >>>>>> Using semicolon(;) to concat topic strings.
> >>>>>>
> >>>>>> AS3: Fix it with kafka-share-groups.sh. Thanks.
> >>>>>>
> >>>>>> If there is no further suggestion, I will start a vote thread next
> >>>> Monday.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> PoAn
> >>>>>>
> >>>>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield <
> >>>>>> andrew_schofield_jira@outlook.com> wrote:
> >>>>>>>
> >>>>>>> Hi PoAn,
> >>>>>>> DJ2: I was just going to comment that "-" would be a more
> appropriate
> >>>>>> missing value, but
> >>>>>>> you got there first.
> >>>>>>>
> >>>>>>> AS3: The examples for kafka-share-groups.sh include
> >>>>>> kafka-consumer-groups.sh in the
> >>>>>>> command line.
> >>>>>>>
> >>>>>>> If this is accepted in time, I'm happy to pick up the
> implementation of
> >>>>>> the share groups
> >>>>>>> part of this if it helps.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Andrew
> >>>>>>>
> >>>>>>> ________________________________________
> >>>>>>> From: Frank Yang <ya...@gmail.com>
> >>>>>>> Sent: 14 November 2024 10:48
> >>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
> >>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups
> command
> >>>>>> line tool to support new consumer group
> >>>>>>>
> >>>>>>> Hi David,
> >>>>>>>
> >>>>>>> Thanks for the review and suggestion! I would like to get this in
> AK
> >>>> 4.0
> >>>>>> as well. I will do my best.
> >>>>>>>
> >>>>>>> DJ1: Update KIP to put GROUP-EPOCH and TARGET-ASSIGNMENT-EPOCH
> before
> >>>>>> #MEMBERS.
> >>>>>>>
> >>>>>>> DJ2: I prefer to follow current missing column value “-“.
> (reference <
> >>>>>>
> >>>>
> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92
> >>>>>>> )
> >>>>>>>
> >>>>>>> DJ3: Update KIP to use CURRENT-EPOCH CURRENT-ASSIGNMENT
> TARGET-EPOCH
> >>>>>> TARGET-ASSIGNMENT.
> >>>>>>> Remove GROUP-EPOCH.
> >>>>>>>
> >>>>>>> For assignment value, it follows current output (reference <
> >>>>>>
> >>>>
> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418
> >>>>> ).
> >>>>>> I think the form of `topicid-partitionid` is more clear.
> >>>>>>> If we would like to use this form, I will update both output in
> >>>>>> kafka-consumer-groups.sh and kafka-share-groups.sh.
> >>>>>>>
> >>>>>>> DJ4: It looks like DescribeGroupsResponseData only has protocol
> type at
> >>>>>> group level.
> >>>>>>> Both DescribeGroupsResponseData and
> ConsumerGroupDescribeResponseData
> >>>>>> don’t have protocol at member level.
> >>>>>>> Could we use a followup to add it?
> >>>>>>>
> >>>>>>> DJ5: Update KIP to put LEADER-EPOCH before CURRENT-OFFSET.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> PoAn
> >>>>>>>
> >>>>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot
> <djacot@confluent.io.INVALID
> >>>>>
> >>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hi PoAn,
> >>>>>>>>
> >>>>>>>> Thanks for the KIP! I have a few minor comments/suggestions:
> >>>>>>>>
> >>>>>>>> DJ1: In the output of `--describe --verbose`, I would suggest
> putting
> >>>>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before `#MEMBERS`.
> >>>>>>>> DJ2: Continuing on the above, I assume that we will print out N/A
> for
> >>>>>> the
> >>>>>>>> fields not supported by classic groups. Is this correct?
> >>>>>>>> DJ3: In the output of `--describe --members --verbose`, I wonder
> if we
> >>>>>>>> should use the following order and terms: CURRENT-EPOCH
> >>>>>> CURRENT-ASSIGNMENT
> >>>>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the GROUP-EPOCH
> >>>> because
> >>>>>> it
> >>>>>>>> is already in the group description. The value `(0)` used for the
> >>>>>>>> assignment is incorrect. Here I suppose that we will print out the
> >>>> list
> >>>>>> of
> >>>>>>>> partitions in the form of `topicid-partitionid`.
> >>>>>>>> DJ4: Continuing on the above, I wonder if we should also add the
> >>>>>> protocol
> >>>>>>>> used `classic` or `consumer`. For context, it is possible to have
> >>>>>> `classic`
> >>>>>>>> members and `consumer` members in a `consumer` group during an
> online
> >>>>>>>> upgrade from the classic protocol to the consumer protocol. Having
> >>>> this
> >>>>>>>> information may be handy for administrators. What do you think?
> >>>>>>>> DJ5: In the output of `--describe --offsets --verbose`, I would
> >>>> suggest
> >>>>>>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`.
> >>>>>>>>
> >>>>>>>> It would be great if we could get this one in AK 4.0 as it
> changes the
> >>>>>>>> output of the command.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> DJ
> >>>>>>>>
> >>>>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <ya...@gmail.com>
> wrote:
> >>>>>>>>
> >>>>>>>>> Hi Sean / Andrew / Lianet,
> >>>>>>>>>
> >>>>>>>>> Thanks for all your review and suggestions.
> >>>>>>>>>
> >>>>>>>>> AS1, LM1, LM4: Change to add KIP-848 information when users give
> >>>>>> —verbose
> >>>>>>>>> option.
> >>>>>>>>> —describe —verbose: shows group epoch and target assignment
> epoch.
> >>>>>>>>> —describe —members —verbose: shows above information, member
> epoch,
> >>>> and
> >>>>>>>>> target assignment.
> >>>>>>>>>
> >>>>>>>>> AS2: Change to use MEMBER-EPOCH to align with KIP-848 definition.
> >>>>>>>>>
> >>>>>>>>> LM2: For classic group, it doesn’t have epoch, so I use Optional
> >>>>>> fields in
> >>>>>>>>> ConsumerGroupDescription.
> >>>>>>>>> For share group, it relies on KIP-848. It must have epoch, so I
> use
> >>>> int
> >>>>>>>>> fields in ShareGroupDescription.
> >>>>>>>>>
> >>>>>>>>> LM3: Remove —state change. We can get group level information by
> >>>>>> —describe
> >>>>>>>>> —verbose.
> >>>>>>>>>
> >>>>>>>>> SQ1: Add LEADER-EPOCH when users give —describe —offsets
> —verbose.
> >>>>>>>>>
> >>>>>>>>> Thanks.
> >>>>>>>>> PoAn
> >>>>>>>>>
> >>>>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <li...@gmail.com>
> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hello Frank, thanks for the KIP! A few comments:
> >>>>>>>>>>
> >>>>>>>>>> LM1. I strongly agree with Andrew's suggestion of moving this
> into a
> >>>>>>>>>> --verbose option. I expect someone would only attempt to make
> sense
> >>>> of
> >>>>>>>>> the
> >>>>>>>>>> epochs while debugging an issue, not while trying to view the
> group
> >>>> or
> >>>>>>>>>> member's info. (Also member-epoch makes more sense to me, even
> more
> >>>>>> if we
> >>>>>>>>>> end up agreeing in a --verbose). Related to both issues, my
> take is
> >>>>>> that
> >>>>>>>>>> whoever is close to the protocol will expect member-epoch,
> whoever
> >>>> is
> >>>>>> not
> >>>>>>>>>> will probably won't even need to see the epochs at all.
> >>>>>>>>>>
> >>>>>>>>>> LM2. Why are the epochs defined as Optional (don't we expect to
> >>>> always
> >>>>>>>>> have
> >>>>>>>>>> them? with 0 initially, defined on the broker side for the group
> >>>> ones,
> >>>>>>>>> and
> >>>>>>>>>> client side for the member)
> >>>>>>>>>>
> >>>>>>>>>> LM3. Why is the KIP including the –state option in the proposed
> >>>>>> changes?
> >>>>>>>>> (
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
> >>>>>>>>>> ).
> >>>>>>>>>> I get that the output would change in that example, but it’s not
> >>>>>> because
> >>>>>>>>> of
> >>>>>>>>>> any change to the –state option itself. It's because of the
> change
> >>>>>>>>> proposed
> >>>>>>>>>> to the –described (required with the --state), and the changes
> to
> >>>>>>>>>> --describe are already explained above (seems confusing, got me
> >>>>>> looking
> >>>>>>>>> for
> >>>>>>>>>> a change to the state filter which seemed unrelated).
> >>>>>>>>>>
> >>>>>>>>>> LM4. Group epoch and target assignment epoch are conceptually
> at the
> >>>>>>>>> group
> >>>>>>>>>> level. vs member epoch that lands at a member level. So wonder
> if we
> >>>>>>>>> should
> >>>>>>>>>> show them accordingly (ex. using the --verbose option)
> >>>>>>>>>> –describe –verbose => shows group epoch and target assignment
> epoch
> >>>>>>>>>> –describe –members –verbose => shows member epoch
> (along
> >>>>>>>>>> with group epoch and target assignment epoch)
> >>>>>>>>>>
> >>>>>>>>>> Thanks!
> >>>>>>>>>>
> >>>>>>>>>> Lianet
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield <
> >>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi PoAn,
> >>>>>>>>>>> Thanks for the KIP. I have a few comments.
> >>>>>>>>>>>
> >>>>>>>>>>> AS1: It seems to me that these new additions are most useful to
> >>>>>> people
> >>>>>>>>>>> trying to understand
> >>>>>>>>>>> the progress of rebalancing in quite some detail. The
> information
> >>>> is
> >>>>>>>>>>> really not understandable
> >>>>>>>>>>> for most users who do not have deep knowledge of KIP-848/932.
> >>>>>>>>>>>
> >>>>>>>>>>> As a result, I suggest for kafka-share-groups.sh that you add a
> >>>>>>>>> --members
> >>>>>>>>>>> --verbose option
> >>>>>>>>>>> and only include the new information in the output for that
> option,
> >>>>>>>>> rather
> >>>>>>>>>>> than changing the
> >>>>>>>>>>> non-verbose --members output.
> >>>>>>>>>>>
> >>>>>>>>>>> I also make a similar suggestion for kafka-consumer-groups.sh
> >>>>>> --members
> >>>>>>>>>>> and only add the
> >>>>>>>>>>> new information for the --verbose output.
> >>>>>>>>>>>
> >>>>>>>>>>> AS2: I strongly suggest that you use MEMBER-EPOCH instead of
> >>>>>>>>>>> CONSUMER-EPOCH.
> >>>>>>>>>>> I think it's more confusing not following the terminology of
> the
> >>>>>> KIPs.
> >>>>>>>>> For
> >>>>>>>>>>> one thing,
> >>>>>>>>>>> we already have "member" in the admin client such as
> >>>>>> MemberDescription.
> >>>>>>>>>>> It's not a
> >>>>>>>>>>> new term.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>> Andrew
> >>>>>>>>>>> ________________________________________
> >>>>>>>>>>> From: PoAn Yang <pa...@apache.org>
> >>>>>>>>>>> Sent: 25 October 2024 13:55
> >>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
> >>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups
> >>>> command
> >>>>>>>>> line
> >>>>>>>>>>> tool to support new consumer group
> >>>>>>>>>>>
> >>>>>>>>>>> Hi Lucas,
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for the review!
> >>>>>>>>>>>
> >>>>>>>>>>> 1) Yes, I add related change for kafka-share-groups.sh to the
> KIP.
> >>>>>> Could
> >>>>>>>>>>> you take a look? Thanks for the suggestion.
> >>>>>>>>>>>
> >>>>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use MEMBER-EPOCH
> here,
> >>>>>> users
> >>>>>>>>> may
> >>>>>>>>>>> confuse what is different between CONSUMER and MEMBER.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>> PoAn
> >>>>>>>>>>>
> >>>>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
> >>>>>>>>>>>> Hi Frank,
> >>>>>>>>>>>>
> >>>>>>>>>>>> thanks for the KIP!
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1) For consistency, should we do the same for
> >>>>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription, etc. ? Even if
> we do
> >>>>>> not
> >>>>>>>>>>>> implement it right now if the share group implementation may
> still
> >>>>>> be
> >>>>>>>>>>>> incomplete, it may make sense to include it in the KIP.
> >>>>>>>>>>>>
> >>>>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH? That would
> seem
> >>>>>> more
> >>>>>>>>>>>> consistent.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Cheers,
> >>>>>>>>>>>> Lucas
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <
> yangpoan@gmail.com>
> >>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I would like to kick off the discussion of KIP-1099. This KIP
> >>>>>> enhances
> >>>>>>>>>>> the
> >>>>>>>>>>>>> kafka-consumer-groups tools to include state which is
> introduced
> >>>> by
> >>>>>>>>>>> KIP-848.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups command line tool to
> >>>> support
> >>>>>>>>> new
> >>>>>>>>>>>>> consumer group - Apache Kafka - Apache Software Foundation
> >>>>>>>>>>>>> <
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>
> >>>>>>>>>>>>> cwiki.apache.org
> >>>>>>>>>>>>> <
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>
> >>>>>>>>>>>>> [image: favicon.ico]
> >>>>>>>>>>>>> <
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>
> >>>>>>>>>>>>> <
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thank you,
> >>>>>>>>>>>>> PoAn
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
> >>
>
>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by PoAn Yang <ya...@gmail.com>.
Hi Chia-Ping,
Thanks for the review and suggestions.
CI0: Thanks for the reminder. Update validVersions in ConsumerGroupDescribeRequest to 0-1.
CI1: Yes, I use `ConsumerGroupMember#useClassicProtocol` to check whether a member in consumer group uses “classic” or “consumer” protocol.
CI2: Yes, a member in share group always uses “share” protocol.
CI3: Add a table to show meaning of “classic”, “consumer”, and “share” protocol.
BTW, the vote thread is in https://lists.apache.org/thread/rb25tf75tzf4c7jqqldlo5jh9w8chsq6.
Thanks,
PoAn
> On Nov 20, 2024, at 11:46 AM, Chia-Ping Tsai <ch...@apache.org> wrote:
>
> hi PoAn
>
> CI0: We have to bump the version of ConsumerGroupDescribeRequest as well, so server can distinguish the new and old behavior.
>
> CI1: the type of new filed is string, so I guess you plan to use `ConsumerGroupMember#useClassicProtocol` [0] flag to return either "classic" or "consumer", right?
>
> CI2: `MemberDescription` is used by `ShareGroupDescription` too, so the new filed protocol in shared group is always "shared", right?
>
> CI3: Could you consider adding a table to show the value of the protocol field in each case? Andrew has a beautiful table in KIP-1043 that lists all possible protocol names.
>
> [0] https://github.com/apache/kafka/blob/441a6d0b790f4a17b454caeea7588a6b90fbd9db/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/consumer/ConsumerGroupMember.java#L454
>
> Best,
> Chia-Ping
>
> On 2024/11/19 15:45:16 PoAn Yang wrote:
>> Hi David,
>>
>> Thanks for the reminder.
>>
>> DJ4: Add related content to the KIP.
>> Bump validVersions in ConsumerGroupDescribeResponse to “0-1”.
>> Add a new field “Protocol” to ConsumerGroupDescribeResponse.
>>
>> Thanks,
>> PoAn
>>
>>> On Nov 19, 2024, at 3:57 PM, David Jacot <dj...@confluent.io.INVALID> wrote:
>>>
>>> Hi PoAn,
>>>
>>> DJ4: Could you please also include the required changes to the
>>> ConsumerGroupDescribed API in the public interface section? We basically
>>> need to bump the version, add the new field, etc.
>>>
>>> Thanks,
>>> David
>>>
>>> On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang <ya...@gmail.com> wrote:
>>>
>>>> Hi David,
>>>>
>>>> DJ4: Add a new field protocol to MemberDescription,
>>>> so the command line tool can show protocol information when users describe
>>>> members.
>>>>
>>>> If there is no further suggestion, I will start a vote thread today.
>>>>
>>>> Thanks,
>>>> PoAn
>>>>
>>>>> On Nov 15, 2024, at 11:50 PM, David Jacot <dj...@confluent.io.INVALID>
>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> DJ2: Using "-" sounds good to me.
>>>>>
>>>>> DJ3: That seems reasonable to me.
>>>>>
>>>>> DJ4: Why not add it right now? I don't want to change the output of the
>>>>> tool too many times.
>>>>>
>>>>> Best,
>>>>> David
>>>>>
>>>>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang <ya...@gmail.com> wrote:
>>>>>
>>>>>> Hi David / Andrew,
>>>>>>
>>>>>> Thanks for review. Thanks Andrew for picking up kafka-share-groups.sh
>>>>>> implementation.
>>>>>> I will handle kafka-consumer-groups.sh.
>>>>>>
>>>>>> DJ3: After discussing with @Chia-Ping Tsai, we think that using new
>>>> format
>>>>>> is more clear.
>>>>>>
>>>>>> The new format will be like
>>>>>>
>>>> <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2>
>>>>>>
>>>>>> Using colon(:) to concat topic name and partition IDs.
>>>>>> Using comma(,) to concat partition IDs within same topic name.
>>>>>> Using semicolon(;) to concat topic strings.
>>>>>>
>>>>>> AS3: Fix it with kafka-share-groups.sh. Thanks.
>>>>>>
>>>>>> If there is no further suggestion, I will start a vote thread next
>>>> Monday.
>>>>>>
>>>>>> Thanks,
>>>>>> PoAn
>>>>>>
>>>>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield <
>>>>>> andrew_schofield_jira@outlook.com> wrote:
>>>>>>>
>>>>>>> Hi PoAn,
>>>>>>> DJ2: I was just going to comment that "-" would be a more appropriate
>>>>>> missing value, but
>>>>>>> you got there first.
>>>>>>>
>>>>>>> AS3: The examples for kafka-share-groups.sh include
>>>>>> kafka-consumer-groups.sh in the
>>>>>>> command line.
>>>>>>>
>>>>>>> If this is accepted in time, I'm happy to pick up the implementation of
>>>>>> the share groups
>>>>>>> part of this if it helps.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Andrew
>>>>>>>
>>>>>>> ________________________________________
>>>>>>> From: Frank Yang <ya...@gmail.com>
>>>>>>> Sent: 14 November 2024 10:48
>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command
>>>>>> line tool to support new consumer group
>>>>>>>
>>>>>>> Hi David,
>>>>>>>
>>>>>>> Thanks for the review and suggestion! I would like to get this in AK
>>>> 4.0
>>>>>> as well. I will do my best.
>>>>>>>
>>>>>>> DJ1: Update KIP to put GROUP-EPOCH and TARGET-ASSIGNMENT-EPOCH before
>>>>>> #MEMBERS.
>>>>>>>
>>>>>>> DJ2: I prefer to follow current missing column value “-“. (reference <
>>>>>>
>>>> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92
>>>>>>> )
>>>>>>>
>>>>>>> DJ3: Update KIP to use CURRENT-EPOCH CURRENT-ASSIGNMENT TARGET-EPOCH
>>>>>> TARGET-ASSIGNMENT.
>>>>>>> Remove GROUP-EPOCH.
>>>>>>>
>>>>>>> For assignment value, it follows current output (reference <
>>>>>>
>>>> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418
>>>>> ).
>>>>>> I think the form of `topicid-partitionid` is more clear.
>>>>>>> If we would like to use this form, I will update both output in
>>>>>> kafka-consumer-groups.sh and kafka-share-groups.sh.
>>>>>>>
>>>>>>> DJ4: It looks like DescribeGroupsResponseData only has protocol type at
>>>>>> group level.
>>>>>>> Both DescribeGroupsResponseData and ConsumerGroupDescribeResponseData
>>>>>> don’t have protocol at member level.
>>>>>>> Could we use a followup to add it?
>>>>>>>
>>>>>>> DJ5: Update KIP to put LEADER-EPOCH before CURRENT-OFFSET.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> PoAn
>>>>>>>
>>>>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot <djacot@confluent.io.INVALID
>>>>>
>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi PoAn,
>>>>>>>>
>>>>>>>> Thanks for the KIP! I have a few minor comments/suggestions:
>>>>>>>>
>>>>>>>> DJ1: In the output of `--describe --verbose`, I would suggest putting
>>>>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before `#MEMBERS`.
>>>>>>>> DJ2: Continuing on the above, I assume that we will print out N/A for
>>>>>> the
>>>>>>>> fields not supported by classic groups. Is this correct?
>>>>>>>> DJ3: In the output of `--describe --members --verbose`, I wonder if we
>>>>>>>> should use the following order and terms: CURRENT-EPOCH
>>>>>> CURRENT-ASSIGNMENT
>>>>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the GROUP-EPOCH
>>>> because
>>>>>> it
>>>>>>>> is already in the group description. The value `(0)` used for the
>>>>>>>> assignment is incorrect. Here I suppose that we will print out the
>>>> list
>>>>>> of
>>>>>>>> partitions in the form of `topicid-partitionid`.
>>>>>>>> DJ4: Continuing on the above, I wonder if we should also add the
>>>>>> protocol
>>>>>>>> used `classic` or `consumer`. For context, it is possible to have
>>>>>> `classic`
>>>>>>>> members and `consumer` members in a `consumer` group during an online
>>>>>>>> upgrade from the classic protocol to the consumer protocol. Having
>>>> this
>>>>>>>> information may be handy for administrators. What do you think?
>>>>>>>> DJ5: In the output of `--describe --offsets --verbose`, I would
>>>> suggest
>>>>>>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`.
>>>>>>>>
>>>>>>>> It would be great if we could get this one in AK 4.0 as it changes the
>>>>>>>> output of the command.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> DJ
>>>>>>>>
>>>>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <ya...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Sean / Andrew / Lianet,
>>>>>>>>>
>>>>>>>>> Thanks for all your review and suggestions.
>>>>>>>>>
>>>>>>>>> AS1, LM1, LM4: Change to add KIP-848 information when users give
>>>>>> —verbose
>>>>>>>>> option.
>>>>>>>>> —describe —verbose: shows group epoch and target assignment epoch.
>>>>>>>>> —describe —members —verbose: shows above information, member epoch,
>>>> and
>>>>>>>>> target assignment.
>>>>>>>>>
>>>>>>>>> AS2: Change to use MEMBER-EPOCH to align with KIP-848 definition.
>>>>>>>>>
>>>>>>>>> LM2: For classic group, it doesn’t have epoch, so I use Optional
>>>>>> fields in
>>>>>>>>> ConsumerGroupDescription.
>>>>>>>>> For share group, it relies on KIP-848. It must have epoch, so I use
>>>> int
>>>>>>>>> fields in ShareGroupDescription.
>>>>>>>>>
>>>>>>>>> LM3: Remove —state change. We can get group level information by
>>>>>> —describe
>>>>>>>>> —verbose.
>>>>>>>>>
>>>>>>>>> SQ1: Add LEADER-EPOCH when users give —describe —offsets —verbose.
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>> PoAn
>>>>>>>>>
>>>>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <li...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Hello Frank, thanks for the KIP! A few comments:
>>>>>>>>>>
>>>>>>>>>> LM1. I strongly agree with Andrew's suggestion of moving this into a
>>>>>>>>>> --verbose option. I expect someone would only attempt to make sense
>>>> of
>>>>>>>>> the
>>>>>>>>>> epochs while debugging an issue, not while trying to view the group
>>>> or
>>>>>>>>>> member's info. (Also member-epoch makes more sense to me, even more
>>>>>> if we
>>>>>>>>>> end up agreeing in a --verbose). Related to both issues, my take is
>>>>>> that
>>>>>>>>>> whoever is close to the protocol will expect member-epoch, whoever
>>>> is
>>>>>> not
>>>>>>>>>> will probably won't even need to see the epochs at all.
>>>>>>>>>>
>>>>>>>>>> LM2. Why are the epochs defined as Optional (don't we expect to
>>>> always
>>>>>>>>> have
>>>>>>>>>> them? with 0 initially, defined on the broker side for the group
>>>> ones,
>>>>>>>>> and
>>>>>>>>>> client side for the member)
>>>>>>>>>>
>>>>>>>>>> LM3. Why is the KIP including the –state option in the proposed
>>>>>> changes?
>>>>>>>>> (
>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
>>>>>>>>>> ).
>>>>>>>>>> I get that the output would change in that example, but it’s not
>>>>>> because
>>>>>>>>> of
>>>>>>>>>> any change to the –state option itself. It's because of the change
>>>>>>>>> proposed
>>>>>>>>>> to the –described (required with the --state), and the changes to
>>>>>>>>>> --describe are already explained above (seems confusing, got me
>>>>>> looking
>>>>>>>>> for
>>>>>>>>>> a change to the state filter which seemed unrelated).
>>>>>>>>>>
>>>>>>>>>> LM4. Group epoch and target assignment epoch are conceptually at the
>>>>>>>>> group
>>>>>>>>>> level. vs member epoch that lands at a member level. So wonder if we
>>>>>>>>> should
>>>>>>>>>> show them accordingly (ex. using the --verbose option)
>>>>>>>>>> –describe –verbose => shows group epoch and target assignment epoch
>>>>>>>>>> –describe –members –verbose => shows member epoch (along
>>>>>>>>>> with group epoch and target assignment epoch)
>>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>>
>>>>>>>>>> Lianet
>>>>>>>>>>
>>>>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield <
>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi PoAn,
>>>>>>>>>>> Thanks for the KIP. I have a few comments.
>>>>>>>>>>>
>>>>>>>>>>> AS1: It seems to me that these new additions are most useful to
>>>>>> people
>>>>>>>>>>> trying to understand
>>>>>>>>>>> the progress of rebalancing in quite some detail. The information
>>>> is
>>>>>>>>>>> really not understandable
>>>>>>>>>>> for most users who do not have deep knowledge of KIP-848/932.
>>>>>>>>>>>
>>>>>>>>>>> As a result, I suggest for kafka-share-groups.sh that you add a
>>>>>>>>> --members
>>>>>>>>>>> --verbose option
>>>>>>>>>>> and only include the new information in the output for that option,
>>>>>>>>> rather
>>>>>>>>>>> than changing the
>>>>>>>>>>> non-verbose --members output.
>>>>>>>>>>>
>>>>>>>>>>> I also make a similar suggestion for kafka-consumer-groups.sh
>>>>>> --members
>>>>>>>>>>> and only add the
>>>>>>>>>>> new information for the --verbose output.
>>>>>>>>>>>
>>>>>>>>>>> AS2: I strongly suggest that you use MEMBER-EPOCH instead of
>>>>>>>>>>> CONSUMER-EPOCH.
>>>>>>>>>>> I think it's more confusing not following the terminology of the
>>>>>> KIPs.
>>>>>>>>> For
>>>>>>>>>>> one thing,
>>>>>>>>>>> we already have "member" in the admin client such as
>>>>>> MemberDescription.
>>>>>>>>>>> It's not a
>>>>>>>>>>> new term.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Andrew
>>>>>>>>>>> ________________________________________
>>>>>>>>>>> From: PoAn Yang <pa...@apache.org>
>>>>>>>>>>> Sent: 25 October 2024 13:55
>>>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups
>>>> command
>>>>>>>>> line
>>>>>>>>>>> tool to support new consumer group
>>>>>>>>>>>
>>>>>>>>>>> Hi Lucas,
>>>>>>>>>>>
>>>>>>>>>>> Thanks for the review!
>>>>>>>>>>>
>>>>>>>>>>> 1) Yes, I add related change for kafka-share-groups.sh to the KIP.
>>>>>> Could
>>>>>>>>>>> you take a look? Thanks for the suggestion.
>>>>>>>>>>>
>>>>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use MEMBER-EPOCH here,
>>>>>> users
>>>>>>>>> may
>>>>>>>>>>> confuse what is different between CONSUMER and MEMBER.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> PoAn
>>>>>>>>>>>
>>>>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
>>>>>>>>>>>> Hi Frank,
>>>>>>>>>>>>
>>>>>>>>>>>> thanks for the KIP!
>>>>>>>>>>>>
>>>>>>>>>>>> 1) For consistency, should we do the same for
>>>>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription, etc. ? Even if we do
>>>>>> not
>>>>>>>>>>>> implement it right now if the share group implementation may still
>>>>>> be
>>>>>>>>>>>> incomplete, it may make sense to include it in the KIP.
>>>>>>>>>>>>
>>>>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH? That would seem
>>>>>> more
>>>>>>>>>>>> consistent.
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> Lucas
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <ya...@gmail.com>
>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I would like to kick off the discussion of KIP-1099. This KIP
>>>>>> enhances
>>>>>>>>>>> the
>>>>>>>>>>>>> kafka-consumer-groups tools to include state which is introduced
>>>> by
>>>>>>>>>>> KIP-848.
>>>>>>>>>>>>>
>>>>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups command line tool to
>>>> support
>>>>>>>>> new
>>>>>>>>>>>>> consumer group - Apache Kafka - Apache Software Foundation
>>>>>>>>>>>>> <
>>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>
>>>>>>>>>>>>> cwiki.apache.org
>>>>>>>>>>>>> <
>>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>
>>>>>>>>>>>>> [image: favicon.ico]
>>>>>>>>>>>>> <
>>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>
>>>>>>>>>>>>> <
>>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>> PoAn
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>
>>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by Chia-Ping Tsai <ch...@apache.org>.
hi PoAn
CI0: We have to bump the version of ConsumerGroupDescribeRequest as well, so server can distinguish the new and old behavior.
CI1: the type of new filed is string, so I guess you plan to use `ConsumerGroupMember#useClassicProtocol` [0] flag to return either "classic" or "consumer", right?
CI2: `MemberDescription` is used by `ShareGroupDescription` too, so the new filed protocol in shared group is always "shared", right?
CI3: Could you consider adding a table to show the value of the protocol field in each case? Andrew has a beautiful table in KIP-1043 that lists all possible protocol names.
[0] https://github.com/apache/kafka/blob/441a6d0b790f4a17b454caeea7588a6b90fbd9db/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/consumer/ConsumerGroupMember.java#L454
Best,
Chia-Ping
On 2024/11/19 15:45:16 PoAn Yang wrote:
> Hi David,
>
> Thanks for the reminder.
>
> DJ4: Add related content to the KIP.
> Bump validVersions in ConsumerGroupDescribeResponse to “0-1”.
> Add a new field “Protocol” to ConsumerGroupDescribeResponse.
>
> Thanks,
> PoAn
>
> > On Nov 19, 2024, at 3:57 PM, David Jacot <dj...@confluent.io.INVALID> wrote:
> >
> > Hi PoAn,
> >
> > DJ4: Could you please also include the required changes to the
> > ConsumerGroupDescribed API in the public interface section? We basically
> > need to bump the version, add the new field, etc.
> >
> > Thanks,
> > David
> >
> > On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang <ya...@gmail.com> wrote:
> >
> >> Hi David,
> >>
> >> DJ4: Add a new field protocol to MemberDescription,
> >> so the command line tool can show protocol information when users describe
> >> members.
> >>
> >> If there is no further suggestion, I will start a vote thread today.
> >>
> >> Thanks,
> >> PoAn
> >>
> >>> On Nov 15, 2024, at 11:50 PM, David Jacot <dj...@confluent.io.INVALID>
> >> wrote:
> >>>
> >>> Hi,
> >>>
> >>> DJ2: Using "-" sounds good to me.
> >>>
> >>> DJ3: That seems reasonable to me.
> >>>
> >>> DJ4: Why not add it right now? I don't want to change the output of the
> >>> tool too many times.
> >>>
> >>> Best,
> >>> David
> >>>
> >>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang <ya...@gmail.com> wrote:
> >>>
> >>>> Hi David / Andrew,
> >>>>
> >>>> Thanks for review. Thanks Andrew for picking up kafka-share-groups.sh
> >>>> implementation.
> >>>> I will handle kafka-consumer-groups.sh.
> >>>>
> >>>> DJ3: After discussing with @Chia-Ping Tsai, we think that using new
> >> format
> >>>> is more clear.
> >>>>
> >>>> The new format will be like
> >>>>
> >> <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2>
> >>>>
> >>>> Using colon(:) to concat topic name and partition IDs.
> >>>> Using comma(,) to concat partition IDs within same topic name.
> >>>> Using semicolon(;) to concat topic strings.
> >>>>
> >>>> AS3: Fix it with kafka-share-groups.sh. Thanks.
> >>>>
> >>>> If there is no further suggestion, I will start a vote thread next
> >> Monday.
> >>>>
> >>>> Thanks,
> >>>> PoAn
> >>>>
> >>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield <
> >>>> andrew_schofield_jira@outlook.com> wrote:
> >>>>>
> >>>>> Hi PoAn,
> >>>>> DJ2: I was just going to comment that "-" would be a more appropriate
> >>>> missing value, but
> >>>>> you got there first.
> >>>>>
> >>>>> AS3: The examples for kafka-share-groups.sh include
> >>>> kafka-consumer-groups.sh in the
> >>>>> command line.
> >>>>>
> >>>>> If this is accepted in time, I'm happy to pick up the implementation of
> >>>> the share groups
> >>>>> part of this if it helps.
> >>>>>
> >>>>> Thanks,
> >>>>> Andrew
> >>>>>
> >>>>> ________________________________________
> >>>>> From: Frank Yang <ya...@gmail.com>
> >>>>> Sent: 14 November 2024 10:48
> >>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
> >>>>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command
> >>>> line tool to support new consumer group
> >>>>>
> >>>>> Hi David,
> >>>>>
> >>>>> Thanks for the review and suggestion! I would like to get this in AK
> >> 4.0
> >>>> as well. I will do my best.
> >>>>>
> >>>>> DJ1: Update KIP to put GROUP-EPOCH and TARGET-ASSIGNMENT-EPOCH before
> >>>> #MEMBERS.
> >>>>>
> >>>>> DJ2: I prefer to follow current missing column value “-“. (reference <
> >>>>
> >> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92
> >>>>> )
> >>>>>
> >>>>> DJ3: Update KIP to use CURRENT-EPOCH CURRENT-ASSIGNMENT TARGET-EPOCH
> >>>> TARGET-ASSIGNMENT.
> >>>>> Remove GROUP-EPOCH.
> >>>>>
> >>>>> For assignment value, it follows current output (reference <
> >>>>
> >> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418
> >>> ).
> >>>> I think the form of `topicid-partitionid` is more clear.
> >>>>> If we would like to use this form, I will update both output in
> >>>> kafka-consumer-groups.sh and kafka-share-groups.sh.
> >>>>>
> >>>>> DJ4: It looks like DescribeGroupsResponseData only has protocol type at
> >>>> group level.
> >>>>> Both DescribeGroupsResponseData and ConsumerGroupDescribeResponseData
> >>>> don’t have protocol at member level.
> >>>>> Could we use a followup to add it?
> >>>>>
> >>>>> DJ5: Update KIP to put LEADER-EPOCH before CURRENT-OFFSET.
> >>>>>
> >>>>> Thanks,
> >>>>> PoAn
> >>>>>
> >>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot <djacot@confluent.io.INVALID
> >>>
> >>>> wrote:
> >>>>>>
> >>>>>> Hi PoAn,
> >>>>>>
> >>>>>> Thanks for the KIP! I have a few minor comments/suggestions:
> >>>>>>
> >>>>>> DJ1: In the output of `--describe --verbose`, I would suggest putting
> >>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before `#MEMBERS`.
> >>>>>> DJ2: Continuing on the above, I assume that we will print out N/A for
> >>>> the
> >>>>>> fields not supported by classic groups. Is this correct?
> >>>>>> DJ3: In the output of `--describe --members --verbose`, I wonder if we
> >>>>>> should use the following order and terms: CURRENT-EPOCH
> >>>> CURRENT-ASSIGNMENT
> >>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the GROUP-EPOCH
> >> because
> >>>> it
> >>>>>> is already in the group description. The value `(0)` used for the
> >>>>>> assignment is incorrect. Here I suppose that we will print out the
> >> list
> >>>> of
> >>>>>> partitions in the form of `topicid-partitionid`.
> >>>>>> DJ4: Continuing on the above, I wonder if we should also add the
> >>>> protocol
> >>>>>> used `classic` or `consumer`. For context, it is possible to have
> >>>> `classic`
> >>>>>> members and `consumer` members in a `consumer` group during an online
> >>>>>> upgrade from the classic protocol to the consumer protocol. Having
> >> this
> >>>>>> information may be handy for administrators. What do you think?
> >>>>>> DJ5: In the output of `--describe --offsets --verbose`, I would
> >> suggest
> >>>>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`.
> >>>>>>
> >>>>>> It would be great if we could get this one in AK 4.0 as it changes the
> >>>>>> output of the command.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> DJ
> >>>>>>
> >>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <ya...@gmail.com> wrote:
> >>>>>>
> >>>>>>> Hi Sean / Andrew / Lianet,
> >>>>>>>
> >>>>>>> Thanks for all your review and suggestions.
> >>>>>>>
> >>>>>>> AS1, LM1, LM4: Change to add KIP-848 information when users give
> >>>> —verbose
> >>>>>>> option.
> >>>>>>> —describe —verbose: shows group epoch and target assignment epoch.
> >>>>>>> —describe —members —verbose: shows above information, member epoch,
> >> and
> >>>>>>> target assignment.
> >>>>>>>
> >>>>>>> AS2: Change to use MEMBER-EPOCH to align with KIP-848 definition.
> >>>>>>>
> >>>>>>> LM2: For classic group, it doesn’t have epoch, so I use Optional
> >>>> fields in
> >>>>>>> ConsumerGroupDescription.
> >>>>>>> For share group, it relies on KIP-848. It must have epoch, so I use
> >> int
> >>>>>>> fields in ShareGroupDescription.
> >>>>>>>
> >>>>>>> LM3: Remove —state change. We can get group level information by
> >>>> —describe
> >>>>>>> —verbose.
> >>>>>>>
> >>>>>>> SQ1: Add LEADER-EPOCH when users give —describe —offsets —verbose.
> >>>>>>>
> >>>>>>> Thanks.
> >>>>>>> PoAn
> >>>>>>>
> >>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <li...@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>> Hello Frank, thanks for the KIP! A few comments:
> >>>>>>>>
> >>>>>>>> LM1. I strongly agree with Andrew's suggestion of moving this into a
> >>>>>>>> --verbose option. I expect someone would only attempt to make sense
> >> of
> >>>>>>> the
> >>>>>>>> epochs while debugging an issue, not while trying to view the group
> >> or
> >>>>>>>> member's info. (Also member-epoch makes more sense to me, even more
> >>>> if we
> >>>>>>>> end up agreeing in a --verbose). Related to both issues, my take is
> >>>> that
> >>>>>>>> whoever is close to the protocol will expect member-epoch, whoever
> >> is
> >>>> not
> >>>>>>>> will probably won't even need to see the epochs at all.
> >>>>>>>>
> >>>>>>>> LM2. Why are the epochs defined as Optional (don't we expect to
> >> always
> >>>>>>> have
> >>>>>>>> them? with 0 initially, defined on the broker side for the group
> >> ones,
> >>>>>>> and
> >>>>>>>> client side for the member)
> >>>>>>>>
> >>>>>>>> LM3. Why is the KIP including the –state option in the proposed
> >>>> changes?
> >>>>>>> (
> >>>>>>>>
> >>>>>>>
> >>>>
> >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
> >>>>>>>> ).
> >>>>>>>> I get that the output would change in that example, but it’s not
> >>>> because
> >>>>>>> of
> >>>>>>>> any change to the –state option itself. It's because of the change
> >>>>>>> proposed
> >>>>>>>> to the –described (required with the --state), and the changes to
> >>>>>>>> --describe are already explained above (seems confusing, got me
> >>>> looking
> >>>>>>> for
> >>>>>>>> a change to the state filter which seemed unrelated).
> >>>>>>>>
> >>>>>>>> LM4. Group epoch and target assignment epoch are conceptually at the
> >>>>>>> group
> >>>>>>>> level. vs member epoch that lands at a member level. So wonder if we
> >>>>>>> should
> >>>>>>>> show them accordingly (ex. using the --verbose option)
> >>>>>>>> –describe –verbose => shows group epoch and target assignment epoch
> >>>>>>>> –describe –members –verbose => shows member epoch (along
> >>>>>>>> with group epoch and target assignment epoch)
> >>>>>>>>
> >>>>>>>> Thanks!
> >>>>>>>>
> >>>>>>>> Lianet
> >>>>>>>>
> >>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield <
> >>>>>>>> andrew_schofield_jira@outlook.com> wrote:
> >>>>>>>>
> >>>>>>>>> Hi PoAn,
> >>>>>>>>> Thanks for the KIP. I have a few comments.
> >>>>>>>>>
> >>>>>>>>> AS1: It seems to me that these new additions are most useful to
> >>>> people
> >>>>>>>>> trying to understand
> >>>>>>>>> the progress of rebalancing in quite some detail. The information
> >> is
> >>>>>>>>> really not understandable
> >>>>>>>>> for most users who do not have deep knowledge of KIP-848/932.
> >>>>>>>>>
> >>>>>>>>> As a result, I suggest for kafka-share-groups.sh that you add a
> >>>>>>> --members
> >>>>>>>>> --verbose option
> >>>>>>>>> and only include the new information in the output for that option,
> >>>>>>> rather
> >>>>>>>>> than changing the
> >>>>>>>>> non-verbose --members output.
> >>>>>>>>>
> >>>>>>>>> I also make a similar suggestion for kafka-consumer-groups.sh
> >>>> --members
> >>>>>>>>> and only add the
> >>>>>>>>> new information for the --verbose output.
> >>>>>>>>>
> >>>>>>>>> AS2: I strongly suggest that you use MEMBER-EPOCH instead of
> >>>>>>>>> CONSUMER-EPOCH.
> >>>>>>>>> I think it's more confusing not following the terminology of the
> >>>> KIPs.
> >>>>>>> For
> >>>>>>>>> one thing,
> >>>>>>>>> we already have "member" in the admin client such as
> >>>> MemberDescription.
> >>>>>>>>> It's not a
> >>>>>>>>> new term.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Andrew
> >>>>>>>>> ________________________________________
> >>>>>>>>> From: PoAn Yang <pa...@apache.org>
> >>>>>>>>> Sent: 25 October 2024 13:55
> >>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
> >>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups
> >> command
> >>>>>>> line
> >>>>>>>>> tool to support new consumer group
> >>>>>>>>>
> >>>>>>>>> Hi Lucas,
> >>>>>>>>>
> >>>>>>>>> Thanks for the review!
> >>>>>>>>>
> >>>>>>>>> 1) Yes, I add related change for kafka-share-groups.sh to the KIP.
> >>>> Could
> >>>>>>>>> you take a look? Thanks for the suggestion.
> >>>>>>>>>
> >>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use MEMBER-EPOCH here,
> >>>> users
> >>>>>>> may
> >>>>>>>>> confuse what is different between CONSUMER and MEMBER.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> PoAn
> >>>>>>>>>
> >>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
> >>>>>>>>>> Hi Frank,
> >>>>>>>>>>
> >>>>>>>>>> thanks for the KIP!
> >>>>>>>>>>
> >>>>>>>>>> 1) For consistency, should we do the same for
> >>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription, etc. ? Even if we do
> >>>> not
> >>>>>>>>>> implement it right now if the share group implementation may still
> >>>> be
> >>>>>>>>>> incomplete, it may make sense to include it in the KIP.
> >>>>>>>>>>
> >>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH? That would seem
> >>>> more
> >>>>>>>>>> consistent.
> >>>>>>>>>>
> >>>>>>>>>> Cheers,
> >>>>>>>>>> Lucas
> >>>>>>>>>>
> >>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <ya...@gmail.com>
> >>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi all,
> >>>>>>>>>>>
> >>>>>>>>>>> I would like to kick off the discussion of KIP-1099. This KIP
> >>>> enhances
> >>>>>>>>> the
> >>>>>>>>>>> kafka-consumer-groups tools to include state which is introduced
> >> by
> >>>>>>>>> KIP-848.
> >>>>>>>>>>>
> >>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups command line tool to
> >> support
> >>>>>>> new
> >>>>>>>>>>> consumer group - Apache Kafka - Apache Software Foundation
> >>>>>>>>>>> <
> >>>>>>>>>
> >>>>>>>
> >>>>
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>
> >>>>>>>>>>> cwiki.apache.org
> >>>>>>>>>>> <
> >>>>>>>>>
> >>>>>>>
> >>>>
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>
> >>>>>>>>>>> [image: favicon.ico]
> >>>>>>>>>>> <
> >>>>>>>>>
> >>>>>>>
> >>>>
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>
> >>>>>>>>>>> <
> >>>>>>>>>
> >>>>>>>
> >>>>
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Thank you,
> >>>>>>>>>>> PoAn
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>
> >>>>
> >>
> >>
>
>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by PoAn Yang <ya...@gmail.com>.
Hi David,
Thanks for the reminder.
DJ4: Add related content to the KIP.
Bump validVersions in ConsumerGroupDescribeResponse to “0-1”.
Add a new field “Protocol” to ConsumerGroupDescribeResponse.
Thanks,
PoAn
> On Nov 19, 2024, at 3:57 PM, David Jacot <dj...@confluent.io.INVALID> wrote:
>
> Hi PoAn,
>
> DJ4: Could you please also include the required changes to the
> ConsumerGroupDescribed API in the public interface section? We basically
> need to bump the version, add the new field, etc.
>
> Thanks,
> David
>
> On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang <ya...@gmail.com> wrote:
>
>> Hi David,
>>
>> DJ4: Add a new field protocol to MemberDescription,
>> so the command line tool can show protocol information when users describe
>> members.
>>
>> If there is no further suggestion, I will start a vote thread today.
>>
>> Thanks,
>> PoAn
>>
>>> On Nov 15, 2024, at 11:50 PM, David Jacot <dj...@confluent.io.INVALID>
>> wrote:
>>>
>>> Hi,
>>>
>>> DJ2: Using "-" sounds good to me.
>>>
>>> DJ3: That seems reasonable to me.
>>>
>>> DJ4: Why not add it right now? I don't want to change the output of the
>>> tool too many times.
>>>
>>> Best,
>>> David
>>>
>>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang <ya...@gmail.com> wrote:
>>>
>>>> Hi David / Andrew,
>>>>
>>>> Thanks for review. Thanks Andrew for picking up kafka-share-groups.sh
>>>> implementation.
>>>> I will handle kafka-consumer-groups.sh.
>>>>
>>>> DJ3: After discussing with @Chia-Ping Tsai, we think that using new
>> format
>>>> is more clear.
>>>>
>>>> The new format will be like
>>>>
>> <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2>
>>>>
>>>> Using colon(:) to concat topic name and partition IDs.
>>>> Using comma(,) to concat partition IDs within same topic name.
>>>> Using semicolon(;) to concat topic strings.
>>>>
>>>> AS3: Fix it with kafka-share-groups.sh. Thanks.
>>>>
>>>> If there is no further suggestion, I will start a vote thread next
>> Monday.
>>>>
>>>> Thanks,
>>>> PoAn
>>>>
>>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield <
>>>> andrew_schofield_jira@outlook.com> wrote:
>>>>>
>>>>> Hi PoAn,
>>>>> DJ2: I was just going to comment that "-" would be a more appropriate
>>>> missing value, but
>>>>> you got there first.
>>>>>
>>>>> AS3: The examples for kafka-share-groups.sh include
>>>> kafka-consumer-groups.sh in the
>>>>> command line.
>>>>>
>>>>> If this is accepted in time, I'm happy to pick up the implementation of
>>>> the share groups
>>>>> part of this if it helps.
>>>>>
>>>>> Thanks,
>>>>> Andrew
>>>>>
>>>>> ________________________________________
>>>>> From: Frank Yang <ya...@gmail.com>
>>>>> Sent: 14 November 2024 10:48
>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command
>>>> line tool to support new consumer group
>>>>>
>>>>> Hi David,
>>>>>
>>>>> Thanks for the review and suggestion! I would like to get this in AK
>> 4.0
>>>> as well. I will do my best.
>>>>>
>>>>> DJ1: Update KIP to put GROUP-EPOCH and TARGET-ASSIGNMENT-EPOCH before
>>>> #MEMBERS.
>>>>>
>>>>> DJ2: I prefer to follow current missing column value “-“. (reference <
>>>>
>> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92
>>>>> )
>>>>>
>>>>> DJ3: Update KIP to use CURRENT-EPOCH CURRENT-ASSIGNMENT TARGET-EPOCH
>>>> TARGET-ASSIGNMENT.
>>>>> Remove GROUP-EPOCH.
>>>>>
>>>>> For assignment value, it follows current output (reference <
>>>>
>> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418
>>> ).
>>>> I think the form of `topicid-partitionid` is more clear.
>>>>> If we would like to use this form, I will update both output in
>>>> kafka-consumer-groups.sh and kafka-share-groups.sh.
>>>>>
>>>>> DJ4: It looks like DescribeGroupsResponseData only has protocol type at
>>>> group level.
>>>>> Both DescribeGroupsResponseData and ConsumerGroupDescribeResponseData
>>>> don’t have protocol at member level.
>>>>> Could we use a followup to add it?
>>>>>
>>>>> DJ5: Update KIP to put LEADER-EPOCH before CURRENT-OFFSET.
>>>>>
>>>>> Thanks,
>>>>> PoAn
>>>>>
>>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot <djacot@confluent.io.INVALID
>>>
>>>> wrote:
>>>>>>
>>>>>> Hi PoAn,
>>>>>>
>>>>>> Thanks for the KIP! I have a few minor comments/suggestions:
>>>>>>
>>>>>> DJ1: In the output of `--describe --verbose`, I would suggest putting
>>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before `#MEMBERS`.
>>>>>> DJ2: Continuing on the above, I assume that we will print out N/A for
>>>> the
>>>>>> fields not supported by classic groups. Is this correct?
>>>>>> DJ3: In the output of `--describe --members --verbose`, I wonder if we
>>>>>> should use the following order and terms: CURRENT-EPOCH
>>>> CURRENT-ASSIGNMENT
>>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the GROUP-EPOCH
>> because
>>>> it
>>>>>> is already in the group description. The value `(0)` used for the
>>>>>> assignment is incorrect. Here I suppose that we will print out the
>> list
>>>> of
>>>>>> partitions in the form of `topicid-partitionid`.
>>>>>> DJ4: Continuing on the above, I wonder if we should also add the
>>>> protocol
>>>>>> used `classic` or `consumer`. For context, it is possible to have
>>>> `classic`
>>>>>> members and `consumer` members in a `consumer` group during an online
>>>>>> upgrade from the classic protocol to the consumer protocol. Having
>> this
>>>>>> information may be handy for administrators. What do you think?
>>>>>> DJ5: In the output of `--describe --offsets --verbose`, I would
>> suggest
>>>>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`.
>>>>>>
>>>>>> It would be great if we could get this one in AK 4.0 as it changes the
>>>>>> output of the command.
>>>>>>
>>>>>> Thanks,
>>>>>> DJ
>>>>>>
>>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <ya...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Sean / Andrew / Lianet,
>>>>>>>
>>>>>>> Thanks for all your review and suggestions.
>>>>>>>
>>>>>>> AS1, LM1, LM4: Change to add KIP-848 information when users give
>>>> —verbose
>>>>>>> option.
>>>>>>> —describe —verbose: shows group epoch and target assignment epoch.
>>>>>>> —describe —members —verbose: shows above information, member epoch,
>> and
>>>>>>> target assignment.
>>>>>>>
>>>>>>> AS2: Change to use MEMBER-EPOCH to align with KIP-848 definition.
>>>>>>>
>>>>>>> LM2: For classic group, it doesn’t have epoch, so I use Optional
>>>> fields in
>>>>>>> ConsumerGroupDescription.
>>>>>>> For share group, it relies on KIP-848. It must have epoch, so I use
>> int
>>>>>>> fields in ShareGroupDescription.
>>>>>>>
>>>>>>> LM3: Remove —state change. We can get group level information by
>>>> —describe
>>>>>>> —verbose.
>>>>>>>
>>>>>>> SQ1: Add LEADER-EPOCH when users give —describe —offsets —verbose.
>>>>>>>
>>>>>>> Thanks.
>>>>>>> PoAn
>>>>>>>
>>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <li...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Hello Frank, thanks for the KIP! A few comments:
>>>>>>>>
>>>>>>>> LM1. I strongly agree with Andrew's suggestion of moving this into a
>>>>>>>> --verbose option. I expect someone would only attempt to make sense
>> of
>>>>>>> the
>>>>>>>> epochs while debugging an issue, not while trying to view the group
>> or
>>>>>>>> member's info. (Also member-epoch makes more sense to me, even more
>>>> if we
>>>>>>>> end up agreeing in a --verbose). Related to both issues, my take is
>>>> that
>>>>>>>> whoever is close to the protocol will expect member-epoch, whoever
>> is
>>>> not
>>>>>>>> will probably won't even need to see the epochs at all.
>>>>>>>>
>>>>>>>> LM2. Why are the epochs defined as Optional (don't we expect to
>> always
>>>>>>> have
>>>>>>>> them? with 0 initially, defined on the broker side for the group
>> ones,
>>>>>>> and
>>>>>>>> client side for the member)
>>>>>>>>
>>>>>>>> LM3. Why is the KIP including the –state option in the proposed
>>>> changes?
>>>>>>> (
>>>>>>>>
>>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
>>>>>>>> ).
>>>>>>>> I get that the output would change in that example, but it’s not
>>>> because
>>>>>>> of
>>>>>>>> any change to the –state option itself. It's because of the change
>>>>>>> proposed
>>>>>>>> to the –described (required with the --state), and the changes to
>>>>>>>> --describe are already explained above (seems confusing, got me
>>>> looking
>>>>>>> for
>>>>>>>> a change to the state filter which seemed unrelated).
>>>>>>>>
>>>>>>>> LM4. Group epoch and target assignment epoch are conceptually at the
>>>>>>> group
>>>>>>>> level. vs member epoch that lands at a member level. So wonder if we
>>>>>>> should
>>>>>>>> show them accordingly (ex. using the --verbose option)
>>>>>>>> –describe –verbose => shows group epoch and target assignment epoch
>>>>>>>> –describe –members –verbose => shows member epoch (along
>>>>>>>> with group epoch and target assignment epoch)
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> Lianet
>>>>>>>>
>>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield <
>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
>>>>>>>>
>>>>>>>>> Hi PoAn,
>>>>>>>>> Thanks for the KIP. I have a few comments.
>>>>>>>>>
>>>>>>>>> AS1: It seems to me that these new additions are most useful to
>>>> people
>>>>>>>>> trying to understand
>>>>>>>>> the progress of rebalancing in quite some detail. The information
>> is
>>>>>>>>> really not understandable
>>>>>>>>> for most users who do not have deep knowledge of KIP-848/932.
>>>>>>>>>
>>>>>>>>> As a result, I suggest for kafka-share-groups.sh that you add a
>>>>>>> --members
>>>>>>>>> --verbose option
>>>>>>>>> and only include the new information in the output for that option,
>>>>>>> rather
>>>>>>>>> than changing the
>>>>>>>>> non-verbose --members output.
>>>>>>>>>
>>>>>>>>> I also make a similar suggestion for kafka-consumer-groups.sh
>>>> --members
>>>>>>>>> and only add the
>>>>>>>>> new information for the --verbose output.
>>>>>>>>>
>>>>>>>>> AS2: I strongly suggest that you use MEMBER-EPOCH instead of
>>>>>>>>> CONSUMER-EPOCH.
>>>>>>>>> I think it's more confusing not following the terminology of the
>>>> KIPs.
>>>>>>> For
>>>>>>>>> one thing,
>>>>>>>>> we already have "member" in the admin client such as
>>>> MemberDescription.
>>>>>>>>> It's not a
>>>>>>>>> new term.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Andrew
>>>>>>>>> ________________________________________
>>>>>>>>> From: PoAn Yang <pa...@apache.org>
>>>>>>>>> Sent: 25 October 2024 13:55
>>>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups
>> command
>>>>>>> line
>>>>>>>>> tool to support new consumer group
>>>>>>>>>
>>>>>>>>> Hi Lucas,
>>>>>>>>>
>>>>>>>>> Thanks for the review!
>>>>>>>>>
>>>>>>>>> 1) Yes, I add related change for kafka-share-groups.sh to the KIP.
>>>> Could
>>>>>>>>> you take a look? Thanks for the suggestion.
>>>>>>>>>
>>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use MEMBER-EPOCH here,
>>>> users
>>>>>>> may
>>>>>>>>> confuse what is different between CONSUMER and MEMBER.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> PoAn
>>>>>>>>>
>>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
>>>>>>>>>> Hi Frank,
>>>>>>>>>>
>>>>>>>>>> thanks for the KIP!
>>>>>>>>>>
>>>>>>>>>> 1) For consistency, should we do the same for
>>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription, etc. ? Even if we do
>>>> not
>>>>>>>>>> implement it right now if the share group implementation may still
>>>> be
>>>>>>>>>> incomplete, it may make sense to include it in the KIP.
>>>>>>>>>>
>>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH? That would seem
>>>> more
>>>>>>>>>> consistent.
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Lucas
>>>>>>>>>>
>>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <ya...@gmail.com>
>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi all,
>>>>>>>>>>>
>>>>>>>>>>> I would like to kick off the discussion of KIP-1099. This KIP
>>>> enhances
>>>>>>>>> the
>>>>>>>>>>> kafka-consumer-groups tools to include state which is introduced
>> by
>>>>>>>>> KIP-848.
>>>>>>>>>>>
>>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups command line tool to
>> support
>>>>>>> new
>>>>>>>>>>> consumer group - Apache Kafka - Apache Software Foundation
>>>>>>>>>>> <
>>>>>>>>>
>>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>
>>>>>>>>>>> cwiki.apache.org
>>>>>>>>>>> <
>>>>>>>>>
>>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>
>>>>>>>>>>> [image: favicon.ico]
>>>>>>>>>>> <
>>>>>>>>>
>>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>
>>>>>>>>>>> <
>>>>>>>>>
>>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thank you,
>>>>>>>>>>> PoAn
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>
>>>>
>>
>>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by David Jacot <dj...@confluent.io.INVALID>.
Hi PoAn,
DJ4: Could you please also include the required changes to the
ConsumerGroupDescribed API in the public interface section? We basically
need to bump the version, add the new field, etc.
Thanks,
David
On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang <ya...@gmail.com> wrote:
> Hi David,
>
> DJ4: Add a new field protocol to MemberDescription,
> so the command line tool can show protocol information when users describe
> members.
>
> If there is no further suggestion, I will start a vote thread today.
>
> Thanks,
> PoAn
>
> > On Nov 15, 2024, at 11:50 PM, David Jacot <dj...@confluent.io.INVALID>
> wrote:
> >
> > Hi,
> >
> > DJ2: Using "-" sounds good to me.
> >
> > DJ3: That seems reasonable to me.
> >
> > DJ4: Why not add it right now? I don't want to change the output of the
> > tool too many times.
> >
> > Best,
> > David
> >
> > On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang <ya...@gmail.com> wrote:
> >
> >> Hi David / Andrew,
> >>
> >> Thanks for review. Thanks Andrew for picking up kafka-share-groups.sh
> >> implementation.
> >> I will handle kafka-consumer-groups.sh.
> >>
> >> DJ3: After discussing with @Chia-Ping Tsai, we think that using new
> format
> >> is more clear.
> >>
> >> The new format will be like
> >>
> <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2>
> >>
> >> Using colon(:) to concat topic name and partition IDs.
> >> Using comma(,) to concat partition IDs within same topic name.
> >> Using semicolon(;) to concat topic strings.
> >>
> >> AS3: Fix it with kafka-share-groups.sh. Thanks.
> >>
> >> If there is no further suggestion, I will start a vote thread next
> Monday.
> >>
> >> Thanks,
> >> PoAn
> >>
> >>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield <
> >> andrew_schofield_jira@outlook.com> wrote:
> >>>
> >>> Hi PoAn,
> >>> DJ2: I was just going to comment that "-" would be a more appropriate
> >> missing value, but
> >>> you got there first.
> >>>
> >>> AS3: The examples for kafka-share-groups.sh include
> >> kafka-consumer-groups.sh in the
> >>> command line.
> >>>
> >>> If this is accepted in time, I'm happy to pick up the implementation of
> >> the share groups
> >>> part of this if it helps.
> >>>
> >>> Thanks,
> >>> Andrew
> >>>
> >>> ________________________________________
> >>> From: Frank Yang <ya...@gmail.com>
> >>> Sent: 14 November 2024 10:48
> >>> To: dev@kafka.apache.org <de...@kafka.apache.org>
> >>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command
> >> line tool to support new consumer group
> >>>
> >>> Hi David,
> >>>
> >>> Thanks for the review and suggestion! I would like to get this in AK
> 4.0
> >> as well. I will do my best.
> >>>
> >>> DJ1: Update KIP to put GROUP-EPOCH and TARGET-ASSIGNMENT-EPOCH before
> >> #MEMBERS.
> >>>
> >>> DJ2: I prefer to follow current missing column value “-“. (reference <
> >>
> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92
> >>> )
> >>>
> >>> DJ3: Update KIP to use CURRENT-EPOCH CURRENT-ASSIGNMENT TARGET-EPOCH
> >> TARGET-ASSIGNMENT.
> >>> Remove GROUP-EPOCH.
> >>>
> >>> For assignment value, it follows current output (reference <
> >>
> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418
> >).
> >> I think the form of `topicid-partitionid` is more clear.
> >>> If we would like to use this form, I will update both output in
> >> kafka-consumer-groups.sh and kafka-share-groups.sh.
> >>>
> >>> DJ4: It looks like DescribeGroupsResponseData only has protocol type at
> >> group level.
> >>> Both DescribeGroupsResponseData and ConsumerGroupDescribeResponseData
> >> don’t have protocol at member level.
> >>> Could we use a followup to add it?
> >>>
> >>> DJ5: Update KIP to put LEADER-EPOCH before CURRENT-OFFSET.
> >>>
> >>> Thanks,
> >>> PoAn
> >>>
> >>>> On Nov 14, 2024, at 3:43 PM, David Jacot <djacot@confluent.io.INVALID
> >
> >> wrote:
> >>>>
> >>>> Hi PoAn,
> >>>>
> >>>> Thanks for the KIP! I have a few minor comments/suggestions:
> >>>>
> >>>> DJ1: In the output of `--describe --verbose`, I would suggest putting
> >>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before `#MEMBERS`.
> >>>> DJ2: Continuing on the above, I assume that we will print out N/A for
> >> the
> >>>> fields not supported by classic groups. Is this correct?
> >>>> DJ3: In the output of `--describe --members --verbose`, I wonder if we
> >>>> should use the following order and terms: CURRENT-EPOCH
> >> CURRENT-ASSIGNMENT
> >>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the GROUP-EPOCH
> because
> >> it
> >>>> is already in the group description. The value `(0)` used for the
> >>>> assignment is incorrect. Here I suppose that we will print out the
> list
> >> of
> >>>> partitions in the form of `topicid-partitionid`.
> >>>> DJ4: Continuing on the above, I wonder if we should also add the
> >> protocol
> >>>> used `classic` or `consumer`. For context, it is possible to have
> >> `classic`
> >>>> members and `consumer` members in a `consumer` group during an online
> >>>> upgrade from the classic protocol to the consumer protocol. Having
> this
> >>>> information may be handy for administrators. What do you think?
> >>>> DJ5: In the output of `--describe --offsets --verbose`, I would
> suggest
> >>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`.
> >>>>
> >>>> It would be great if we could get this one in AK 4.0 as it changes the
> >>>> output of the command.
> >>>>
> >>>> Thanks,
> >>>> DJ
> >>>>
> >>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <ya...@gmail.com> wrote:
> >>>>
> >>>>> Hi Sean / Andrew / Lianet,
> >>>>>
> >>>>> Thanks for all your review and suggestions.
> >>>>>
> >>>>> AS1, LM1, LM4: Change to add KIP-848 information when users give
> >> —verbose
> >>>>> option.
> >>>>> —describe —verbose: shows group epoch and target assignment epoch.
> >>>>> —describe —members —verbose: shows above information, member epoch,
> and
> >>>>> target assignment.
> >>>>>
> >>>>> AS2: Change to use MEMBER-EPOCH to align with KIP-848 definition.
> >>>>>
> >>>>> LM2: For classic group, it doesn’t have epoch, so I use Optional
> >> fields in
> >>>>> ConsumerGroupDescription.
> >>>>> For share group, it relies on KIP-848. It must have epoch, so I use
> int
> >>>>> fields in ShareGroupDescription.
> >>>>>
> >>>>> LM3: Remove —state change. We can get group level information by
> >> —describe
> >>>>> —verbose.
> >>>>>
> >>>>> SQ1: Add LEADER-EPOCH when users give —describe —offsets —verbose.
> >>>>>
> >>>>> Thanks.
> >>>>> PoAn
> >>>>>
> >>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <li...@gmail.com> wrote:
> >>>>>>
> >>>>>> Hello Frank, thanks for the KIP! A few comments:
> >>>>>>
> >>>>>> LM1. I strongly agree with Andrew's suggestion of moving this into a
> >>>>>> --verbose option. I expect someone would only attempt to make sense
> of
> >>>>> the
> >>>>>> epochs while debugging an issue, not while trying to view the group
> or
> >>>>>> member's info. (Also member-epoch makes more sense to me, even more
> >> if we
> >>>>>> end up agreeing in a --verbose). Related to both issues, my take is
> >> that
> >>>>>> whoever is close to the protocol will expect member-epoch, whoever
> is
> >> not
> >>>>>> will probably won't even need to see the epochs at all.
> >>>>>>
> >>>>>> LM2. Why are the epochs defined as Optional (don't we expect to
> always
> >>>>> have
> >>>>>> them? with 0 initially, defined on the broker side for the group
> ones,
> >>>>> and
> >>>>>> client side for the member)
> >>>>>>
> >>>>>> LM3. Why is the KIP including the –state option in the proposed
> >> changes?
> >>>>> (
> >>>>>>
> >>>>>
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
> >>>>>> ).
> >>>>>> I get that the output would change in that example, but it’s not
> >> because
> >>>>> of
> >>>>>> any change to the –state option itself. It's because of the change
> >>>>> proposed
> >>>>>> to the –described (required with the --state), and the changes to
> >>>>>> --describe are already explained above (seems confusing, got me
> >> looking
> >>>>> for
> >>>>>> a change to the state filter which seemed unrelated).
> >>>>>>
> >>>>>> LM4. Group epoch and target assignment epoch are conceptually at the
> >>>>> group
> >>>>>> level. vs member epoch that lands at a member level. So wonder if we
> >>>>> should
> >>>>>> show them accordingly (ex. using the --verbose option)
> >>>>>> –describe –verbose => shows group epoch and target assignment epoch
> >>>>>> –describe –members –verbose => shows member epoch (along
> >>>>>> with group epoch and target assignment epoch)
> >>>>>>
> >>>>>> Thanks!
> >>>>>>
> >>>>>> Lianet
> >>>>>>
> >>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield <
> >>>>>> andrew_schofield_jira@outlook.com> wrote:
> >>>>>>
> >>>>>>> Hi PoAn,
> >>>>>>> Thanks for the KIP. I have a few comments.
> >>>>>>>
> >>>>>>> AS1: It seems to me that these new additions are most useful to
> >> people
> >>>>>>> trying to understand
> >>>>>>> the progress of rebalancing in quite some detail. The information
> is
> >>>>>>> really not understandable
> >>>>>>> for most users who do not have deep knowledge of KIP-848/932.
> >>>>>>>
> >>>>>>> As a result, I suggest for kafka-share-groups.sh that you add a
> >>>>> --members
> >>>>>>> --verbose option
> >>>>>>> and only include the new information in the output for that option,
> >>>>> rather
> >>>>>>> than changing the
> >>>>>>> non-verbose --members output.
> >>>>>>>
> >>>>>>> I also make a similar suggestion for kafka-consumer-groups.sh
> >> --members
> >>>>>>> and only add the
> >>>>>>> new information for the --verbose output.
> >>>>>>>
> >>>>>>> AS2: I strongly suggest that you use MEMBER-EPOCH instead of
> >>>>>>> CONSUMER-EPOCH.
> >>>>>>> I think it's more confusing not following the terminology of the
> >> KIPs.
> >>>>> For
> >>>>>>> one thing,
> >>>>>>> we already have "member" in the admin client such as
> >> MemberDescription.
> >>>>>>> It's not a
> >>>>>>> new term.
> >>>>>>>
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Andrew
> >>>>>>> ________________________________________
> >>>>>>> From: PoAn Yang <pa...@apache.org>
> >>>>>>> Sent: 25 October 2024 13:55
> >>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
> >>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups
> command
> >>>>> line
> >>>>>>> tool to support new consumer group
> >>>>>>>
> >>>>>>> Hi Lucas,
> >>>>>>>
> >>>>>>> Thanks for the review!
> >>>>>>>
> >>>>>>> 1) Yes, I add related change for kafka-share-groups.sh to the KIP.
> >> Could
> >>>>>>> you take a look? Thanks for the suggestion.
> >>>>>>>
> >>>>>>> 2) We use CONSUMER-ID as member ID. If we use MEMBER-EPOCH here,
> >> users
> >>>>> may
> >>>>>>> confuse what is different between CONSUMER and MEMBER.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> PoAn
> >>>>>>>
> >>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
> >>>>>>>> Hi Frank,
> >>>>>>>>
> >>>>>>>> thanks for the KIP!
> >>>>>>>>
> >>>>>>>> 1) For consistency, should we do the same for
> >>>>>>>> kafka-share-groups.sh, ShareGroupDescription, etc. ? Even if we do
> >> not
> >>>>>>>> implement it right now if the share group implementation may still
> >> be
> >>>>>>>> incomplete, it may make sense to include it in the KIP.
> >>>>>>>>
> >>>>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH? That would seem
> >> more
> >>>>>>>> consistent.
> >>>>>>>>
> >>>>>>>> Cheers,
> >>>>>>>> Lucas
> >>>>>>>>
> >>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <ya...@gmail.com>
> >> wrote:
> >>>>>>>>
> >>>>>>>>> Hi all,
> >>>>>>>>>
> >>>>>>>>> I would like to kick off the discussion of KIP-1099. This KIP
> >> enhances
> >>>>>>> the
> >>>>>>>>> kafka-consumer-groups tools to include state which is introduced
> by
> >>>>>>> KIP-848.
> >>>>>>>>>
> >>>>>>>>> KIP-1099: Extend kafka-consumer-groups command line tool to
> support
> >>>>> new
> >>>>>>>>> consumer group - Apache Kafka - Apache Software Foundation
> >>>>>>>>> <
> >>>>>>>
> >>>>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>
> >>>>>>>>> cwiki.apache.org
> >>>>>>>>> <
> >>>>>>>
> >>>>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>
> >>>>>>>>> [image: favicon.ico]
> >>>>>>>>> <
> >>>>>>>
> >>>>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>
> >>>>>>>>> <
> >>>>>>>
> >>>>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Thank you,
> >>>>>>>>> PoAn
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>
> >>
>
>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by PoAn Yang <ya...@gmail.com>.
Hi David,
DJ4: Add a new field protocol to MemberDescription,
so the command line tool can show protocol information when users describe members.
If there is no further suggestion, I will start a vote thread today.
Thanks,
PoAn
> On Nov 15, 2024, at 11:50 PM, David Jacot <dj...@confluent.io.INVALID> wrote:
>
> Hi,
>
> DJ2: Using "-" sounds good to me.
>
> DJ3: That seems reasonable to me.
>
> DJ4: Why not add it right now? I don't want to change the output of the
> tool too many times.
>
> Best,
> David
>
> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang <ya...@gmail.com> wrote:
>
>> Hi David / Andrew,
>>
>> Thanks for review. Thanks Andrew for picking up kafka-share-groups.sh
>> implementation.
>> I will handle kafka-consumer-groups.sh.
>>
>> DJ3: After discussing with @Chia-Ping Tsai, we think that using new format
>> is more clear.
>>
>> The new format will be like
>> <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2>
>>
>> Using colon(:) to concat topic name and partition IDs.
>> Using comma(,) to concat partition IDs within same topic name.
>> Using semicolon(;) to concat topic strings.
>>
>> AS3: Fix it with kafka-share-groups.sh. Thanks.
>>
>> If there is no further suggestion, I will start a vote thread next Monday.
>>
>> Thanks,
>> PoAn
>>
>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield <
>> andrew_schofield_jira@outlook.com> wrote:
>>>
>>> Hi PoAn,
>>> DJ2: I was just going to comment that "-" would be a more appropriate
>> missing value, but
>>> you got there first.
>>>
>>> AS3: The examples for kafka-share-groups.sh include
>> kafka-consumer-groups.sh in the
>>> command line.
>>>
>>> If this is accepted in time, I'm happy to pick up the implementation of
>> the share groups
>>> part of this if it helps.
>>>
>>> Thanks,
>>> Andrew
>>>
>>> ________________________________________
>>> From: Frank Yang <ya...@gmail.com>
>>> Sent: 14 November 2024 10:48
>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
>>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command
>> line tool to support new consumer group
>>>
>>> Hi David,
>>>
>>> Thanks for the review and suggestion! I would like to get this in AK 4.0
>> as well. I will do my best.
>>>
>>> DJ1: Update KIP to put GROUP-EPOCH and TARGET-ASSIGNMENT-EPOCH before
>> #MEMBERS.
>>>
>>> DJ2: I prefer to follow current missing column value “-“. (reference <
>> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92
>>> )
>>>
>>> DJ3: Update KIP to use CURRENT-EPOCH CURRENT-ASSIGNMENT TARGET-EPOCH
>> TARGET-ASSIGNMENT.
>>> Remove GROUP-EPOCH.
>>>
>>> For assignment value, it follows current output (reference <
>> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418>).
>> I think the form of `topicid-partitionid` is more clear.
>>> If we would like to use this form, I will update both output in
>> kafka-consumer-groups.sh and kafka-share-groups.sh.
>>>
>>> DJ4: It looks like DescribeGroupsResponseData only has protocol type at
>> group level.
>>> Both DescribeGroupsResponseData and ConsumerGroupDescribeResponseData
>> don’t have protocol at member level.
>>> Could we use a followup to add it?
>>>
>>> DJ5: Update KIP to put LEADER-EPOCH before CURRENT-OFFSET.
>>>
>>> Thanks,
>>> PoAn
>>>
>>>> On Nov 14, 2024, at 3:43 PM, David Jacot <dj...@confluent.io.INVALID>
>> wrote:
>>>>
>>>> Hi PoAn,
>>>>
>>>> Thanks for the KIP! I have a few minor comments/suggestions:
>>>>
>>>> DJ1: In the output of `--describe --verbose`, I would suggest putting
>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before `#MEMBERS`.
>>>> DJ2: Continuing on the above, I assume that we will print out N/A for
>> the
>>>> fields not supported by classic groups. Is this correct?
>>>> DJ3: In the output of `--describe --members --verbose`, I wonder if we
>>>> should use the following order and terms: CURRENT-EPOCH
>> CURRENT-ASSIGNMENT
>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the GROUP-EPOCH because
>> it
>>>> is already in the group description. The value `(0)` used for the
>>>> assignment is incorrect. Here I suppose that we will print out the list
>> of
>>>> partitions in the form of `topicid-partitionid`.
>>>> DJ4: Continuing on the above, I wonder if we should also add the
>> protocol
>>>> used `classic` or `consumer`. For context, it is possible to have
>> `classic`
>>>> members and `consumer` members in a `consumer` group during an online
>>>> upgrade from the classic protocol to the consumer protocol. Having this
>>>> information may be handy for administrators. What do you think?
>>>> DJ5: In the output of `--describe --offsets --verbose`, I would suggest
>>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`.
>>>>
>>>> It would be great if we could get this one in AK 4.0 as it changes the
>>>> output of the command.
>>>>
>>>> Thanks,
>>>> DJ
>>>>
>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <ya...@gmail.com> wrote:
>>>>
>>>>> Hi Sean / Andrew / Lianet,
>>>>>
>>>>> Thanks for all your review and suggestions.
>>>>>
>>>>> AS1, LM1, LM4: Change to add KIP-848 information when users give
>> —verbose
>>>>> option.
>>>>> —describe —verbose: shows group epoch and target assignment epoch.
>>>>> —describe —members —verbose: shows above information, member epoch, and
>>>>> target assignment.
>>>>>
>>>>> AS2: Change to use MEMBER-EPOCH to align with KIP-848 definition.
>>>>>
>>>>> LM2: For classic group, it doesn’t have epoch, so I use Optional
>> fields in
>>>>> ConsumerGroupDescription.
>>>>> For share group, it relies on KIP-848. It must have epoch, so I use int
>>>>> fields in ShareGroupDescription.
>>>>>
>>>>> LM3: Remove —state change. We can get group level information by
>> —describe
>>>>> —verbose.
>>>>>
>>>>> SQ1: Add LEADER-EPOCH when users give —describe —offsets —verbose.
>>>>>
>>>>> Thanks.
>>>>> PoAn
>>>>>
>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <li...@gmail.com> wrote:
>>>>>>
>>>>>> Hello Frank, thanks for the KIP! A few comments:
>>>>>>
>>>>>> LM1. I strongly agree with Andrew's suggestion of moving this into a
>>>>>> --verbose option. I expect someone would only attempt to make sense of
>>>>> the
>>>>>> epochs while debugging an issue, not while trying to view the group or
>>>>>> member's info. (Also member-epoch makes more sense to me, even more
>> if we
>>>>>> end up agreeing in a --verbose). Related to both issues, my take is
>> that
>>>>>> whoever is close to the protocol will expect member-epoch, whoever is
>> not
>>>>>> will probably won't even need to see the epochs at all.
>>>>>>
>>>>>> LM2. Why are the epochs defined as Optional (don't we expect to always
>>>>> have
>>>>>> them? with 0 initially, defined on the broker side for the group ones,
>>>>> and
>>>>>> client side for the member)
>>>>>>
>>>>>> LM3. Why is the KIP including the –state option in the proposed
>> changes?
>>>>> (
>>>>>>
>>>>>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
>>>>>> ).
>>>>>> I get that the output would change in that example, but it’s not
>> because
>>>>> of
>>>>>> any change to the –state option itself. It's because of the change
>>>>> proposed
>>>>>> to the –described (required with the --state), and the changes to
>>>>>> --describe are already explained above (seems confusing, got me
>> looking
>>>>> for
>>>>>> a change to the state filter which seemed unrelated).
>>>>>>
>>>>>> LM4. Group epoch and target assignment epoch are conceptually at the
>>>>> group
>>>>>> level. vs member epoch that lands at a member level. So wonder if we
>>>>> should
>>>>>> show them accordingly (ex. using the --verbose option)
>>>>>> –describe –verbose => shows group epoch and target assignment epoch
>>>>>> –describe –members –verbose => shows member epoch (along
>>>>>> with group epoch and target assignment epoch)
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> Lianet
>>>>>>
>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield <
>>>>>> andrew_schofield_jira@outlook.com> wrote:
>>>>>>
>>>>>>> Hi PoAn,
>>>>>>> Thanks for the KIP. I have a few comments.
>>>>>>>
>>>>>>> AS1: It seems to me that these new additions are most useful to
>> people
>>>>>>> trying to understand
>>>>>>> the progress of rebalancing in quite some detail. The information is
>>>>>>> really not understandable
>>>>>>> for most users who do not have deep knowledge of KIP-848/932.
>>>>>>>
>>>>>>> As a result, I suggest for kafka-share-groups.sh that you add a
>>>>> --members
>>>>>>> --verbose option
>>>>>>> and only include the new information in the output for that option,
>>>>> rather
>>>>>>> than changing the
>>>>>>> non-verbose --members output.
>>>>>>>
>>>>>>> I also make a similar suggestion for kafka-consumer-groups.sh
>> --members
>>>>>>> and only add the
>>>>>>> new information for the --verbose output.
>>>>>>>
>>>>>>> AS2: I strongly suggest that you use MEMBER-EPOCH instead of
>>>>>>> CONSUMER-EPOCH.
>>>>>>> I think it's more confusing not following the terminology of the
>> KIPs.
>>>>> For
>>>>>>> one thing,
>>>>>>> we already have "member" in the admin client such as
>> MemberDescription.
>>>>>>> It's not a
>>>>>>> new term.
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Andrew
>>>>>>> ________________________________________
>>>>>>> From: PoAn Yang <pa...@apache.org>
>>>>>>> Sent: 25 October 2024 13:55
>>>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command
>>>>> line
>>>>>>> tool to support new consumer group
>>>>>>>
>>>>>>> Hi Lucas,
>>>>>>>
>>>>>>> Thanks for the review!
>>>>>>>
>>>>>>> 1) Yes, I add related change for kafka-share-groups.sh to the KIP.
>> Could
>>>>>>> you take a look? Thanks for the suggestion.
>>>>>>>
>>>>>>> 2) We use CONSUMER-ID as member ID. If we use MEMBER-EPOCH here,
>> users
>>>>> may
>>>>>>> confuse what is different between CONSUMER and MEMBER.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> PoAn
>>>>>>>
>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
>>>>>>>> Hi Frank,
>>>>>>>>
>>>>>>>> thanks for the KIP!
>>>>>>>>
>>>>>>>> 1) For consistency, should we do the same for
>>>>>>>> kafka-share-groups.sh, ShareGroupDescription, etc. ? Even if we do
>> not
>>>>>>>> implement it right now if the share group implementation may still
>> be
>>>>>>>> incomplete, it may make sense to include it in the KIP.
>>>>>>>>
>>>>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH? That would seem
>> more
>>>>>>>> consistent.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Lucas
>>>>>>>>
>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <ya...@gmail.com>
>> wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> I would like to kick off the discussion of KIP-1099. This KIP
>> enhances
>>>>>>> the
>>>>>>>>> kafka-consumer-groups tools to include state which is introduced by
>>>>>>> KIP-848.
>>>>>>>>>
>>>>>>>>> KIP-1099: Extend kafka-consumer-groups command line tool to support
>>>>> new
>>>>>>>>> consumer group - Apache Kafka - Apache Software Foundation
>>>>>>>>> <
>>>>>>>
>>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>
>>>>>>>>> cwiki.apache.org
>>>>>>>>> <
>>>>>>>
>>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>
>>>>>>>>> [image: favicon.ico]
>>>>>>>>> <
>>>>>>>
>>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>
>>>>>>>>> <
>>>>>>>
>>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thank you,
>>>>>>>>> PoAn
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>
>>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by David Jacot <dj...@confluent.io.INVALID>.
Hi,
DJ2: Using "-" sounds good to me.
DJ3: That seems reasonable to me.
DJ4: Why not add it right now? I don't want to change the output of the
tool too many times.
Best,
David
On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang <ya...@gmail.com> wrote:
> Hi David / Andrew,
>
> Thanks for review. Thanks Andrew for picking up kafka-share-groups.sh
> implementation.
> I will handle kafka-consumer-groups.sh.
>
> DJ3: After discussing with @Chia-Ping Tsai, we think that using new format
> is more clear.
>
> The new format will be like
> <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2>
>
> Using colon(:) to concat topic name and partition IDs.
> Using comma(,) to concat partition IDs within same topic name.
> Using semicolon(;) to concat topic strings.
>
> AS3: Fix it with kafka-share-groups.sh. Thanks.
>
> If there is no further suggestion, I will start a vote thread next Monday.
>
> Thanks,
> PoAn
>
> > On Nov 14, 2024, at 9:05 PM, Andrew Schofield <
> andrew_schofield_jira@outlook.com> wrote:
> >
> > Hi PoAn,
> > DJ2: I was just going to comment that "-" would be a more appropriate
> missing value, but
> > you got there first.
> >
> > AS3: The examples for kafka-share-groups.sh include
> kafka-consumer-groups.sh in the
> > command line.
> >
> > If this is accepted in time, I'm happy to pick up the implementation of
> the share groups
> > part of this if it helps.
> >
> > Thanks,
> > Andrew
> >
> > ________________________________________
> > From: Frank Yang <ya...@gmail.com>
> > Sent: 14 November 2024 10:48
> > To: dev@kafka.apache.org <de...@kafka.apache.org>
> > Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command
> line tool to support new consumer group
> >
> > Hi David,
> >
> > Thanks for the review and suggestion! I would like to get this in AK 4.0
> as well. I will do my best.
> >
> > DJ1: Update KIP to put GROUP-EPOCH and TARGET-ASSIGNMENT-EPOCH before
> #MEMBERS.
> >
> > DJ2: I prefer to follow current missing column value “-“. (reference <
> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92
> >)
> >
> > DJ3: Update KIP to use CURRENT-EPOCH CURRENT-ASSIGNMENT TARGET-EPOCH
> TARGET-ASSIGNMENT.
> > Remove GROUP-EPOCH.
> >
> > For assignment value, it follows current output (reference <
> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418>).
> I think the form of `topicid-partitionid` is more clear.
> > If we would like to use this form, I will update both output in
> kafka-consumer-groups.sh and kafka-share-groups.sh.
> >
> > DJ4: It looks like DescribeGroupsResponseData only has protocol type at
> group level.
> > Both DescribeGroupsResponseData and ConsumerGroupDescribeResponseData
> don’t have protocol at member level.
> > Could we use a followup to add it?
> >
> > DJ5: Update KIP to put LEADER-EPOCH before CURRENT-OFFSET.
> >
> > Thanks,
> > PoAn
> >
> >> On Nov 14, 2024, at 3:43 PM, David Jacot <dj...@confluent.io.INVALID>
> wrote:
> >>
> >> Hi PoAn,
> >>
> >> Thanks for the KIP! I have a few minor comments/suggestions:
> >>
> >> DJ1: In the output of `--describe --verbose`, I would suggest putting
> >> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before `#MEMBERS`.
> >> DJ2: Continuing on the above, I assume that we will print out N/A for
> the
> >> fields not supported by classic groups. Is this correct?
> >> DJ3: In the output of `--describe --members --verbose`, I wonder if we
> >> should use the following order and terms: CURRENT-EPOCH
> CURRENT-ASSIGNMENT
> >> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the GROUP-EPOCH because
> it
> >> is already in the group description. The value `(0)` used for the
> >> assignment is incorrect. Here I suppose that we will print out the list
> of
> >> partitions in the form of `topicid-partitionid`.
> >> DJ4: Continuing on the above, I wonder if we should also add the
> protocol
> >> used `classic` or `consumer`. For context, it is possible to have
> `classic`
> >> members and `consumer` members in a `consumer` group during an online
> >> upgrade from the classic protocol to the consumer protocol. Having this
> >> information may be handy for administrators. What do you think?
> >> DJ5: In the output of `--describe --offsets --verbose`, I would suggest
> >> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`.
> >>
> >> It would be great if we could get this one in AK 4.0 as it changes the
> >> output of the command.
> >>
> >> Thanks,
> >> DJ
> >>
> >> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <ya...@gmail.com> wrote:
> >>
> >>> Hi Sean / Andrew / Lianet,
> >>>
> >>> Thanks for all your review and suggestions.
> >>>
> >>> AS1, LM1, LM4: Change to add KIP-848 information when users give
> —verbose
> >>> option.
> >>> —describe —verbose: shows group epoch and target assignment epoch.
> >>> —describe —members —verbose: shows above information, member epoch, and
> >>> target assignment.
> >>>
> >>> AS2: Change to use MEMBER-EPOCH to align with KIP-848 definition.
> >>>
> >>> LM2: For classic group, it doesn’t have epoch, so I use Optional
> fields in
> >>> ConsumerGroupDescription.
> >>> For share group, it relies on KIP-848. It must have epoch, so I use int
> >>> fields in ShareGroupDescription.
> >>>
> >>> LM3: Remove —state change. We can get group level information by
> —describe
> >>> —verbose.
> >>>
> >>> SQ1: Add LEADER-EPOCH when users give —describe —offsets —verbose.
> >>>
> >>> Thanks.
> >>> PoAn
> >>>
> >>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <li...@gmail.com> wrote:
> >>>>
> >>>> Hello Frank, thanks for the KIP! A few comments:
> >>>>
> >>>> LM1. I strongly agree with Andrew's suggestion of moving this into a
> >>>> --verbose option. I expect someone would only attempt to make sense of
> >>> the
> >>>> epochs while debugging an issue, not while trying to view the group or
> >>>> member's info. (Also member-epoch makes more sense to me, even more
> if we
> >>>> end up agreeing in a --verbose). Related to both issues, my take is
> that
> >>>> whoever is close to the protocol will expect member-epoch, whoever is
> not
> >>>> will probably won't even need to see the epochs at all.
> >>>>
> >>>> LM2. Why are the epochs defined as Optional (don't we expect to always
> >>> have
> >>>> them? with 0 initially, defined on the broker side for the group ones,
> >>> and
> >>>> client side for the member)
> >>>>
> >>>> LM3. Why is the KIP including the –state option in the proposed
> changes?
> >>> (
> >>>>
> >>>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
> >>>> ).
> >>>> I get that the output would change in that example, but it’s not
> because
> >>> of
> >>>> any change to the –state option itself. It's because of the change
> >>> proposed
> >>>> to the –described (required with the --state), and the changes to
> >>>> --describe are already explained above (seems confusing, got me
> looking
> >>> for
> >>>> a change to the state filter which seemed unrelated).
> >>>>
> >>>> LM4. Group epoch and target assignment epoch are conceptually at the
> >>> group
> >>>> level. vs member epoch that lands at a member level. So wonder if we
> >>> should
> >>>> show them accordingly (ex. using the --verbose option)
> >>>> –describe –verbose => shows group epoch and target assignment epoch
> >>>> –describe –members –verbose => shows member epoch (along
> >>>> with group epoch and target assignment epoch)
> >>>>
> >>>> Thanks!
> >>>>
> >>>> Lianet
> >>>>
> >>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield <
> >>>> andrew_schofield_jira@outlook.com> wrote:
> >>>>
> >>>>> Hi PoAn,
> >>>>> Thanks for the KIP. I have a few comments.
> >>>>>
> >>>>> AS1: It seems to me that these new additions are most useful to
> people
> >>>>> trying to understand
> >>>>> the progress of rebalancing in quite some detail. The information is
> >>>>> really not understandable
> >>>>> for most users who do not have deep knowledge of KIP-848/932.
> >>>>>
> >>>>> As a result, I suggest for kafka-share-groups.sh that you add a
> >>> --members
> >>>>> --verbose option
> >>>>> and only include the new information in the output for that option,
> >>> rather
> >>>>> than changing the
> >>>>> non-verbose --members output.
> >>>>>
> >>>>> I also make a similar suggestion for kafka-consumer-groups.sh
> --members
> >>>>> and only add the
> >>>>> new information for the --verbose output.
> >>>>>
> >>>>> AS2: I strongly suggest that you use MEMBER-EPOCH instead of
> >>>>> CONSUMER-EPOCH.
> >>>>> I think it's more confusing not following the terminology of the
> KIPs.
> >>> For
> >>>>> one thing,
> >>>>> we already have "member" in the admin client such as
> MemberDescription.
> >>>>> It's not a
> >>>>> new term.
> >>>>>
> >>>>>
> >>>>> Thanks,
> >>>>> Andrew
> >>>>> ________________________________________
> >>>>> From: PoAn Yang <pa...@apache.org>
> >>>>> Sent: 25 October 2024 13:55
> >>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
> >>>>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command
> >>> line
> >>>>> tool to support new consumer group
> >>>>>
> >>>>> Hi Lucas,
> >>>>>
> >>>>> Thanks for the review!
> >>>>>
> >>>>> 1) Yes, I add related change for kafka-share-groups.sh to the KIP.
> Could
> >>>>> you take a look? Thanks for the suggestion.
> >>>>>
> >>>>> 2) We use CONSUMER-ID as member ID. If we use MEMBER-EPOCH here,
> users
> >>> may
> >>>>> confuse what is different between CONSUMER and MEMBER.
> >>>>>
> >>>>> Thanks,
> >>>>> PoAn
> >>>>>
> >>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
> >>>>>> Hi Frank,
> >>>>>>
> >>>>>> thanks for the KIP!
> >>>>>>
> >>>>>> 1) For consistency, should we do the same for
> >>>>>> kafka-share-groups.sh, ShareGroupDescription, etc. ? Even if we do
> not
> >>>>>> implement it right now if the share group implementation may still
> be
> >>>>>> incomplete, it may make sense to include it in the KIP.
> >>>>>>
> >>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH? That would seem
> more
> >>>>>> consistent.
> >>>>>>
> >>>>>> Cheers,
> >>>>>> Lucas
> >>>>>>
> >>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <ya...@gmail.com>
> wrote:
> >>>>>>
> >>>>>>> Hi all,
> >>>>>>>
> >>>>>>> I would like to kick off the discussion of KIP-1099. This KIP
> enhances
> >>>>> the
> >>>>>>> kafka-consumer-groups tools to include state which is introduced by
> >>>>> KIP-848.
> >>>>>>>
> >>>>>>> KIP-1099: Extend kafka-consumer-groups command line tool to support
> >>> new
> >>>>>>> consumer group - Apache Kafka - Apache Software Foundation
> >>>>>>> <
> >>>>>
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>
> >>>>>>> cwiki.apache.org
> >>>>>>> <
> >>>>>
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>
> >>>>>>> [image: favicon.ico]
> >>>>>>> <
> >>>>>
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>
> >>>>>>> <
> >>>>>
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>>>>
> >>>>>>>
> >>>>>>> Thank you,
> >>>>>>> PoAn
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>>
>
>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by PoAn Yang <ya...@gmail.com>.
Hi David / Andrew,
Thanks for review. Thanks Andrew for picking up kafka-share-groups.sh implementation.
I will handle kafka-consumer-groups.sh.
DJ3: After discussing with @Chia-Ping Tsai, we think that using new format is more clear.
The new format will be like <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2>
Using colon(:) to concat topic name and partition IDs.
Using comma(,) to concat partition IDs within same topic name.
Using semicolon(;) to concat topic strings.
AS3: Fix it with kafka-share-groups.sh. Thanks.
If there is no further suggestion, I will start a vote thread next Monday.
Thanks,
PoAn
> On Nov 14, 2024, at 9:05 PM, Andrew Schofield <an...@outlook.com> wrote:
>
> Hi PoAn,
> DJ2: I was just going to comment that "-" would be a more appropriate missing value, but
> you got there first.
>
> AS3: The examples for kafka-share-groups.sh include kafka-consumer-groups.sh in the
> command line.
>
> If this is accepted in time, I'm happy to pick up the implementation of the share groups
> part of this if it helps.
>
> Thanks,
> Andrew
>
> ________________________________________
> From: Frank Yang <ya...@gmail.com>
> Sent: 14 November 2024 10:48
> To: dev@kafka.apache.org <de...@kafka.apache.org>
> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
>
> Hi David,
>
> Thanks for the review and suggestion! I would like to get this in AK 4.0 as well. I will do my best.
>
> DJ1: Update KIP to put GROUP-EPOCH and TARGET-ASSIGNMENT-EPOCH before #MEMBERS.
>
> DJ2: I prefer to follow current missing column value “-“. (reference <https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92>)
>
> DJ3: Update KIP to use CURRENT-EPOCH CURRENT-ASSIGNMENT TARGET-EPOCH TARGET-ASSIGNMENT.
> Remove GROUP-EPOCH.
>
> For assignment value, it follows current output (reference <https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418>). I think the form of `topicid-partitionid` is more clear.
> If we would like to use this form, I will update both output in kafka-consumer-groups.sh and kafka-share-groups.sh.
>
> DJ4: It looks like DescribeGroupsResponseData only has protocol type at group level.
> Both DescribeGroupsResponseData and ConsumerGroupDescribeResponseData don’t have protocol at member level.
> Could we use a followup to add it?
>
> DJ5: Update KIP to put LEADER-EPOCH before CURRENT-OFFSET.
>
> Thanks,
> PoAn
>
>> On Nov 14, 2024, at 3:43 PM, David Jacot <dj...@confluent.io.INVALID> wrote:
>>
>> Hi PoAn,
>>
>> Thanks for the KIP! I have a few minor comments/suggestions:
>>
>> DJ1: In the output of `--describe --verbose`, I would suggest putting
>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before `#MEMBERS`.
>> DJ2: Continuing on the above, I assume that we will print out N/A for the
>> fields not supported by classic groups. Is this correct?
>> DJ3: In the output of `--describe --members --verbose`, I wonder if we
>> should use the following order and terms: CURRENT-EPOCH CURRENT-ASSIGNMENT
>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the GROUP-EPOCH because it
>> is already in the group description. The value `(0)` used for the
>> assignment is incorrect. Here I suppose that we will print out the list of
>> partitions in the form of `topicid-partitionid`.
>> DJ4: Continuing on the above, I wonder if we should also add the protocol
>> used `classic` or `consumer`. For context, it is possible to have `classic`
>> members and `consumer` members in a `consumer` group during an online
>> upgrade from the classic protocol to the consumer protocol. Having this
>> information may be handy for administrators. What do you think?
>> DJ5: In the output of `--describe --offsets --verbose`, I would suggest
>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`.
>>
>> It would be great if we could get this one in AK 4.0 as it changes the
>> output of the command.
>>
>> Thanks,
>> DJ
>>
>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <ya...@gmail.com> wrote:
>>
>>> Hi Sean / Andrew / Lianet,
>>>
>>> Thanks for all your review and suggestions.
>>>
>>> AS1, LM1, LM4: Change to add KIP-848 information when users give —verbose
>>> option.
>>> —describe —verbose: shows group epoch and target assignment epoch.
>>> —describe —members —verbose: shows above information, member epoch, and
>>> target assignment.
>>>
>>> AS2: Change to use MEMBER-EPOCH to align with KIP-848 definition.
>>>
>>> LM2: For classic group, it doesn’t have epoch, so I use Optional fields in
>>> ConsumerGroupDescription.
>>> For share group, it relies on KIP-848. It must have epoch, so I use int
>>> fields in ShareGroupDescription.
>>>
>>> LM3: Remove —state change. We can get group level information by —describe
>>> —verbose.
>>>
>>> SQ1: Add LEADER-EPOCH when users give —describe —offsets —verbose.
>>>
>>> Thanks.
>>> PoAn
>>>
>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <li...@gmail.com> wrote:
>>>>
>>>> Hello Frank, thanks for the KIP! A few comments:
>>>>
>>>> LM1. I strongly agree with Andrew's suggestion of moving this into a
>>>> --verbose option. I expect someone would only attempt to make sense of
>>> the
>>>> epochs while debugging an issue, not while trying to view the group or
>>>> member's info. (Also member-epoch makes more sense to me, even more if we
>>>> end up agreeing in a --verbose). Related to both issues, my take is that
>>>> whoever is close to the protocol will expect member-epoch, whoever is not
>>>> will probably won't even need to see the epochs at all.
>>>>
>>>> LM2. Why are the epochs defined as Optional (don't we expect to always
>>> have
>>>> them? with 0 initially, defined on the broker side for the group ones,
>>> and
>>>> client side for the member)
>>>>
>>>> LM3. Why is the KIP including the –state option in the proposed changes?
>>> (
>>>>
>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
>>>> ).
>>>> I get that the output would change in that example, but it’s not because
>>> of
>>>> any change to the –state option itself. It's because of the change
>>> proposed
>>>> to the –described (required with the --state), and the changes to
>>>> --describe are already explained above (seems confusing, got me looking
>>> for
>>>> a change to the state filter which seemed unrelated).
>>>>
>>>> LM4. Group epoch and target assignment epoch are conceptually at the
>>> group
>>>> level. vs member epoch that lands at a member level. So wonder if we
>>> should
>>>> show them accordingly (ex. using the --verbose option)
>>>> –describe –verbose => shows group epoch and target assignment epoch
>>>> –describe –members –verbose => shows member epoch (along
>>>> with group epoch and target assignment epoch)
>>>>
>>>> Thanks!
>>>>
>>>> Lianet
>>>>
>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield <
>>>> andrew_schofield_jira@outlook.com> wrote:
>>>>
>>>>> Hi PoAn,
>>>>> Thanks for the KIP. I have a few comments.
>>>>>
>>>>> AS1: It seems to me that these new additions are most useful to people
>>>>> trying to understand
>>>>> the progress of rebalancing in quite some detail. The information is
>>>>> really not understandable
>>>>> for most users who do not have deep knowledge of KIP-848/932.
>>>>>
>>>>> As a result, I suggest for kafka-share-groups.sh that you add a
>>> --members
>>>>> --verbose option
>>>>> and only include the new information in the output for that option,
>>> rather
>>>>> than changing the
>>>>> non-verbose --members output.
>>>>>
>>>>> I also make a similar suggestion for kafka-consumer-groups.sh --members
>>>>> and only add the
>>>>> new information for the --verbose output.
>>>>>
>>>>> AS2: I strongly suggest that you use MEMBER-EPOCH instead of
>>>>> CONSUMER-EPOCH.
>>>>> I think it's more confusing not following the terminology of the KIPs.
>>> For
>>>>> one thing,
>>>>> we already have "member" in the admin client such as MemberDescription.
>>>>> It's not a
>>>>> new term.
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Andrew
>>>>> ________________________________________
>>>>> From: PoAn Yang <pa...@apache.org>
>>>>> Sent: 25 October 2024 13:55
>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command
>>> line
>>>>> tool to support new consumer group
>>>>>
>>>>> Hi Lucas,
>>>>>
>>>>> Thanks for the review!
>>>>>
>>>>> 1) Yes, I add related change for kafka-share-groups.sh to the KIP. Could
>>>>> you take a look? Thanks for the suggestion.
>>>>>
>>>>> 2) We use CONSUMER-ID as member ID. If we use MEMBER-EPOCH here, users
>>> may
>>>>> confuse what is different between CONSUMER and MEMBER.
>>>>>
>>>>> Thanks,
>>>>> PoAn
>>>>>
>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
>>>>>> Hi Frank,
>>>>>>
>>>>>> thanks for the KIP!
>>>>>>
>>>>>> 1) For consistency, should we do the same for
>>>>>> kafka-share-groups.sh, ShareGroupDescription, etc. ? Even if we do not
>>>>>> implement it right now if the share group implementation may still be
>>>>>> incomplete, it may make sense to include it in the KIP.
>>>>>>
>>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH? That would seem more
>>>>>> consistent.
>>>>>>
>>>>>> Cheers,
>>>>>> Lucas
>>>>>>
>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <ya...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I would like to kick off the discussion of KIP-1099. This KIP enhances
>>>>> the
>>>>>>> kafka-consumer-groups tools to include state which is introduced by
>>>>> KIP-848.
>>>>>>>
>>>>>>> KIP-1099: Extend kafka-consumer-groups command line tool to support
>>> new
>>>>>>> consumer group - Apache Kafka - Apache Software Foundation
>>>>>>> <
>>>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>
>>>>>>> cwiki.apache.org
>>>>>>> <
>>>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>
>>>>>>> [image: favicon.ico]
>>>>>>> <
>>>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>
>>>>>>> <
>>>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>>
>>>>>>>
>>>>>>> Thank you,
>>>>>>> PoAn
>>>>>>>
>>>>>>
>>>>>
>>>
>>>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by Andrew Schofield <an...@outlook.com>.
Hi PoAn,
DJ2: I was just going to comment that "-" would be a more appropriate missing value, but
you got there first.
AS3: The examples for kafka-share-groups.sh include kafka-consumer-groups.sh in the
command line.
If this is accepted in time, I'm happy to pick up the implementation of the share groups
part of this if it helps.
Thanks,
Andrew
________________________________________
From: Frank Yang <ya...@gmail.com>
Sent: 14 November 2024 10:48
To: dev@kafka.apache.org <de...@kafka.apache.org>
Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Hi David,
Thanks for the review and suggestion! I would like to get this in AK 4.0 as well. I will do my best.
DJ1: Update KIP to put GROUP-EPOCH and TARGET-ASSIGNMENT-EPOCH before #MEMBERS.
DJ2: I prefer to follow current missing column value “-“. (reference <https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92>)
DJ3: Update KIP to use CURRENT-EPOCH CURRENT-ASSIGNMENT TARGET-EPOCH TARGET-ASSIGNMENT.
Remove GROUP-EPOCH.
For assignment value, it follows current output (reference <https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418>). I think the form of `topicid-partitionid` is more clear.
If we would like to use this form, I will update both output in kafka-consumer-groups.sh and kafka-share-groups.sh.
DJ4: It looks like DescribeGroupsResponseData only has protocol type at group level.
Both DescribeGroupsResponseData and ConsumerGroupDescribeResponseData don’t have protocol at member level.
Could we use a followup to add it?
DJ5: Update KIP to put LEADER-EPOCH before CURRENT-OFFSET.
Thanks,
PoAn
> On Nov 14, 2024, at 3:43 PM, David Jacot <dj...@confluent.io.INVALID> wrote:
>
> Hi PoAn,
>
> Thanks for the KIP! I have a few minor comments/suggestions:
>
> DJ1: In the output of `--describe --verbose`, I would suggest putting
> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before `#MEMBERS`.
> DJ2: Continuing on the above, I assume that we will print out N/A for the
> fields not supported by classic groups. Is this correct?
> DJ3: In the output of `--describe --members --verbose`, I wonder if we
> should use the following order and terms: CURRENT-EPOCH CURRENT-ASSIGNMENT
> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the GROUP-EPOCH because it
> is already in the group description. The value `(0)` used for the
> assignment is incorrect. Here I suppose that we will print out the list of
> partitions in the form of `topicid-partitionid`.
> DJ4: Continuing on the above, I wonder if we should also add the protocol
> used `classic` or `consumer`. For context, it is possible to have `classic`
> members and `consumer` members in a `consumer` group during an online
> upgrade from the classic protocol to the consumer protocol. Having this
> information may be handy for administrators. What do you think?
> DJ5: In the output of `--describe --offsets --verbose`, I would suggest
> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`.
>
> It would be great if we could get this one in AK 4.0 as it changes the
> output of the command.
>
> Thanks,
> DJ
>
> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <ya...@gmail.com> wrote:
>
>> Hi Sean / Andrew / Lianet,
>>
>> Thanks for all your review and suggestions.
>>
>> AS1, LM1, LM4: Change to add KIP-848 information when users give —verbose
>> option.
>> —describe —verbose: shows group epoch and target assignment epoch.
>> —describe —members —verbose: shows above information, member epoch, and
>> target assignment.
>>
>> AS2: Change to use MEMBER-EPOCH to align with KIP-848 definition.
>>
>> LM2: For classic group, it doesn’t have epoch, so I use Optional fields in
>> ConsumerGroupDescription.
>> For share group, it relies on KIP-848. It must have epoch, so I use int
>> fields in ShareGroupDescription.
>>
>> LM3: Remove —state change. We can get group level information by —describe
>> —verbose.
>>
>> SQ1: Add LEADER-EPOCH when users give —describe —offsets —verbose.
>>
>> Thanks.
>> PoAn
>>
>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <li...@gmail.com> wrote:
>>>
>>> Hello Frank, thanks for the KIP! A few comments:
>>>
>>> LM1. I strongly agree with Andrew's suggestion of moving this into a
>>> --verbose option. I expect someone would only attempt to make sense of
>> the
>>> epochs while debugging an issue, not while trying to view the group or
>>> member's info. (Also member-epoch makes more sense to me, even more if we
>>> end up agreeing in a --verbose). Related to both issues, my take is that
>>> whoever is close to the protocol will expect member-epoch, whoever is not
>>> will probably won't even need to see the epochs at all.
>>>
>>> LM2. Why are the epochs defined as Optional (don't we expect to always
>> have
>>> them? with 0 initially, defined on the broker side for the group ones,
>> and
>>> client side for the member)
>>>
>>> LM3. Why is the KIP including the –state option in the proposed changes?
>> (
>>>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
>>> ).
>>> I get that the output would change in that example, but it’s not because
>> of
>>> any change to the –state option itself. It's because of the change
>> proposed
>>> to the –described (required with the --state), and the changes to
>>> --describe are already explained above (seems confusing, got me looking
>> for
>>> a change to the state filter which seemed unrelated).
>>>
>>> LM4. Group epoch and target assignment epoch are conceptually at the
>> group
>>> level. vs member epoch that lands at a member level. So wonder if we
>> should
>>> show them accordingly (ex. using the --verbose option)
>>> –describe –verbose => shows group epoch and target assignment epoch
>>> –describe –members –verbose => shows member epoch (along
>>> with group epoch and target assignment epoch)
>>>
>>> Thanks!
>>>
>>> Lianet
>>>
>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield <
>>> andrew_schofield_jira@outlook.com> wrote:
>>>
>>>> Hi PoAn,
>>>> Thanks for the KIP. I have a few comments.
>>>>
>>>> AS1: It seems to me that these new additions are most useful to people
>>>> trying to understand
>>>> the progress of rebalancing in quite some detail. The information is
>>>> really not understandable
>>>> for most users who do not have deep knowledge of KIP-848/932.
>>>>
>>>> As a result, I suggest for kafka-share-groups.sh that you add a
>> --members
>>>> --verbose option
>>>> and only include the new information in the output for that option,
>> rather
>>>> than changing the
>>>> non-verbose --members output.
>>>>
>>>> I also make a similar suggestion for kafka-consumer-groups.sh --members
>>>> and only add the
>>>> new information for the --verbose output.
>>>>
>>>> AS2: I strongly suggest that you use MEMBER-EPOCH instead of
>>>> CONSUMER-EPOCH.
>>>> I think it's more confusing not following the terminology of the KIPs.
>> For
>>>> one thing,
>>>> we already have "member" in the admin client such as MemberDescription.
>>>> It's not a
>>>> new term.
>>>>
>>>>
>>>> Thanks,
>>>> Andrew
>>>> ________________________________________
>>>> From: PoAn Yang <pa...@apache.org>
>>>> Sent: 25 October 2024 13:55
>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
>>>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command
>> line
>>>> tool to support new consumer group
>>>>
>>>> Hi Lucas,
>>>>
>>>> Thanks for the review!
>>>>
>>>> 1) Yes, I add related change for kafka-share-groups.sh to the KIP. Could
>>>> you take a look? Thanks for the suggestion.
>>>>
>>>> 2) We use CONSUMER-ID as member ID. If we use MEMBER-EPOCH here, users
>> may
>>>> confuse what is different between CONSUMER and MEMBER.
>>>>
>>>> Thanks,
>>>> PoAn
>>>>
>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
>>>>> Hi Frank,
>>>>>
>>>>> thanks for the KIP!
>>>>>
>>>>> 1) For consistency, should we do the same for
>>>>> kafka-share-groups.sh, ShareGroupDescription, etc. ? Even if we do not
>>>>> implement it right now if the share group implementation may still be
>>>>> incomplete, it may make sense to include it in the KIP.
>>>>>
>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH? That would seem more
>>>>> consistent.
>>>>>
>>>>> Cheers,
>>>>> Lucas
>>>>>
>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <ya...@gmail.com> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I would like to kick off the discussion of KIP-1099. This KIP enhances
>>>> the
>>>>>> kafka-consumer-groups tools to include state which is introduced by
>>>> KIP-848.
>>>>>>
>>>>>> KIP-1099: Extend kafka-consumer-groups command line tool to support
>> new
>>>>>> consumer group - Apache Kafka - Apache Software Foundation
>>>>>> <
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>
>>>>>> cwiki.apache.org
>>>>>> <
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>
>>>>>> [image: favicon.ico]
>>>>>> <
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>
>>>>>> <
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>
>>>>>>
>>>>>> Thank you,
>>>>>> PoAn
>>>>>>
>>>>>
>>>>
>>
>>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by Frank Yang <ya...@gmail.com>.
Hi David,
Thanks for the review and suggestion! I would like to get this in AK 4.0 as well. I will do my best.
DJ1: Update KIP to put GROUP-EPOCH and TARGET-ASSIGNMENT-EPOCH before #MEMBERS.
DJ2: I prefer to follow current missing column value “-“. (reference <https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92>)
DJ3: Update KIP to use CURRENT-EPOCH CURRENT-ASSIGNMENT TARGET-EPOCH TARGET-ASSIGNMENT.
Remove GROUP-EPOCH.
For assignment value, it follows current output (reference <https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418>). I think the form of `topicid-partitionid` is more clear.
If we would like to use this form, I will update both output in kafka-consumer-groups.sh and kafka-share-groups.sh.
DJ4: It looks like DescribeGroupsResponseData only has protocol type at group level.
Both DescribeGroupsResponseData and ConsumerGroupDescribeResponseData don’t have protocol at member level.
Could we use a followup to add it?
DJ5: Update KIP to put LEADER-EPOCH before CURRENT-OFFSET.
Thanks,
PoAn
> On Nov 14, 2024, at 3:43 PM, David Jacot <dj...@confluent.io.INVALID> wrote:
>
> Hi PoAn,
>
> Thanks for the KIP! I have a few minor comments/suggestions:
>
> DJ1: In the output of `--describe --verbose`, I would suggest putting
> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before `#MEMBERS`.
> DJ2: Continuing on the above, I assume that we will print out N/A for the
> fields not supported by classic groups. Is this correct?
> DJ3: In the output of `--describe --members --verbose`, I wonder if we
> should use the following order and terms: CURRENT-EPOCH CURRENT-ASSIGNMENT
> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the GROUP-EPOCH because it
> is already in the group description. The value `(0)` used for the
> assignment is incorrect. Here I suppose that we will print out the list of
> partitions in the form of `topicid-partitionid`.
> DJ4: Continuing on the above, I wonder if we should also add the protocol
> used `classic` or `consumer`. For context, it is possible to have `classic`
> members and `consumer` members in a `consumer` group during an online
> upgrade from the classic protocol to the consumer protocol. Having this
> information may be handy for administrators. What do you think?
> DJ5: In the output of `--describe --offsets --verbose`, I would suggest
> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`.
>
> It would be great if we could get this one in AK 4.0 as it changes the
> output of the command.
>
> Thanks,
> DJ
>
> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <ya...@gmail.com> wrote:
>
>> Hi Sean / Andrew / Lianet,
>>
>> Thanks for all your review and suggestions.
>>
>> AS1, LM1, LM4: Change to add KIP-848 information when users give —verbose
>> option.
>> —describe —verbose: shows group epoch and target assignment epoch.
>> —describe —members —verbose: shows above information, member epoch, and
>> target assignment.
>>
>> AS2: Change to use MEMBER-EPOCH to align with KIP-848 definition.
>>
>> LM2: For classic group, it doesn’t have epoch, so I use Optional fields in
>> ConsumerGroupDescription.
>> For share group, it relies on KIP-848. It must have epoch, so I use int
>> fields in ShareGroupDescription.
>>
>> LM3: Remove —state change. We can get group level information by —describe
>> —verbose.
>>
>> SQ1: Add LEADER-EPOCH when users give —describe —offsets —verbose.
>>
>> Thanks.
>> PoAn
>>
>>> On Nov 1, 2024, at 5:08 AM, Lianet M. <li...@gmail.com> wrote:
>>>
>>> Hello Frank, thanks for the KIP! A few comments:
>>>
>>> LM1. I strongly agree with Andrew's suggestion of moving this into a
>>> --verbose option. I expect someone would only attempt to make sense of
>> the
>>> epochs while debugging an issue, not while trying to view the group or
>>> member's info. (Also member-epoch makes more sense to me, even more if we
>>> end up agreeing in a --verbose). Related to both issues, my take is that
>>> whoever is close to the protocol will expect member-epoch, whoever is not
>>> will probably won't even need to see the epochs at all.
>>>
>>> LM2. Why are the epochs defined as Optional (don't we expect to always
>> have
>>> them? with 0 initially, defined on the broker side for the group ones,
>> and
>>> client side for the member)
>>>
>>> LM3. Why is the KIP including the –state option in the proposed changes?
>> (
>>>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
>>> ).
>>> I get that the output would change in that example, but it’s not because
>> of
>>> any change to the –state option itself. It's because of the change
>> proposed
>>> to the –described (required with the --state), and the changes to
>>> --describe are already explained above (seems confusing, got me looking
>> for
>>> a change to the state filter which seemed unrelated).
>>>
>>> LM4. Group epoch and target assignment epoch are conceptually at the
>> group
>>> level. vs member epoch that lands at a member level. So wonder if we
>> should
>>> show them accordingly (ex. using the --verbose option)
>>> –describe –verbose => shows group epoch and target assignment epoch
>>> –describe –members –verbose => shows member epoch (along
>>> with group epoch and target assignment epoch)
>>>
>>> Thanks!
>>>
>>> Lianet
>>>
>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield <
>>> andrew_schofield_jira@outlook.com> wrote:
>>>
>>>> Hi PoAn,
>>>> Thanks for the KIP. I have a few comments.
>>>>
>>>> AS1: It seems to me that these new additions are most useful to people
>>>> trying to understand
>>>> the progress of rebalancing in quite some detail. The information is
>>>> really not understandable
>>>> for most users who do not have deep knowledge of KIP-848/932.
>>>>
>>>> As a result, I suggest for kafka-share-groups.sh that you add a
>> --members
>>>> --verbose option
>>>> and only include the new information in the output for that option,
>> rather
>>>> than changing the
>>>> non-verbose --members output.
>>>>
>>>> I also make a similar suggestion for kafka-consumer-groups.sh --members
>>>> and only add the
>>>> new information for the --verbose output.
>>>>
>>>> AS2: I strongly suggest that you use MEMBER-EPOCH instead of
>>>> CONSUMER-EPOCH.
>>>> I think it's more confusing not following the terminology of the KIPs.
>> For
>>>> one thing,
>>>> we already have "member" in the admin client such as MemberDescription.
>>>> It's not a
>>>> new term.
>>>>
>>>>
>>>> Thanks,
>>>> Andrew
>>>> ________________________________________
>>>> From: PoAn Yang <pa...@apache.org>
>>>> Sent: 25 October 2024 13:55
>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
>>>> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command
>> line
>>>> tool to support new consumer group
>>>>
>>>> Hi Lucas,
>>>>
>>>> Thanks for the review!
>>>>
>>>> 1) Yes, I add related change for kafka-share-groups.sh to the KIP. Could
>>>> you take a look? Thanks for the suggestion.
>>>>
>>>> 2) We use CONSUMER-ID as member ID. If we use MEMBER-EPOCH here, users
>> may
>>>> confuse what is different between CONSUMER and MEMBER.
>>>>
>>>> Thanks,
>>>> PoAn
>>>>
>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
>>>>> Hi Frank,
>>>>>
>>>>> thanks for the KIP!
>>>>>
>>>>> 1) For consistency, should we do the same for
>>>>> kafka-share-groups.sh, ShareGroupDescription, etc. ? Even if we do not
>>>>> implement it right now if the share group implementation may still be
>>>>> incomplete, it may make sense to include it in the KIP.
>>>>>
>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH? That would seem more
>>>>> consistent.
>>>>>
>>>>> Cheers,
>>>>> Lucas
>>>>>
>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <ya...@gmail.com> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I would like to kick off the discussion of KIP-1099. This KIP enhances
>>>> the
>>>>>> kafka-consumer-groups tools to include state which is introduced by
>>>> KIP-848.
>>>>>>
>>>>>> KIP-1099: Extend kafka-consumer-groups command line tool to support
>> new
>>>>>> consumer group - Apache Kafka - Apache Software Foundation
>>>>>> <
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>
>>>>>> cwiki.apache.org
>>>>>> <
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>
>>>>>> [image: favicon.ico]
>>>>>> <
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>
>>>>>> <
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
>>>>>
>>>>>>
>>>>>> Thank you,
>>>>>> PoAn
>>>>>>
>>>>>
>>>>
>>
>>
Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group
Posted by David Jacot <dj...@confluent.io.INVALID>.
Hi PoAn,
Thanks for the KIP! I have a few minor comments/suggestions:
DJ1: In the output of `--describe --verbose`, I would suggest putting
`GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before `#MEMBERS`.
DJ2: Continuing on the above, I assume that we will print out N/A for the
fields not supported by classic groups. Is this correct?
DJ3: In the output of `--describe --members --verbose`, I wonder if we
should use the following order and terms: CURRENT-EPOCH CURRENT-ASSIGNMENT
TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the GROUP-EPOCH because it
is already in the group description. The value `(0)` used for the
assignment is incorrect. Here I suppose that we will print out the list of
partitions in the form of `topicid-partitionid`.
DJ4: Continuing on the above, I wonder if we should also add the protocol
used `classic` or `consumer`. For context, it is possible to have `classic`
members and `consumer` members in a `consumer` group during an online
upgrade from the classic protocol to the consumer protocol. Having this
information may be handy for administrators. What do you think?
DJ5: In the output of `--describe --offsets --verbose`, I would suggest
putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`.
It would be great if we could get this one in AK 4.0 as it changes the
output of the command.
Thanks,
DJ
On Fri, Nov 1, 2024 at 7:40 AM Frank Yang <ya...@gmail.com> wrote:
> Hi Sean / Andrew / Lianet,
>
> Thanks for all your review and suggestions.
>
> AS1, LM1, LM4: Change to add KIP-848 information when users give —verbose
> option.
> —describe —verbose: shows group epoch and target assignment epoch.
> —describe —members —verbose: shows above information, member epoch, and
> target assignment.
>
> AS2: Change to use MEMBER-EPOCH to align with KIP-848 definition.
>
> LM2: For classic group, it doesn’t have epoch, so I use Optional fields in
> ConsumerGroupDescription.
> For share group, it relies on KIP-848. It must have epoch, so I use int
> fields in ShareGroupDescription.
>
> LM3: Remove —state change. We can get group level information by —describe
> —verbose.
>
> SQ1: Add LEADER-EPOCH when users give —describe —offsets —verbose.
>
> Thanks.
> PoAn
>
> > On Nov 1, 2024, at 5:08 AM, Lianet M. <li...@gmail.com> wrote:
> >
> > Hello Frank, thanks for the KIP! A few comments:
> >
> > LM1. I strongly agree with Andrew's suggestion of moving this into a
> > --verbose option. I expect someone would only attempt to make sense of
> the
> > epochs while debugging an issue, not while trying to view the group or
> > member's info. (Also member-epoch makes more sense to me, even more if we
> > end up agreeing in a --verbose). Related to both issues, my take is that
> > whoever is close to the protocol will expect member-epoch, whoever is not
> > will probably won't even need to see the epochs at all.
> >
> > LM2. Why are the epochs defined as Optional (don't we expect to always
> have
> > them? with 0 initially, defined on the broker side for the group ones,
> and
> > client side for the member)
> >
> > LM3. Why is the KIP including the –state option in the proposed changes?
> (
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state
> > ).
> > I get that the output would change in that example, but it’s not because
> of
> > any change to the –state option itself. It's because of the change
> proposed
> > to the –described (required with the --state), and the changes to
> > --describe are already explained above (seems confusing, got me looking
> for
> > a change to the state filter which seemed unrelated).
> >
> > LM4. Group epoch and target assignment epoch are conceptually at the
> group
> > level. vs member epoch that lands at a member level. So wonder if we
> should
> > show them accordingly (ex. using the --verbose option)
> > –describe –verbose => shows group epoch and target assignment epoch
> > –describe –members –verbose => shows member epoch (along
> > with group epoch and target assignment epoch)
> >
> > Thanks!
> >
> > Lianet
> >
> > On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield <
> > andrew_schofield_jira@outlook.com> wrote:
> >
> >> Hi PoAn,
> >> Thanks for the KIP. I have a few comments.
> >>
> >> AS1: It seems to me that these new additions are most useful to people
> >> trying to understand
> >> the progress of rebalancing in quite some detail. The information is
> >> really not understandable
> >> for most users who do not have deep knowledge of KIP-848/932.
> >>
> >> As a result, I suggest for kafka-share-groups.sh that you add a
> --members
> >> --verbose option
> >> and only include the new information in the output for that option,
> rather
> >> than changing the
> >> non-verbose --members output.
> >>
> >> I also make a similar suggestion for kafka-consumer-groups.sh --members
> >> and only add the
> >> new information for the --verbose output.
> >>
> >> AS2: I strongly suggest that you use MEMBER-EPOCH instead of
> >> CONSUMER-EPOCH.
> >> I think it's more confusing not following the terminology of the KIPs.
> For
> >> one thing,
> >> we already have "member" in the admin client such as MemberDescription.
> >> It's not a
> >> new term.
> >>
> >>
> >> Thanks,
> >> Andrew
> >> ________________________________________
> >> From: PoAn Yang <pa...@apache.org>
> >> Sent: 25 October 2024 13:55
> >> To: dev@kafka.apache.org <de...@kafka.apache.org>
> >> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command
> line
> >> tool to support new consumer group
> >>
> >> Hi Lucas,
> >>
> >> Thanks for the review!
> >>
> >> 1) Yes, I add related change for kafka-share-groups.sh to the KIP. Could
> >> you take a look? Thanks for the suggestion.
> >>
> >> 2) We use CONSUMER-ID as member ID. If we use MEMBER-EPOCH here, users
> may
> >> confuse what is different between CONSUMER and MEMBER.
> >>
> >> Thanks,
> >> PoAn
> >>
> >> On 2024/10/23 13:28:17 Lucas Brutschy wrote:
> >>> Hi Frank,
> >>>
> >>> thanks for the KIP!
> >>>
> >>> 1) For consistency, should we do the same for
> >>> kafka-share-groups.sh, ShareGroupDescription, etc. ? Even if we do not
> >>> implement it right now if the share group implementation may still be
> >>> incomplete, it may make sense to include it in the KIP.
> >>>
> >>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH? That would seem more
> >>> consistent.
> >>>
> >>> Cheers,
> >>> Lucas
> >>>
> >>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang <ya...@gmail.com> wrote:
> >>>
> >>>> Hi all,
> >>>>
> >>>> I would like to kick off the discussion of KIP-1099. This KIP enhances
> >> the
> >>>> kafka-consumer-groups tools to include state which is introduced by
> >> KIP-848.
> >>>>
> >>>> KIP-1099: Extend kafka-consumer-groups command line tool to support
> new
> >>>> consumer group - Apache Kafka - Apache Software Foundation
> >>>> <
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>
> >>>> cwiki.apache.org
> >>>> <
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>
> >>>> [image: favicon.ico]
> >>>> <
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>
> >>>> <
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group
> >>>
> >>>>
> >>>> Thank you,
> >>>> PoAn
> >>>>
> >>>
> >>
>
>