[DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[TengYao Chi <ki...@gmail.com>]

Hello everyone,

I would like to start a discussion thread on KIP-1082, which proposes
enabling id generation for clients over the ConsumerGroupHeartbeat RPC.

Here is the KIP Link: KIP-1082
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC>
Please take a look and let me know what you think, and I would appreciate
any suggestions and feedback.

Best regards,
TengYao

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[David Jacot <dj...@confluent.io.INVALID>]

> I don't want to be a hater of idempotent HB, but having a "RPC" used to
generate UUID is unnecessary to me.

I actually agree with you. I am just trying to argue for it.

> I'm not sure whether it is worth requiring the UUID format for member id.
In the protocol, we declare the field "memberId" as "String" rather than
"uuid". The scope of member id is in "a group", so I guess the collision
won't be a big issue.

I think that this should be defined. Otherwise, folks may start using
incorrect things. In the java client, we will generate a Kafka UUID.
The javadoc of the Uuid class has a good definition for it.

> For another, I always wonder why we trust clients to generate "unique"
transaction id but we worry about generating "unique" member id on client
side?

This is perhaps not the best comparison. The transactional id is the most
misused thing in Apache Kafka. However, there is a notable difference. The
transaction id is provided by the end user where the member id is generated
by the client itself.


On Wed, Aug 14, 2024 at 11:56 AM Chia-Ping Tsai <ch...@gmail.com> wrote:

> In my opinion, the main downside of this
> approach is that if you leave after receiving the first HB and the member
> id, the server will respond with an unknown member id error because the
> member is not really in the group yet.
>
> I don't want to be a hater of idempotent HB, but having a "RPC" used to
> generate UUID is unnecessary to me.
>
> In the proposed changes section, we should elaborate the uuid generation
> part. Do we have recommendations there? Do we have requirements for the
> uuid (version, uniqueness, etc.)?
>
>
> I'm not sure whether it is worth requiring the UUID format for member id.
> In the protocol, we declare the field "memberId" as "String" rather than
> "uuid". The scope of member id is in "a group", so I guess the collision
> won't be a big issue.
>
> For another, I always wonder why we trust clients to generate "unique"
> transaction id but we worry about generating "unique" member id on client
> side?
>
> Best,
> Chia-Ping
>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[Chia-Ping Tsai <ch...@gmail.com>]

In my opinion, the main downside of this
approach is that if you leave after receiving the first HB and the member
id, the server will respond with an unknown member id error because the
member is not really in the group yet.

I don't want to be a hater of idempotent HB, but having a "RPC" used to
generate UUID is unnecessary to me.

In the proposed changes section, we should elaborate the uuid generation
part. Do we have recommendations there? Do we have requirements for the
uuid (version, uniqueness, etc.)?


I'm not sure whether it is worth requiring the UUID format for member id.
In the protocol, we declare the field "memberId" as "String" rather than
"uuid". The scope of member id is in "a group", so I guess the collision
won't be a big issue.

For another, I always wonder why we trust clients to generate "unique"
transaction id but we worry about generating "unique" member id on client
side?

Best,
Chia-Ping

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[David Jacot <dj...@confluent.io.INVALID>]

Hi TengYao,

Thanks for the KIP! I have a couple of comments.

1. In the motivation section, I would really start from the fundamental
issue which is that the initial heartbeat is not idempotent. Then, we could
describe the undesired side effects (e.g. ghost members, cannot leave
without receiving the member id, etc.). This is the core issue that we want
to solve with this KIP, I think.

2. In the public interface section, we need to explain that we will bump
the version of the API and that we will require the member id to be
provided from the new version on. I would also mention that we will return
invalid request if the member id is not provided.

3. In the proposed changes section, we should elaborate the uuid generation
part. Do we have recommendations there? Do we have requirements for the
uuid (version, uniqueness, etc.)?

4. In the compatibility section, I wonder if we could simplify it a bit. In
the end, the change is backward compatible because the version 1 of the RPC
already supports a member id provided by the client.

5. In KIP-848, we actually rejected the proposed option. It would be great
if we could explain why we changed our mind in the motivation section. I
think that the main argument was that generating uuid was requiring extra
libraries in some languages (e.g. for librdkafka).

6. Regarding the second rejected alternative, I feel like the arguments for
rejecting it are quite weak. Backporting is not really an issue,
compatibility neither. KIP-848 is still in preview so we could live with a
few misleading logs, if any. In my opinion, the main downside of this
approach is that if you leave after receiving the first HB and the member
id, the server will respond with an unknown member id error because the
member is not really in the group yet. We could think of having a real
lobby as suggested by Andrew. I am not a fan of this because we have to
bookkeep more state on the server. The client side generated member id is
simpler and more reliable overall.

7. Another argument for the client side generated member id is that the
member id will be equivalent to an incarnation id in the sense that it
won't change at all during the lifetime of the client.

8. It seems tricky to validate the uuid on the server because it is just a
string. Could you elaborate on this?

Best,
David

On Wed, Aug 14, 2024 at 9:48 AM TengYao Chi <ki...@gmail.com> wrote:

> Hello Andrew
> Thank you for your thoughtful suggestions and getting the discussion going.
>
> To AS1:
> In the current scenario where the server generates the UUID, if the client
> shuts down before receiving the memberId generated by the GC (regardless of
> whether it’s a graceful shutdown or not), the GC will still have to wait
> for the heartbeat timeout because the client doesn’t know its memberId.
> This KIP indeed cannot completely resolve the idempotency issue, but it can
> better handle shutdown scenarios under normal circumstances because the
> client always knows its memberId. Even if the client shuts down immediately
> after the initial heartbeat, as long as it performs a graceful shutdown and
> sends a leave heartbeat, the GC can manage the situation and remove the
> member. Therefore, the goal of this KIP is to address the issue where the
> GC has to wait for the heartbeat timeout due to the client leaving without
> knowing its memberId, which leads to reduced throughput and limited
> scalability.
>
> The solution you suggest has also been proposed by David. The concern with
> this approach is that it introduces additional complexity for
> compatibility, as the new server would not immediately add the member to
> the group, while the old server would. This requires clients to
> differentiate whether their memberId has been added to the group or not,
> which could result in unexpected logs.
>
> Best Regards,
> TengYao
>
> Andrew Schofield <an...@live.com> 於 2024年8月14日 週三 上午12:29寫道：
>
> > Hi TengYao,
> > Thanks for the KIP. I wonder if there’s a different way to close what
> > is quite a small window.
> >
> > AS1: It is true that the initial heartbeat is not idempotent, but this
> > remains
> > true with this KIP. It’s just differently not idempotent. If the client
> > makes its
> > own member ID, sends a request and dies, the GC will still have added
> > the member to the group and it will hang around until the session
> expires.
> >
> > I wonder if the GC could still generate the member ID in response to the
> > first
> > heartbeat, and put the member in a special PENDING state with no
> > assignments until the client sends the next heartbeat, thus confirming it
> > has received the member ID. This would not be a protocol change at all,
> > just
> > a change to the GC to keep a new member in the lobby until it’s comfirmed
> > it knows its member ID.
> >
> >
> > Thanks,
> > Andrew
> >
> > > On 13 Aug 2024, at 15:59, TengYao Chi <ki...@gmail.com> wrote:
> > >
> > > Hi Chia-Ping,
> > >
> > > Thanks for review and suggestions.
> > > I have updated the content of KIP accordingly.
> > > Please take a look.
> > >
> > > Best regards,
> > > TengYao
> > >
> > > Chia-Ping Tsai <ch...@apache.org> 於 2024年8月13日 週二 下午9:45寫道：
> > >
> > >> hi TengYao
> > >>
> > >> thanks for this KIP.
> > >>
> > >> 1) could you please describe the before/after behavior in the
> "Proposed
> > >> Changes" section? IIRC, current RPC allows HB having member id
> > generated by
> > >> client, right? If HB has no member ID, server will generate one and
> then
> > >> return. The new behavior will enforce HB "must" have member ID.
> > >>
> > >> 2) could you please write the version number explicitly in the KIP
> > >>
> > >> 3) how new client code handle the old HB? Does it always generate
> member
> > >> ID on client-side even though that is not restricted?
> > >>
> > >> Best,
> > >> Chia-Ping
> > >>
> > >> On 2024/08/13 06:20:42 TengYao Chi wrote:
> > >>> Hello everyone,
> > >>>
> > >>> I would like to start a discussion thread on KIP-1082, which proposes
> > >>> enabling id generation for clients over the ConsumerGroupHeartbeat
> RPC.
> > >>>
> > >>> Here is the KIP Link: KIP-1082
> > >>> <
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
> > >>>
> > >>> Please take a look and let me know what you think, and I would
> > appreciate
> > >>> any suggestions and feedback.
> > >>>
> > >>> Best regards,
> > >>> TengYao
> > >>>
> > >>
> >
> >
>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[TengYao Chi <ki...@gmail.com>]

Hi David and Chia-Ping

Thank you for your comments, which have helped move the discussion forward.
I have updated the content of the KIP accordingly. Please take a look.

Sincerely,
TengYao

Chia-Ping Tsai <ch...@gmail.com> 於 2024年9月23日 週一 下午11:01寫道：

> Hi David
>
> > I tend to believe that generating a uuid on the client should not be that
> difficult so I am not too concerned about it. How do you see this,
>
> You're right. It's not that difficult, and we should have more confidence
> in third-party libraries.
>
> hi TengYao
>
> Could you please summarize the discussion on "duplicate member ID" and add
> it to KIP-1082? Also, please remove "The member is also removed from the
> group." from KIP-848
>
> Best,
> Chia-Ping
>
> David Jacot <dj...@confluent.io.invalid> 於 2024年9月23日 週一 下午10:43寫道：
>
> > Hi Chia-Ping,
> >
> > > However, my concern is that consumers are now required to generate the
> > member ID, so as a responsible server, we need to handle the incorrect
> > client behavior. At least, based on David's comment, we can create
> another
> > member id when it is already used and it is rejoining the group (member
> > epoch = 0).
> >
> > I don't disagree with the concern. However, I don't see how we could make
> > it work in practice. Let's say that a member XYZ sends a heartbeat with
> > epoch zero. The server cannot know whether it is a new member joining for
> > the first time or an existing member rejoining because it was fenced
> (e.g.
> > wrong epoch, session timed out, etc.). We cannot differentiate the two
> > cases. Hence we cannot handle it on the server side. The only viable
> > alternative is to keep the member id generation on the server side if we
> > are concerned by clients not able to generate unique uuids on their
> side. I
> > tend to believe that generating a uuid on the client should not be that
> > difficult so I am not too concerned about it. How do you see
> > this, Chia-Ping?
> >
> > Cheers,
> > David
> >
> > On Mon, Sep 23, 2024 at 4:13 PM Chia-Ping Tsai <ch...@gmail.com>
> wrote:
> >
> > > hi David
> > >
> > > > For instance, we could check whether the member id is already used
> and
> > we
> > > could regenerate another one if it is. However, I tend to believe that
> > the
> > > risk of collision within a group is neglectable.
> > >
> > > I agree the risk is negligible when the member ID is generated by the
> > > server, as we can always ensure the server behaves responsibly. 😀
> > > Therefore, I don't believe the additional check for the member ID is
> > > necessary if the member id is created by server.
> > >
> > > However, my concern is that consumers are now required to generate the
> > > member ID, so as a responsible server, we need to handle the incorrect
> > > client behavior. At least, based on David's comment, we can create
> > another
> > > member id when it is already used and it is rejoining the group (member
> > > epoch = 0).
> > >
> > > By the way, another related issue is that a duplicate consumer may
> > > inadvertently terminate another consumer, as it will always send a
> > > LEAVE_GROUP request upon closing, even if it hasn't received a
> heartbeat
> > > response. However, I tend to believe the risk is negligible 😀
> > >
> > > Best,
> > > Chia-Ping
> > >
> > >
> > > David Jacot <dj...@confluent.io.invalid> 於 2024年9月23日 週一 下午9:07寫道：
> > >
> > > > Hi Chia-Ping,
> > > >
> > > > > Checks whether the member epoch matches the member epoch in its
> > current
> > > > assignment. FENCED_MEMBER_EPOCH is returned otherwise. The member is
> > also
> > > > removed from the group.
> > > >
> > > > We did not implement it like this in the end. We keep the member and
> > only
> > > > request him to re-synchronise himself by returning the
> > > FENCED_MEMBER_EPOCH
> > > > error. We can remove it from the KIP.
> > > >
> > > > > Given that member id is the incarnation of consumer, it is hard to
> > > detect
> > > > the scenario of "two consumers have the same member id" (it may be
> > caused
> > > > by the lost response). I guess the best approach is the later
> -created
> > > > consumer should be idle with fenced error, and we should add this to
> > the
> > > > KIP (maybe we should revise the error message carried by
> > > > `FencedMemberEpochException`). BUT, we need to rethink the approach
> if
> > > > above description "The member is also removed from the group" is true
> > > >
> > > > I don't think that we have a reliable way to detect whether two
> members
> > > end
> > > > up using the same member id. I don't think that it is possible if we
> > > > generate the member id on the client side. By the way, we don't
> handle
> > > this
> > > > case either on the server side today. We blindly generate a member id
> > and
> > > > use it. However, we could do better on the server side. For instance,
> > we
> > > > could check whether the member id is already used and we could
> > regenerate
> > > > another one if it is. However, I tend to believe that the risk of
> > > collision
> > > > within a group is neglectable.
> > > >
> > > > Best,
> > > > David
> > > >
> > > > On Fri, Sep 20, 2024 at 8:11 PM Chia-Ping Tsai <ch...@gmail.com>
> > > wrote:
> > > >
> > > > > wait a minute. There is a description about fenced error in the
> > KIP-848
> > > > >
> > > > > > Checks whether the member epoch matches the member epoch in its
> > > current
> > > > > assignment. FENCED_MEMBER_EPOCH is returned otherwise. The member
> is
> > > also
> > > > > removed from the group.
> > > > >
> > > > > I failed to find the code of implementing "The member is also
> removed
> > > > from
> > > > > the group.". I will appreciate it if someone can share the code
> link
> > to
> > > > me.
> > > > >
> > > > > Let's assume it will be implemented eventually, and I feel that
> will
> > > > cause
> > > > > a serious issue as the consumers having the same member id will
> > remove
> > > > the
> > > > > existing  member and then create itself member?
> > > > >
> > > > > > Once the client receives this exception, it should consider the
> > > > exception
> > > > > as a serious error and stop the process, and the user will need to
> > > > assign a
> > > > > new memberId.
> > > > >
> > > > > Given that member id is the incarnation of consumer, it is hard to
> > > detect
> > > > > the scenario of "two consumers have the same member id" (it may be
> > > caused
> > > > > by the lost response). I guess the best approach is the later
> > -created
> > > > > consumer should be idle with fenced error, and we should add this
> to
> > > the
> > > > > KIP (maybe we should revise the error message carried by
> > > > > `FencedMemberEpochException`). BUT, we need to rethink the approach
> > if
> > > > > above description "The member is also removed from the group" is
> true
> > > > >
> > > > > Best,
> > > > > Chia-Ping
> > > > >
> > > > >
> > > > >
> > > > > TengYao Chi <ki...@gmail.com> 於 2024年9月20日 週五 下午8:17寫道：
> > > > >
> > > > > > Hi Chia-Ping,
> > > > > >
> > > > > > Thanks for pointing out this issue.
> > > > > >
> > > > > > I’m thinking that maybe we might need to define a new Exception
> to
> > > > handle
> > > > > > this scenario.
> > > > > > Once the client receives this exception, it should consider the
> > > > exception
> > > > > > as a serious error and stop the process, and the user will need
> to
> > > > > assign a
> > > > > > new memberId.
> > > > > > What do you think?
> > > > > >
> > > > > > Best Regards,
> > > > > > TengYao
> > > > > >
> > > > > > Chia-Ping Tsai <ch...@apache.org> 於 2024年9月20日 週五 下午7:00寫道：
> > > > > >
> > > > > > > > This part is not clear either. It basically says that if a
> > member
> > > > > joins
> > > > > > > with an existing member id but a different epoch, it will be
> > > fenced.
> > > > > Then
> > > > > > > it must rejoin with the same member id and epoch zero. This is
> > > > already
> > > > > > the
> > > > > > > current behavior and it does not help with detecting
> duplicates,
> > > > right?
> > > > > > >
> > > > > > > The duplicates issue is interesting. The member id is generated
> > on
> > > > > > > server-side by UUID before, so it seems to me the fenced member
> > > epoch
> > > > > > > happens only if the client miss the response with bumped epoch,
> > and
> > > > > this
> > > > > > > case is recoverable.
> > > > > > >
> > > > > > > however, v1 brings a different fenced epoch scenario. Users,
> now,
> > > > have
> > > > > > > responsibility to generate unique member id, hence clients may
> > > > > encounter
> > > > > > > infinite fenced error if there are >1 clients having same
> member
> > id
> > > > due
> > > > > > to
> > > > > > > config error (or other engineer error)
> > > > > > >
> > > > > > > Maybe we can highlight this scenario if we start to requires
> > > clients
> > > > to
> > > > > > > generate unique member id.
> > > > > > >
> > > > > > > Best,
> > > > > > > Chia-Ping
> > > > > > >
> > > > > > > On 2024/09/19 18:36:48 David Jacot wrote:
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > Thanks for the update. I have a few nits:
> > > > > > > >
> > > > > > > > > If the member ID is null or empty, the server will reject
> the
> > > > > request
> > > > > > > > with an InvalidRequestException.
> > > > > > > > We should clarify that this should only apply to version >=
> 1.
> > > > > > > >
> > > > > > > > > The consumer instance must generate a member ID, and this
> ID
> > > > should
> > > > > > > > remain consistent for the duration of the consumer's session.
> > > > Here, a
> > > > > > > > "session" is defined as the period from the consumer's first
> > > > > heartbeat
> > > > > > > > until it leaves the group, either through a graceful
> shutdown,
> > a
> > > > > > > heartbeat
> > > > > > > > timeout, or the process stopping or dying. The consumer
> > instance
> > > > > should
> > > > > > > > reuse the same member ID for all heartbeats and rejoin
> attempts
> > > to
> > > > > > > maintain
> > > > > > > > continuity within the group.
> > > > > > > >
> > > > > > > > This part is not clear to me. When the member leaves the
> group,
> > > it
> > > > > > should
> > > > > > > > not reset the member id. I would rather say that the member
> > must
> > > > > > generate
> > > > > > > > its member id when it starts and it must keep it until the
> > > process
> > > > > > stops.
> > > > > > > > It is basically an incarnation of the process.
> > > > > > > >
> > > > > > > > > If a conflict arises where the member ID generated by the
> > > client
> > > > is
> > > > > > > > detected to be a duplicate within the same group (for
> example,
> > > the
> > > > > same
> > > > > > > > member ID is associated with another active member in the
> > group),
> > > > the
> > > > > > > > server will handle this by comparing the memberEpoch values
> of
> > > the
> > > > > > > > conflicting members. The member with the lower memberEpoch is
> > > > > > considered
> > > > > > > > outdated and will be fenced off by the server. When this
> > occurs,
> > > > the
> > > > > > > server
> > > > > > > > responds with a FENCED_MEMBER_EPOCH error to the client,
> > > signaling
> > > > it
> > > > > > to
> > > > > > > > rejoin the group with the same member ID while resetting the
> > > > > > memberEpoch
> > > > > > > to
> > > > > > > > zero. This ensures that the client properly resynchronizes
> and
> > > > > > maintains
> > > > > > > > the continuity and consistency of the group membership.
> > > > > > > >
> > > > > > > > This part is not clear either. It basically says that if a
> > member
> > > > > joins
> > > > > > > > with an existing member id but a different epoch, it will be
> > > > fenced.
> > > > > > Then
> > > > > > > > it must rejoin with the same member id and epoch zero. This
> is
> > > > > already
> > > > > > > the
> > > > > > > > current behavior and it does not help with detecting
> > duplicates,
> > > > > right?
> > > > > > > > Should we just remove the paragraph?
> > > > > > > >
> > > > > > > > > A member ID mismatch occurs within a session: If the server
> > > > > detects a
> > > > > > > > mismatch between the provided member ID and the expected
> member
> > > ID
> > > > > for
> > > > > > an
> > > > > > > > ongoing session, it should return a UNKNOWN_MEMBER_ID  error.
> > > > > > > >
> > > > > > > > How could we detect a mismatch between the provided and the
> > > > expected
> > > > > > > member
> > > > > > > > id? My understanding is that we can only know whether the
> > > provided
> > > > > > member
> > > > > > > > id exists or not. This is already implemented.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > David
> > > > > > > >
> > > > > > > > On Sat, Sep 14, 2024 at 9:31 AM TengYao Chi <
> > kitingiao@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello everyone,
> > > > > > > > >
> > > > > > > > > Since this KIP has been fully discussed, I will initiate a
> > vote
> > > > for
> > > > > > it
> > > > > > > next
> > > > > > > > > Monday.
> > > > > > > > > Thank you and have a nice weekend.
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > TengYao
> > > > > > > > >
> > > > > > > > > TengYao Chi <ki...@gmail.com> 於 2024年9月5日 週四 下午2:19寫道：
> > > > > > > > >
> > > > > > > > > > Hello everyone,
> > > > > > > > > >
> > > > > > > > > > KT2: It looks like everyone who has expressed an opinion
> > > > supports
> > > > > > the
> > > > > > > > > > second option: “Document a recommendation for clients to
> > use
> > > > > UUIDs
> > > > > > as
> > > > > > > > > > member IDs, without strictly enforcing it.”
> > > > > > > > > > I have updated the KIP accordingly.
> > > > > > > > > > Please take a look, and let me know if you have any
> > thoughts
> > > or
> > > > > > > feedback.
> > > > > > > > > >
> > > > > > > > > > Thank you!
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > TengYao
> > > > > > > > > >
> > > > > > > > > > Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月30日 週五
> > > 下午9:56寫道：
> > > > > > > > > >
> > > > > > > > > >> hi TengYao
> > > > > > > > > >>
> > > > > > > > > >> KT2: +1 to second approach
> > > > > > > > > >>
> > > > > > > > > >> Best,
> > > > > > > > > >> Chia-Ping
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> David Jacot <dj...@confluent.io.invalid> 於 2024年8月30日
> 週五
> > > > > > 下午9:15寫道：
> > > > > > > > > >>
> > > > > > > > > >> > Hi TengYao,
> > > > > > > > > >> >
> > > > > > > > > >> > KT2: I don't think that we can realistically validate
> > the
> > > > uuid
> > > > > > on
> > > > > > > the
> > > > > > > > > >> > server. It is basically a string of chars. So I lean
> > > towards
> > > > > > > having a
> > > > > > > > > >> good
> > > > > > > > > >> > recommendation in the KIP and in the document of the
> > field
> > > > in
> > > > > > the
> > > > > > > > > RPC's
> > > > > > > > > >> > definition.
> > > > > > > > > >> >
> > > > > > > > > >> > Best,
> > > > > > > > > >> > David
> > > > > > > > > >> >
> > > > > > > > > >> > On Fri, Aug 30, 2024 at 3:02 PM TengYao Chi <
> > > > > > kitingiao@gmail.com>
> > > > > > > > > >> wrote:
> > > > > > > > > >> >
> > > > > > > > > >> > > Hello Kirk !
> > > > > > > > > >> > >
> > > > > > > > > >> > > Thank you for your comments !
> > > > > > > > > >> > >
> > > > > > > > > >> > > KT1: Yes, you are correct. The issue is not unique
> to
> > > the
> > > > > > > initial
> > > > > > > > > >> > > heartbeat; there can always be cases where the
> broker
> > > > might
> > > > > > lose
> > > > > > > > > >> > connection
> > > > > > > > > >> > > with a member.
> > > > > > > > > >> > >
> > > > > > > > > >> > > KT2: Currently, if the client doesn't have a member
> ID
> > > and
> > > > > the
> > > > > > > > > >> > memberEpoch
> > > > > > > > > >> > > equals 0, the coordinator will generate a UUID as
> the
> > > > member
> > > > > > ID
> > > > > > > for
> > > > > > > > > >> the
> > > > > > > > > >> > > client. However, at the RPC level, the member ID is
> > sent
> > > > as
> > > > > a
> > > > > > > > > literal
> > > > > > > > > >> > > string, meaning there are no restrictions on the
> > format
> > > at
> > > > > > this
> > > > > > > > > level.
> > > > > > > > > >> > > This also reminds me that we haven't reached a final
> > > > > > conclusion
> > > > > > > on
> > > > > > > > > >> how to
> > > > > > > > > >> > > enforce the use of UUIDs.
> > > > > > > > > >> > > From our previous discussions, I recall two possible
> > > > > > approaches:
> > > > > > > > > >> > > The first is to validate the UUID on the server
> side,
> > > and
> > > > if
> > > > > > > it's
> > > > > > > > > not
> > > > > > > > > >> > > valid, throw an exception to the client.
> > > > > > > > > >> > > The second is to document a recommendation for
> clients
> > > to
> > > > > use
> > > > > > > UUIDs
> > > > > > > > > as
> > > > > > > > > >> > > member IDs, without strictly enforcing it.
> > > > > > > > > >> > > I think it's time to decide on the approach we want
> to
> > > > take.
> > > > > > > > > >> > >
> > > > > > > > > >> > > KT3: Yes, "session" can be considered synonymous
> with
> > > > > > > "membership"
> > > > > > > > > in
> > > > > > > > > >> > this
> > > > > > > > > >> > > context.
> > > > > > > > > >> > >
> > > > > > > > > >> > > KT4: Thank you for pointing that out. I will update
> > the
> > > > > > wording
> > > > > > > to
> > > > > > > > > >> > > specifically say this behavior is for consumers.
> > > > > > > > > >> > >
> > > > > > > > > >> > > Thanks again for your comments.
> > > > > > > > > >> > >
> > > > > > > > > >> > > Best regards,
> > > > > > > > > >> > > TengYao
> > > > > > > > > >> > >
> > > > > > > > > >> > > Kirk True <ki...@kirktrue.pro> 於 2024年8月30日 週五
> > > 上午12:39寫道：
> > > > > > > > > >> > >
> > > > > > > > > >> > > > Hi TengYao!
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Sorry for being late to the discussion...
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > After reading the thread and then the KIP, I had a
> > few
> > > > > > > > > >> > > questions/comments:
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > KT1: In Motivation, it states: "This scenario can
> > > result
> > > > > in
> > > > > > > the
> > > > > > > > > >> broker
> > > > > > > > > >> > > > registering a new member for which it will never
> > > > receive a
> > > > > > > proper
> > > > > > > > > >> leave
> > > > > > > > > >> > > > request.” Just to be clear, the broker will always
> > > have
> > > > > > cases
> > > > > > > > > where
> > > > > > > > > >> it
> > > > > > > > > >> > > > might lose connection with a member. That’s not
> > unique
> > > > to
> > > > > > the
> > > > > > > > > >> initial
> > > > > > > > > >> > > > heartbeat, right?
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > KT2: There was a bit of back and forth about
> format
> > of
> > > > the
> > > > > > > member
> > > > > > > > > >> ID.
> > > > > > > > > >> > > From
> > > > > > > > > >> > > > what I gathered in the thread, the member ID is
> > still
> > > > > > defined
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > >> > RPC
> > > > > > > > > >> > > as
> > > > > > > > > >> > > > a string and not a UUID, right? The KIP states
> that
> > > the
> > > > > > > “client
> > > > > > > > > must
> > > > > > > > > >> > > > generate a UUID as the member ID” and that the
> > “server
> > > > > will
> > > > > > > > > validate
> > > > > > > > > >> > > that a
> > > > > > > > > >> > > > valid UUID is provided.” Is that a change for the
> > > > server,
> > > > > or
> > > > > > > is it
> > > > > > > > > >> > > already
> > > > > > > > > >> > > > enforced as a UUID?
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > KT3: Lianet mentioned some confusion over the use
> of
> > > the
> > > > > > word
> > > > > > > > > >> > “session.”
> > > > > > > > > >> > > > Isn’t “session” synonymous with “membership?”
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > KT4: Under “Member ID Lifecycle,” it states: "The
> > > client
> > > > > > > should
> > > > > > > > > >> reuse
> > > > > > > > > >> > the
> > > > > > > > > >> > > > same UUID as the member ID for all heartbeats and
> > > rejoin
> > > > > > > attempts
> > > > > > > > > to
> > > > > > > > > >> > > > maintain continuity within the group.” Could we
> > change
> > > > the
> > > > > > > first
> > > > > > > > > >> part
> > > > > > > > > >> > of
> > > > > > > > > >> > > > that to “The Consumer instance should…” We do have
> > > > > lifetimes
> > > > > > > that
> > > > > > > > > >> > extend
> > > > > > > > > >> > > > past the lifetime of a client instance (such as
> the
> > > > > > > transaction
> > > > > > > > > ID).
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Thanks,
> > > > > > > > > >> > > > Kirk
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > > On Aug 29, 2024, at 1:28 AM, TengYao Chi <
> > > > > > > kitingiao@gmail.com>
> > > > > > > > > >> > wrote:
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > Hi David,
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > Thank you for pointing that out.
> > > > > > > > > >> > > > > I have updated the content of the KIP based on
> > > > Lianet's
> > > > > > and
> > > > > > > your
> > > > > > > > > >> > > > feedback.
> > > > > > > > > >> > > > > Please take a look and let me know your
> thoughts.
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > Best regards,
> > > > > > > > > >> > > > > TengYao
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > David Jacot <dj...@confluent.io.invalid> 於
> > > > 2024年8月29日
> > > > > 週四
> > > > > > > > > >> 下午3:20寫道：
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > >> Hi TengYao,
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> Thanks for the update. I haven't fully read it
> > yet
> > > > but
> > > > > I
> > > > > > > will
> > > > > > > > > >> soon.
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> LM4: This is incorrect. The consumer must keep
> > its
> > > > > member
> > > > > > > id
> > > > > > > > > >> during
> > > > > > > > > >> > > its
> > > > > > > > > >> > > > >> entire lifetime (until the process stops or
> > dies).
> > > > The
> > > > > > > protocol
> > > > > > > > > >> > > > stipulates
> > > > > > > > > >> > > > >> that a member must rejoin with the same member
> id
> > > and
> > > > > the
> > > > > > > > > member
> > > > > > > > > >> > epoch
> > > > > > > > > >> > > > set
> > > > > > > > > >> > > > >> to zero when an FENCED_MEMBER_EPOCH occurs.
> This
> > > > allows
> > > > > > the
> > > > > > > > > >> member
> > > > > > > > > >> > to
> > > > > > > > > >> > > > >> resynchronize itself. We should not change this
> > > > > > behavior. I
> > > > > > > > > think
> > > > > > > > > >> > that
> > > > > > > > > >> > > > we
> > > > > > > > > >> > > > >> should see the client side generation id as an
> > > > > > incarnation
> > > > > > > id
> > > > > > > > > of
> > > > > > > > > >> the
> > > > > > > > > >> > > > >> application. It is generated once and kept
> until
> > it
> > > > > stops
> > > > > > > or
> > > > > > > > > >> dies.
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> Best,
> > > > > > > > > >> > > > >> David
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> On Thu, Aug 29, 2024 at 6:21 AM TengYao Chi <
> > > > > > > > > kitingiao@gmail.com
> > > > > > > > > >> >
> > > > > > > > > >> > > > wrote:
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >>> Hello Lianet !
> > > > > > > > > >> > > > >>>
> > > > > > > > > >> > > > >>> Thanks for the reviews and suggestions!
> > > > > > > > > >> > > > >>>
> > > > > > > > > >> > > > >>> LM1: Indeed, we plan to enforce client-side ID
> > > > > > generation
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > >> > > > future,
> > > > > > > > > >> > > > >>> and it is not an alternative. I will change
> the
> > > > title
> > > > > > > > > >> accordingly.
> > > > > > > > > >> > > > >>>
> > > > > > > > > >> > > > >>> LM2: Yes, that's the expectation. I will add
> > that
> > > > > > > statement to
> > > > > > > > > >> the
> > > > > > > > > >> > > > public
> > > > > > > > > >> > > > >>> interface section.
> > > > > > > > > >> > > > >>>
> > > > > > > > > >> > > > >>> LM3: Thank you for the high-level perspective
> > > > review.
> > > > > I
> > > > > > > think
> > > > > > > > > >> > you're
> > > > > > > > > >> > > > >> right;
> > > > > > > > > >> > > > >>> our intention isn't very clear since it was
> > placed
> > > > at
> > > > > > the
> > > > > > > end
> > > > > > > > > of
> > > > > > > > > >> > the
> > > > > > > > > >> > > > >>> section. I will try to rephrase that section
> to
> > > make
> > > > > it
> > > > > > > more
> > > > > > > > > >> > obvious.
> > > > > > > > > >> > > > >>>
> > > > > > > > > >> > > > >>> LM4: Regarding the definition of "session" in
> > this
> > > > > KIP,
> > > > > > I
> > > > > > > > > >> believe
> > > > > > > > > >> > it
> > > > > > > > > >> > > > >> refers
> > > > > > > > > >> > > > >>> to the period between the *first-time
> heartbeat*
> > > and
> > > > > > when
> > > > > > > the
> > > > > > > > > >> > > *consumer
> > > > > > > > > >> > > > >>> leaves the group* (whether through a graceful
> > > > shutdown
> > > > > > or
> > > > > > > a
> > > > > > > > > >> > heartbeat
> > > > > > > > > >> > > > >>> timeout). The consumer should reuse its UUID
> if
> > it
> > > > has
> > > > > > > been
> > > > > > > > > >> > generated
> > > > > > > > > >> > > > >>> before. The only situation in which it will
> > > > regenerate
> > > > > > the
> > > > > > > > > UUID
> > > > > > > > > >> is
> > > > > > > > > >> > if
> > > > > > > > > >> > > > the
> > > > > > > > > >> > > > >>> coordinator finds that there is already a
> > consumer
> > > > > with
> > > > > > > the
> > > > > > > > > same
> > > > > > > > > >> > > UUID.
> > > > > > > > > >> > > > >>> IIRC, the coordinator should compare the
> member
> > > > > epochs,
> > > > > > > and
> > > > > > > > > the
> > > > > > > > > >> > > > >>> later-joined consumer should be fenced off by
> > the
> > > > > > > coordinator
> > > > > > > > > >> due
> > > > > > > > > >> > to
> > > > > > > > > >> > > > >> having
> > > > > > > > > >> > > > >>> a lower member epoch. Once the consumer
> > receives a
> > > > > > > > > >> > > > `FENCED_MEMBER_EPOCH`
> > > > > > > > > >> > > > >>> error, it will generate a new UUID and attempt
> > to
> > > > > > rejoin.
> > > > > > > I
> > > > > > > > > will
> > > > > > > > > >> > > > clarify
> > > > > > > > > >> > > > >>> this in the KIP.
> > > > > > > > > >> > > > >>>
> > > > > > > > > >> > > > >>> Thanks again for your reviews, I really
> > appreciate
> > > > it.
> > > > > > > > > >> > > > >>>
> > > > > > > > > >> > > > >>> Best regards,
> > > > > > > > > >> > > > >>> TengYao
> > > > > > > > > >> > > > >>>
> > > > > > > > > >> > > > >>> Lianet M. <li...@gmail.com> 於 2024年8月28日
> 週三
> > > > > > 下午7:12寫道：
> > > > > > > > > >> > > > >>>
> > > > > > > > > >> > > > >>>> Hello TengYao! Thanks for taking on this
> issue,
> > > > we've
> > > > > > > been
> > > > > > > > > >> going
> > > > > > > > > >> > > > around
> > > > > > > > > >> > > > >>> it
> > > > > > > > > >> > > > >>>> for a while.
> > > > > > > > > >> > > > >>>>
> > > > > > > > > >> > > > >>>> LM1: About the title of the KIP: "Enable ID
> > > > > Generation
> > > > > > > for
> > > > > > > > > >> Clients
> > > > > > > > > >> > > > over
> > > > > > > > > >> > > > >>> the
> > > > > > > > > >> > > > >>>> ConsumerGroupHeartbeat RPC". I find it
> > confusing
> > > > > > because
> > > > > > > it
> > > > > > > > > >> hints
> > > > > > > > > >> > > that
> > > > > > > > > >> > > > >>>> we're adding it as an alternative (which was
> > > > > discussed
> > > > > > > and
> > > > > > > > > >> > > discarded,
> > > > > > > > > >> > > > >> in
> > > > > > > > > >> > > > >>>> favour of really enforcing it). It's also
> > missing
> > > > the
> > > > > > > core
> > > > > > > > > >> change
> > > > > > > > > >> > > imo,
> > > > > > > > > >> > > > >>>> which is "where" the generation happens. So,
> > > maybe
> > > > > more
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > >> > point
> > > > > > > > > >> > > > >> with
> > > > > > > > > >> > > > >>>> something along the lines of "Client-side
> > > generated
> > > > > ID
> > > > > > > for
> > > > > > > > > >> clients
> > > > > > > > > >> > > > over
> > > > > > > > > >> > > > >>>> ConsumerGroupHeartbeat RPC"?
> > > > > > > > > >> > > > >>>>
> > > > > > > > > >> > > > >>>> LM2: On the public interfaces section, the
> KIP
> > > > states
> > > > > > > that
> > > > > > > > > "the
> > > > > > > > > >> > > server
> > > > > > > > > >> > > > >>> will
> > > > > > > > > >> > > > >>>> reject the request", but we should agree on
> the
> > > > > > specific
> > > > > > > > > error
> > > > > > > > > >> > > type. I
> > > > > > > > > >> > > > >>>> expect it should fail with an
> > > > > InvalidRequestException,
> > > > > > is
> > > > > > > > > that
> > > > > > > > > >> the
> > > > > > > > > >> > > > >>>> intention? (This was also suggested in the
> > > > discussion
> > > > > > > thread
> > > > > > > > > >> > before
> > > > > > > > > >> > > > but
> > > > > > > > > >> > > > >>> is
> > > > > > > > > >> > > > >>>> not in the KIP).
> > > > > > > > > >> > > > >>>>
> > > > > > > > > >> > > > >>>> LM3. Related to my previous point, I find
> that
> > to
> > > > be
> > > > > > the
> > > > > > > true
> > > > > > > > > >> > > > >>> public-facing
> > > > > > > > > >> > > > >>>> change (member ID mandatory at the protocol
> > > level),
> > > > > but
> > > > > > > it's
> > > > > > > > > >> only
> > > > > > > > > >> > at
> > > > > > > > > >> > > > >> the
> > > > > > > > > >> > > > >>>> end of the Public interfaces changes, kind of
> > > lost
> > > > > > among
> > > > > > > > > >> details
> > > > > > > > > >> > of
> > > > > > > > > >> > > > how
> > > > > > > > > >> > > > >>>> we're going to do it. Should we rephrase that
> > > > section
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > >> > > actual
> > > > > > > > > >> > > > >>>> change first, and the hows after (ex. Bumping
> > the
> > > > > > > version is
> > > > > > > > > >> not
> > > > > > > > > >> > the
> > > > > > > > > >> > > > >>>> public-facing change in this case, it's just
> > the
> > > > > > > mechanism to
> > > > > > > > > >> > > properly
> > > > > > > > > >> > > > >>>> introduce our change)
> > > > > > > > > >> > > > >>>>
> > > > > > > > > >> > > > >>>> LM4. Regarding the lifetime of the UUID: the
> > KIP
> > > > > states
> > > > > > > we
> > > > > > > > > will
> > > > > > > > > >> > > > "Verify
> > > > > > > > > >> > > > >>>> that the UUID remains consistent across all
> > > > > subsequent
> > > > > > > > > >> heartbeats
> > > > > > > > > >> > > > >> during
> > > > > > > > > >> > > > >>>> the session". What is this "session"
> referring
> > to
> > > > > > here? I
> > > > > > > > > would
> > > > > > > > > >> > > expect
> > > > > > > > > >> > > > >>> that
> > > > > > > > > >> > > > >>>> the UUID is associated to a consumer instance
> > > > > > (generated
> > > > > > > for
> > > > > > > > > >> the
> > > > > > > > > >> > > > >> consumer
> > > > > > > > > >> > > > >>>> the first time it needs to send a HB if it
> > > doesn't
> > > > > have
> > > > > > > the
> > > > > > > > > >> UUID
> > > > > > > > > >> > > yet.
> > > > > > > > > >> > > > >>> From
> > > > > > > > > >> > > > >>>> there on, every time it needs to send a
> "first
> > > HB"
> > > > > > > again, it
> > > > > > > > > >> will
> > > > > > > > > >> > > > reuse
> > > > > > > > > >> > > > >>> its
> > > > > > > > > >> > > > >>>> UUID, is that the intention? Note that we
> > should
> > > > > > consider
> > > > > > > > > that
> > > > > > > > > >> the
> > > > > > > > > >> > > > same
> > > > > > > > > >> > > > >>>> consumer instance may have many "first
> > > heartbeats",
> > > > > > > meaning
> > > > > > > > > >> > > heartbeats
> > > > > > > > > >> > > > >> to
> > > > > > > > > >> > > > >>>> join the group when it's not part of it (ex.
> > > > consumer
> > > > > > > > > >> unsubscribe
> > > > > > > > > >> > +
> > > > > > > > > >> > > > >>>> subscribe, fenced, stale). Is this the
> > intention
> > > or
> > > > > are
> > > > > > > you
> > > > > > > > > >> > > > considering
> > > > > > > > > >> > > > >>> the
> > > > > > > > > >> > > > >>>> lifetime differently? We should clarify it in
> > the
> > > > > KIP.
> > > > > > > > > >> > > > >>>>
> > > > > > > > > >> > > > >>>> Thanks!
> > > > > > > > > >> > > > >>>>
> > > > > > > > > >> > > > >>>> Lianet
> > > > > > > > > >> > > > >>>>
> > > > > > > > > >> > > > >>>> On Tue, Aug 27, 2024 at 2:27 AM TengYao Chi <
> > > > > > > > > >> kitingiao@gmail.com>
> > > > > > > > > >> > > > >> wrote:
> > > > > > > > > >> > > > >>>>
> > > > > > > > > >> > > > >>>>> Hi everyone,
> > > > > > > > > >> > > > >>>>>
> > > > > > > > > >> > > > >>>>> I have revised this KIP multiple times based
> > on
> > > > the
> > > > > > > feedback
> > > > > > > > > >> from
> > > > > > > > > >> > > our
> > > > > > > > > >> > > > >>>>> discussions.
> > > > > > > > > >> > > > >>>>> I would greatly appreciate it if you could
> > > review
> > > > it
> > > > > > > when
> > > > > > > > > you
> > > > > > > > > >> > have
> > > > > > > > > >> > > > >> the
> > > > > > > > > >> > > > >>>>> time.
> > > > > > > > > >> > > > >>>>> If there are no further comments or
> > > suggestions, I
> > > > > > plan
> > > > > > > to
> > > > > > > > > >> > proceed
> > > > > > > > > >> > > > >> with
> > > > > > > > > >> > > > >>>>> initiating a vote soon.
> > > > > > > > > >> > > > >>>>>
> > > > > > > > > >> > > > >>>>> Best regards,
> > > > > > > > > >> > > > >>>>> TengYao
> > > > > > > > > >> > > > >>>>>
> > > > > > > > > >> > > > >>>>> TengYao Chi <ki...@gmail.com> 於
> > 2024年8月23日
> > > 週五
> > > > > > > 下午2:43寫道：
> > > > > > > > > >> > > > >>>>>
> > > > > > > > > >> > > > >>>>>> Hi Andrew,
> > > > > > > > > >> > > > >>>>>> Thank you for your previous feedback and
> > > > insights.
> > > > > > > > > >> > > > >>>>>> Your contributions have added valuable
> > > > perspectives
> > > > > > to
> > > > > > > the
> > > > > > > > > >> > > > >>> discussions.
> > > > > > > > > >> > > > >>>>>> And we also benefit from the comparison of
> > > > > different
> > > > > > > > > >> solutions.
> > > > > > > > > >> > > > >>>>>> I’m also looking forward to seeing an
> initial
> > > > > version
> > > > > > > in
> > > > > > > > > >> > KIP-932,
> > > > > > > > > >> > > > >> as
> > > > > > > > > >> > > > >>> it
> > > > > > > > > >> > > > >>>>>> will provide a good reference for future
> > > > > > > implementations.
> > > > > > > > > >> > > > >>>>>>
> > > > > > > > > >> > > > >>>>>> Regarding your comment on AS2, I wanted to
> > > > clarify
> > > > > > > that my
> > > > > > > > > >> > > > >>>> specification
> > > > > > > > > >> > > > >>>>>> references org.apache.kafka.common.Uuid.
> > > > > > > > > >> > > > >>>>>> I believe we’re referring to the same
> class,
> > > and
> > > > it
> > > > > > > might
> > > > > > > > > >> just
> > > > > > > > > >> > be
> > > > > > > > > >> > > a
> > > > > > > > > >> > > > >>>> small
> > > > > > > > > >> > > > >>>>>> oversight due to the busy schedule.
> > > > > > > > > >> > > > >>>>>>
> > > > > > > > > >> > > > >>>>>> I want to express my gratitude once again
> for
> > > > your
> > > > > > many
> > > > > > > > > >> > insightful
> > > > > > > > > >> > > > >>>>>> comments, which have helped the discussion
> > > > progress
> > > > > > > > > smoothly.
> > > > > > > > > >> > > > >>>>>>
> > > > > > > > > >> > > > >>>>>> Best regards,
> > > > > > > > > >> > > > >>>>>> TengYao
> > > > > > > > > >> > > > >>>>>>
> > > > > > > > > >> > > > >>>>>>
> > > > > > > > > >> > > > >>>>>> Andrew Schofield <
> andrew_schofield@live.com>
> > 於
> > > > > > > 2024年8月22日
> > > > > > > > > 週四
> > > > > > > > > >> > > > >>>> 下午11:28寫道：
> > > > > > > > > >> > > > >>>>>>
> > > > > > > > > >> > > > >>>>>>> Hi TengYao,
> > > > > > > > > >> > > > >>>>>>> I’ve been reading through the comments and
> > I’m
> > > > > happy
> > > > > > > that
> > > > > > > > > >> the
> > > > > > > > > >> > > > >> lobby
> > > > > > > > > >> > > > >>>>>>> approach has not gained support.
> > > > > > > > > >> > > > >>>>>>>
> > > > > > > > > >> > > > >>>>>>> Assuming that this KIP is voted, I will be
> > > happy
> > > > > to
> > > > > > > change
> > > > > > > > > >> > > KIP-932
> > > > > > > > > >> > > > >>> so
> > > > > > > > > >> > > > >>>>>>> that it only supports client-side member
> ID
> > > > > > > generation.
> > > > > > > > > >> Because
> > > > > > > > > >> > > > >> that
> > > > > > > > > >> > > > >>>> KIP
> > > > > > > > > >> > > > >>>>>>> is still
> > > > > > > > > >> > > > >>>>>>> under development, I can do this in the
> > first
> > > > > > version
> > > > > > > of
> > > > > > > > > >> > > > >>>>>>> ShareGroupHeartbeat.
> > > > > > > > > >> > > > >>>>>>>
> > > > > > > > > >> > > > >>>>>>> AS2: For the encoding section, I suppose
> the
> > > > > > specific
> > > > > > > > > >> encoding
> > > > > > > > > >> > > > >> which
> > > > > > > > > >> > > > >>>>>>> is used is what
> org.apache.kafka.utils.Uuid
> > > > uses.
> > > > > > > > > >> > > > >>>>>>>
> > > > > > > > > >> > > > >>>>>>> Thanks,
> > > > > > > > > >> > > > >>>>>>> Andrew
> > > > > > > > > >> > > > >>>>>>>
> > > > > > > > > >> > > > >>>>>>>> On 14 Aug 2024, at 17:00, TengYao Chi <
> > > > > > > > > kitingiao@gmail.com
> > > > > > > > > >> >
> > > > > > > > > >> > > > >>> wrote:
> > > > > > > > > >> > > > >>>>>>>>
> > > > > > > > > >> > > > >>>>>>>> Hello Apoorv,
> > > > > > > > > >> > > > >>>>>>>> Thank you for your feedback.
> > > > > > > > > >> > > > >>>>>>>> Regarding the questions you raised,
> > > > > unfortunately,
> > > > > > > this
> > > > > > > > > KIP
> > > > > > > > > >> > > > >> cannot
> > > > > > > > > >> > > > >>>>>>>> guarantee the order of heartbeats. As
> with
> > > many
> > > > > > > classic
> > > > > > > > > >> > > > >>> distributed
> > > > > > > > > >> > > > >>>>>>> system
> > > > > > > > > >> > > > >>>>>>>> challenges, what we can do is make our
> best
> > > > > effort
> > > > > > to
> > > > > > > > > >> ensure
> > > > > > > > > >> > > > >> that
> > > > > > > > > >> > > > >>>>> there
> > > > > > > > > >> > > > >>>>>>> are
> > > > > > > > > >> > > > >>>>>>>> no idle members or stale assignments
> under
> > > > normal
> > > > > > > > > >> > circumstances.
> > > > > > > > > >> > > > >>>>>>>>
> > > > > > > > > >> > > > >>>>>>>> As for the lobby approach, I’m not a fan
> of
> > > it
> > > > > > > because it
> > > > > > > > > >> > > > >> requires
> > > > > > > > > >> > > > >>>>>>> adding a
> > > > > > > > > >> > > > >>>>>>>> mechanism to maintain client state within
> > the
> > > > > > > > > >> ConsumerGroup,
> > > > > > > > > >> > > > >>> which,
> > > > > > > > > >> > > > >>>> in
> > > > > > > > > >> > > > >>>>>>> my
> > > > > > > > > >> > > > >>>>>>>> view, resembles something like a
> two-phase
> > > > > commit.
> > > > > > > This
> > > > > > > > > >> would
> > > > > > > > > >> > > > >>>>> introduce
> > > > > > > > > >> > > > >>>>>>>> more complexity than the proposal in this
> > > KIP,
> > > > > > which
> > > > > > > is
> > > > > > > > > >> > > > >> something
> > > > > > > > > >> > > > >>> we
> > > > > > > > > >> > > > >>>>>>> want
> > > > > > > > > >> > > > >>>>>>>> to avoid. KIP-848 aims to simplify the
> > > existing
> > > > > > > protocol,
> > > > > > > > > >> and
> > > > > > > > > >> > > > >>> while
> > > > > > > > > >> > > > >>>>> the
> > > > > > > > > >> > > > >>>>>>>> lobby approach is a good one, I believe
> it
> > is
> > > > not
> > > > > > the
> > > > > > > > > right
> > > > > > > > > >> > fit
> > > > > > > > > >> > > > >>> for
> > > > > > > > > >> > > > >>>>> this
> > > > > > > > > >> > > > >>>>>>>> particular situation.
> > > > > > > > > >> > > > >>>>>>>>
> > > > > > > > > >> > > > >>>>>>>> Best regards,
> > > > > > > > > >> > > > >>>>>>>> TengYao
> > > > > > > > > >> > > > >>>>>>>>
> > > > > > > > > >> > > > >>>>>>>> TengYao Chi <ki...@gmail.com> 於
> > > 2024年8月14日
> > > > > 週三
> > > > > > > > > >> 下午11:45寫道：
> > > > > > > > > >> > > > >>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>> Hi David,
> > > > > > > > > >> > > > >>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>> I really appreciate your review and
> > > > suggestions.
> > > > > > As
> > > > > > > I am
> > > > > > > > > >> > still
> > > > > > > > > >> > > > >>>>> gaining
> > > > > > > > > >> > > > >>>>>>>>> experience in writing KIPs, your input
> has
> > > > been
> > > > > > > > > incredibly
> > > > > > > > > >> > > > >>>> helpful. I
> > > > > > > > > >> > > > >>>>>>> am
> > > > > > > > > >> > > > >>>>>>>>> currently applying your suggestions to
> the
> > > KIP
> > > > > and
> > > > > > > will
> > > > > > > > > >> > > > >> complete
> > > > > > > > > >> > > > >>> it
> > > > > > > > > >> > > > >>>>> as
> > > > > > > > > >> > > > >>>>>>> soon
> > > > > > > > > >> > > > >>>>>>>>> as possible.
> > > > > > > > > >> > > > >>>>>>>>> Regarding the UUID part, I think we
> > haven’t
> > > > > > reached
> > > > > > > a
> > > > > > > > > >> > > > >> conclusion
> > > > > > > > > >> > > > >>>>>>> yet.(So
> > > > > > > > > >> > > > >>>>>>>>> far according to this thread)
> > > > > > > > > >> > > > >>>>>>>>> However, I will review the current
> > > > > implementation
> > > > > > > in the
> > > > > > > > > >> > Kafka
> > > > > > > > > >> > > > >>>> `Uuid`
> > > > > > > > > >> > > > >>>>>>>>> class and include a brief specification
> in
> > > the
> > > > > > KIP.
> > > > > > > > > >> > > > >>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>> Once again, thank you so much for your
> > help.
> > > > > > > > > >> > > > >>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>> Best regards,
> > > > > > > > > >> > > > >>>>>>>>> TengYao
> > > > > > > > > >> > > > >>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> > > > > 2024年8月14日
> > > > > > 週三
> > > > > > > > > >> > 下午11:14寫道：
> > > > > > > > > >> > > > >>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>> hi Apoorv
> > > > > > > > > >> > > > >>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>> As the memberId is now known to the
> > > client,
> > > > > and
> > > > > > > client
> > > > > > > > > >> > might
> > > > > > > > > >> > > > >>> send
> > > > > > > > > >> > > > >>>>> the
> > > > > > > > > >> > > > >>>>>>>>>> leave
> > > > > > > > > >> > > > >>>>>>>>>> group heartbeat on shutdown prior to
> > > > receiving
> > > > > > the
> > > > > > > > > >> initial
> > > > > > > > > >> > > > >>>> heartbeat
> > > > > > > > > >> > > > >>>>>>>>>> response. If that's true then how do we
> > > > > guarantee
> > > > > > > that
> > > > > > > > > >> the 2
> > > > > > > > > >> > > > >>>>> requests
> > > > > > > > > >> > > > >>>>>>> to
> > > > > > > > > >> > > > >>>>>>>>>> join and leave will be processed in
> > order,
> > > > > which
> > > > > > > could
> > > > > > > > > >> still
> > > > > > > > > >> > > > >>> leave
> > > > > > > > > >> > > > >>>>>>> stale
> > > > > > > > > >> > > > >>>>>>>>>> members or throw unknown member id
> > > > exceptions?
> > > > > > > > > >> > > > >>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>> This is definitely a good question. the
> > > short
> > > > > > > answer:
> > > > > > > > > no
> > > > > > > > > >> > > > >>> guarantee
> > > > > > > > > >> > > > >>>>> but
> > > > > > > > > >> > > > >>>>>>>>>> best
> > > > > > > > > >> > > > >>>>>>>>>> efforts
> > > > > > > > > >> > > > >>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>> Please notice the root cause is "we
> have
> > no
> > > > > > enough
> > > > > > > time
> > > > > > > > > >> to
> > > > > > > > > >> > > > >> wait
> > > > > > > > > >> > > > >>>>>>> member id
> > > > > > > > > >> > > > >>>>>>>>>> (response) when closing consumer".
> Sadly,
> > > we
> > > > > can'
> > > > > > > > > >> guarantee
> > > > > > > > > >> > > > >> the
> > > > > > > > > >> > > > >>>>>>> request
> > > > > > > > > >> > > > >>>>>>>>>> order due to the same reason.
> > > > > > > > > >> > > > >>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>> However, in contrast to previous
> > behavior,
> > > > > there
> > > > > > > is one
> > > > > > > > > >> big
> > > > > > > > > >> > > > >>>> benefit
> > > > > > > > > >> > > > >>>>>>> of new
> > > > > > > > > >> > > > >>>>>>>>>> approach - we can try STONITH because
> we
> > > know
> > > > > the
> > > > > > > > > member
> > > > > > > > > >> id
> > > > > > > > > >> > > > >>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>> Best,
> > > > > > > > > >> > > > >>>>>>>>>> Chia-Ping
> > > > > > > > > >> > > > >>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>> Apoorv Mittal <
> apoorvmittal10@gmail.com>
> > 於
> > > > > > > 2024年8月14日
> > > > > > > > > 週三
> > > > > > > > > >> > > > >>>> 下午8:55寫道：
> > > > > > > > > >> > > > >>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>> Hi TengYao,
> > > > > > > > > >> > > > >>>>>>>>>>> Thanks for the KIP. Continuing on the
> > > point
> > > > > > which
> > > > > > > > > Andrew
> > > > > > > > > >> > > > >>>> mentioned
> > > > > > > > > >> > > > >>>>> as
> > > > > > > > > >> > > > >>>>>>>>>> AS1.
> > > > > > > > > >> > > > >>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>> As the memberId is now known to the
> > > client,
> > > > > and
> > > > > > > client
> > > > > > > > > >> > might
> > > > > > > > > >> > > > >>> send
> > > > > > > > > >> > > > >>>>> the
> > > > > > > > > >> > > > >>>>>>>>>> leave
> > > > > > > > > >> > > > >>>>>>>>>>> group heartbeat on shutdown prior to
> > > > receiving
> > > > > > the
> > > > > > > > > >> initial
> > > > > > > > > >> > > > >>>>> heartbeat
> > > > > > > > > >> > > > >>>>>>>>>>> response. If that's true then how do
> we
> > > > > > guarantee
> > > > > > > that
> > > > > > > > > >> the
> > > > > > > > > >> > 2
> > > > > > > > > >> > > > >>>>>>> requests to
> > > > > > > > > >> > > > >>>>>>>>>>> join and leave will be processed in
> > order,
> > > > > which
> > > > > > > could
> > > > > > > > > >> > still
> > > > > > > > > >> > > > >>>> leave
> > > > > > > > > >> > > > >>>>>>> stale
> > > > > > > > > >> > > > >>>>>>>>>>> members or throw unknown member id
> > > > exceptions?
> > > > > > > > > >> > > > >>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>> Though the client side member id
> > > generation
> > > > is
> > > > > > > helpful
> > > > > > > > > >> > which
> > > > > > > > > >> > > > >>> will
> > > > > > > > > >> > > > >>>>>>>>>> represent
> > > > > > > > > >> > > > >>>>>>>>>>> the same group perspective as from
> > client
> > > > and
> > > > > > > broker's
> > > > > > > > > >> end.
> > > > > > > > > >> > > > >>> But I
> > > > > > > > > >> > > > >>>>>>> think
> > > > > > > > > >> > > > >>>>>>>>>> the
> > > > > > > > > >> > > > >>>>>>>>>>> major concern we want to solve here is
> > > Stale
> > > > > > > Partition
> > > > > > > > > >> > > > >>>> Assignments
> > > > > > > > > >> > > > >>>>>>> which
> > > > > > > > > >> > > > >>>>>>>>>>> might still exist with the new
> > approach. I
> > > > am
> > > > > > > leaning
> > > > > > > > > >> > towards
> > > > > > > > > >> > > > >>> the
> > > > > > > > > >> > > > >>>>>>>>>>> suggestion mentioned by Andrew where
> > > > partition
> > > > > > > > > >> assignment
> > > > > > > > > >> > > > >>>> triggers
> > > > > > > > > >> > > > >>>>> on
> > > > > > > > > >> > > > >>>>>>>>>>> subsequent heartbeat when client
> > > > acknowledges
> > > > > > the
> > > > > > > > > >> initial
> > > > > > > > > >> > > > >>>>> heartbeat,
> > > > > > > > > >> > > > >>>>>>>>>>> delayed partition assignment.
> > > > > > > > > >> > > > >>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>> Though on a separate note, I have a
> > > > different
> > > > > > > > > question.
> > > > > > > > > >> > What
> > > > > > > > > >> > > > >>>>> happens
> > > > > > > > > >> > > > >>>>>>>>>> when
> > > > > > > > > >> > > > >>>>>>>>>>> there is an issue with the client
> which
> > > > sends
> > > > > > the
> > > > > > > > > >> initial
> > > > > > > > > >> > > > >>>> heartbeat
> > > > > > > > > >> > > > >>>>>>>>>> without
> > > > > > > > > >> > > > >>>>>>>>>>> memberId, then crashes and restarts? I
> > > think
> > > > > we
> > > > > > > must
> > > > > > > > > be
> > > > > > > > > >> > > > >>>>> re-triggering
> > > > > > > > > >> > > > >>>>>>>>>>> assignments and expiring members only
> > > after
> > > > > the
> > > > > > > > > >> heartbeat
> > > > > > > > > >> > > > >>> session
> > > > > > > > > >> > > > >>>>>>>>>> timeout?
> > > > > > > > > >> > > > >>>>>>>>>>> If that's true then shall delayed
> > > partition
> > > > > > > assignment
> > > > > > > > > >> can
> > > > > > > > > >> > > > >> help
> > > > > > > > > >> > > > >>>>>>> benefit
> > > > > > > > > >> > > > >>>>>>>>>> us
> > > > > > > > > >> > > > >>>>>>>>>>> from this situation as well?
> > > > > > > > > >> > > > >>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>> Regards,
> > > > > > > > > >> > > > >>>>>>>>>>> Apoorv Mittal
> > > > > > > > > >> > > > >>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>> On Wed, Aug 14, 2024 at 12:51 PM David
> > > Jacot
> > > > > > > > > >> > > > >>>>>>>>>> <dj...@confluent.io.invalid>
> > > > > > > > > >> > > > >>>>>>>>>>> wrote:
> > > > > > > > > >> > > > >>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>> Hi Andrew,
> > > > > > > > > >> > > > >>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>> Personally, I don't like the lobby
> > > > approach.
> > > > > It
> > > > > > > makes
> > > > > > > > > >> > things
> > > > > > > > > >> > > > >>>> more
> > > > > > > > > >> > > > >>>>>>>>>>>> complicated and it would require
> > changing
> > > > the
> > > > > > > records
> > > > > > > > > >> on
> > > > > > > > > >> > the
> > > > > > > > > >> > > > >>>>> server
> > > > > > > > > >> > > > >>>>>>>>>> too.
> > > > > > > > > >> > > > >>>>>>>>>>>> This is why I initially suggested the
> > > > > rejected
> > > > > > > > > >> alternative
> > > > > > > > > >> > > > >> #2
> > > > > > > > > >> > > > >>>>> which
> > > > > > > > > >> > > > >>>>>>> is
> > > > > > > > > >> > > > >>>>>>>>>>>> pretty close but also not perfect.
> > > > > > > > > >> > > > >>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>> I'd like to clarify one thing. The
> > > > > > > > > >> ConsumerGroupHeartbeat
> > > > > > > > > >> > > > >> API
> > > > > > > > > >> > > > >>>>>>> already
> > > > > > > > > >> > > > >>>>>>>>>>>> supports generating the member id on
> > the
> > > > > client
> > > > > > > so we
> > > > > > > > > >> > don't
> > > > > > > > > >> > > > >>> need
> > > > > > > > > >> > > > >>>>> any
> > > > > > > > > >> > > > >>>>>>>>>>>> conditional logic on the client side.
> > > This
> > > > is
> > > > > > > > > actually
> > > > > > > > > >> > what
> > > > > > > > > >> > > > >> we
> > > > > > > > > >> > > > >>>>>>> wanted
> > > > > > > > > >> > > > >>>>>>>>>> to
> > > > > > > > > >> > > > >>>>>>>>>>> do
> > > > > > > > > >> > > > >>>>>>>>>>>> in the first place but the idea got
> > > pushed
> > > > > back
> > > > > > > by
> > > > > > > > > >> Magnus
> > > > > > > > > >> > > > >> back
> > > > > > > > > >> > > > >>>>> then
> > > > > > > > > >> > > > >>>>>>>>>>> because
> > > > > > > > > >> > > > >>>>>>>>>>>> generating uuid from librdkafka
> > required
> > > a
> > > > > new
> > > > > > > > > >> dependency.
> > > > > > > > > >> > > > >> It
> > > > > > > > > >> > > > >>>>> turns
> > > > > > > > > >> > > > >>>>>>>>>> out
> > > > > > > > > >> > > > >>>>>>>>>>>> that librdkafka has that dependency
> > > today.
> > > > In
> > > > > > > > > >> retrospect,
> > > > > > > > > >> > we
> > > > > > > > > >> > > > >>>>> should
> > > > > > > > > >> > > > >>>>>>>>>> have
> > > > > > > > > >> > > > >>>>>>>>>>>> pushed back on this. Long story
> short,
> > we
> > > > can
> > > > > > > just do
> > > > > > > > > >> it.
> > > > > > > > > >> > > > >> The
> > > > > > > > > >> > > > >>>>>>>>>> proposal in
> > > > > > > > > >> > > > >>>>>>>>>>>> this KIP is to make the member id
> > > required
> > > > in
> > > > > > > future
> > > > > > > > > >> > > > >> versions.
> > > > > > > > > >> > > > >>>> We
> > > > > > > > > >> > > > >>>>>>>>>> could
> > > > > > > > > >> > > > >>>>>>>>>>>> also decide not to do it and to keep
> > > > > supporting
> > > > > > > both
> > > > > > > > > >> > > > >>>> approaches. I
> > > > > > > > > >> > > > >>>>>>>>>> would
> > > > > > > > > >> > > > >>>>>>>>>>>> also be fine with this.
> > > > > > > > > >> > > > >>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>> Best,
> > > > > > > > > >> > > > >>>>>>>>>>>> David
> > > > > > > > > >> > > > >>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>> On Wed, Aug 14, 2024 at 12:30 PM
> Andrew
> > > > > > > Schofield <
> > > > > > > > > >> > > > >>>>>>>>>>>> andrew_schofield@live.com>
> > > > > > > > > >> > > > >>>>>>>>>>>> wrote:
> > > > > > > > > >> > > > >>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>> Hi TengYao,
> > > > > > > > > >> > > > >>>>>>>>>>>>> Thanks for your response. I’ll have
> > just
> > > > one
> > > > > > > more
> > > > > > > > > try
> > > > > > > > > >> to
> > > > > > > > > >> > > > >>>>> persuade.
> > > > > > > > > >> > > > >>>>>>>>>>>>> I feel that I will need to follow
> the
> > > > > approach
> > > > > > > with
> > > > > > > > > >> > KIP-932
> > > > > > > > > >> > > > >>>> when
> > > > > > > > > >> > > > >>>>>>>>>> we’ve
> > > > > > > > > >> > > > >>>>>>>>>>>>> made a decision, so I do have more
> > than
> > > a
> > > > > > > passing
> > > > > > > > > >> > interest
> > > > > > > > > >> > > > >> in
> > > > > > > > > >> > > > >>>>> this.
> > > > > > > > > >> > > > >>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>> A group member in the lobby is in
> the
> > > > group,
> > > > > > > but it
> > > > > > > > > >> does
> > > > > > > > > >> > > > >> not
> > > > > > > > > >> > > > >>>> have
> > > > > > > > > >> > > > >>>>>>>>>> any
> > > > > > > > > >> > > > >>>>>>>>>>>>> assignments. A member of a consumer
> > > group
> > > > > can
> > > > > > > have
> > > > > > > > > no
> > > > > > > > > >> > > > >>> assigned
> > > > > > > > > >> > > > >>>>>>>>>>>>> partitions (such as 5 CG members
> > > > subscribed
> > > > > > to a
> > > > > > > > > topic
> > > > > > > > > >> > > > >> with 4
> > > > > > > > > >> > > > >>>>>>>>>>>> partitions),
> > > > > > > > > >> > > > >>>>>>>>>>>>> so it’s a situation that consumer
> > group
> > > > > > members
> > > > > > > > > >> already
> > > > > > > > > >> > > > >>> expect.
> > > > > > > > > >> > > > >>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>> One of Kafka’s strengths is the way
> > that
> > > > we
> > > > > > > handle
> > > > > > > > > API
> > > > > > > > > >> > > > >>>>> versioning.
> > > > > > > > > >> > > > >>>>>>>>>>>>> But, there is a cost - the behaviour
> > is
> > > > > > > different
> > > > > > > > > >> > depending
> > > > > > > > > >> > > > >>> on
> > > > > > > > > >> > > > >>>>> the
> > > > > > > > > >> > > > >>>>>>>>>> RPC
> > > > > > > > > >> > > > >>>>>>>>>>>>> version. KIP-848 is on the cusp of
> > > > > completion,
> > > > > > > but
> > > > > > > > > >> we’re
> > > > > > > > > >> > > > >>>> already
> > > > > > > > > >> > > > >>>>>>>>>> adding
> > > > > > > > > >> > > > >>>>>>>>>>>>> conditional logic for v0/v1 for
> > > > > > > > > >> ConsumerGroupHeartbeat.
> > > > > > > > > >> > > > >>> That’s
> > > > > > > > > >> > > > >>>> a
> > > > > > > > > >> > > > >>>>>>>>>> pity.
> > > > > > > > > >> > > > >>>>>>>>>>>>> Only a minor issue, but it’s
> > > unfortunate.
> > > > > > > > > >> > > > >>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>> Thanks,
> > > > > > > > > >> > > > >>>>>>>>>>>>> Andrew
> > > > > > > > > >> > > > >>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>> On 14 Aug 2024, at 08:47, TengYao
> > Chi <
> > > > > > > > > >> > > > >> kitingiao@gmail.com>
> > > > > > > > > >> > > > >>>>>>>>>> wrote:
> > > > > > > > > >> > > > >>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>> Hello Andrew
> > > > > > > > > >> > > > >>>>>>>>>>>>>> Thank you for your thoughtful
> > > suggestions
> > > > > and
> > > > > > > > > getting
> > > > > > > > > >> > the
> > > > > > > > > >> > > > >>>>>>>>>> discussion
> > > > > > > > > >> > > > >>>>>>>>>>>>> going.
> > > > > > > > > >> > > > >>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>> To AS1:
> > > > > > > > > >> > > > >>>>>>>>>>>>>> In the current scenario where the
> > > server
> > > > > > > generates
> > > > > > > > > >> the
> > > > > > > > > >> > > > >> UUID,
> > > > > > > > > >> > > > >>>> if
> > > > > > > > > >> > > > >>>>>>>>>> the
> > > > > > > > > >> > > > >>>>>>>>>>>>> client
> > > > > > > > > >> > > > >>>>>>>>>>>>>> shuts down before receiving the
> > > memberId
> > > > > > > generated
> > > > > > > > > by
> > > > > > > > > >> > the
> > > > > > > > > >> > > > >> GC
> > > > > > > > > >> > > > >>>>>>>>>>>> (regardless
> > > > > > > > > >> > > > >>>>>>>>>>>>> of
> > > > > > > > > >> > > > >>>>>>>>>>>>>> whether it’s a graceful shutdown or
> > > not),
> > > > > the
> > > > > > > GC
> > > > > > > > > will
> > > > > > > > > >> > > > >> still
> > > > > > > > > >> > > > >>>> have
> > > > > > > > > >> > > > >>>>>>>>>> to
> > > > > > > > > >> > > > >>>>>>>>>>>> wait
> > > > > > > > > >> > > > >>>>>>>>>>>>>> for the heartbeat timeout because
> the
> > > > > client
> > > > > > > > > doesn’t
> > > > > > > > > >> > know
> > > > > > > > > >> > > > >>> its
> > > > > > > > > >> > > > >>>>>>>>>>> memberId.
> > > > > > > > > >> > > > >>>>>>>>>>>>>> This KIP indeed cannot completely
> > > resolve
> > > > > the
> > > > > > > > > >> > idempotency
> > > > > > > > > >> > > > >>>> issue,
> > > > > > > > > >> > > > >>>>>>>>>> but
> > > > > > > > > >> > > > >>>>>>>>>>> it
> > > > > > > > > >> > > > >>>>>>>>>>>>> can
> > > > > > > > > >> > > > >>>>>>>>>>>>>> better handle shutdown scenarios
> > under
> > > > > normal
> > > > > > > > > >> > > > >> circumstances
> > > > > > > > > >> > > > >>>>>>>>>> because
> > > > > > > > > >> > > > >>>>>>>>>>> the
> > > > > > > > > >> > > > >>>>>>>>>>>>>> client always knows its memberId.
> > Even
> > > if
> > > > > the
> > > > > > > > > client
> > > > > > > > > >> > shuts
> > > > > > > > > >> > > > >>>> down
> > > > > > > > > >> > > > >>>>>>>>>>>>> immediately
> > > > > > > > > >> > > > >>>>>>>>>>>>>> after the initial heartbeat, as
> long
> > as
> > > > it
> > > > > > > > > performs a
> > > > > > > > > >> > > > >>> graceful
> > > > > > > > > >> > > > >>>>>>>>>>> shutdown
> > > > > > > > > >> > > > >>>>>>>>>>>>> and
> > > > > > > > > >> > > > >>>>>>>>>>>>>> sends a leave heartbeat, the GC can
> > > > manage
> > > > > > the
> > > > > > > > > >> situation
> > > > > > > > > >> > > > >> and
> > > > > > > > > >> > > > >>>>>>>>>> remove
> > > > > > > > > >> > > > >>>>>>>>>>> the
> > > > > > > > > >> > > > >>>>>>>>>>>>>> member. Therefore, the goal of this
> > KIP
> > > > is
> > > > > to
> > > > > > > > > address
> > > > > > > > > >> > the
> > > > > > > > > >> > > > >>>> issue
> > > > > > > > > >> > > > >>>>>>>>>> where
> > > > > > > > > >> > > > >>>>>>>>>>>> the
> > > > > > > > > >> > > > >>>>>>>>>>>>>> GC has to wait for the heartbeat
> > > timeout
> > > > > due
> > > > > > > to the
> > > > > > > > > >> > client
> > > > > > > > > >> > > > >>>>> leaving
> > > > > > > > > >> > > > >>>>>>>>>>>>> without
> > > > > > > > > >> > > > >>>>>>>>>>>>>> knowing its memberId, which leads
> to
> > > > > reduced
> > > > > > > > > >> throughput
> > > > > > > > > >> > > > >> and
> > > > > > > > > >> > > > >>>>>>>>>> limited
> > > > > > > > > >> > > > >>>>>>>>>>>>>> scalability.
> > > > > > > > > >> > > > >>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>> The solution you suggest has also
> > been
> > > > > > > proposed by
> > > > > > > > > >> > David.
> > > > > > > > > >> > > > >>> The
> > > > > > > > > >> > > > >>>>>>>>>> concern
> > > > > > > > > >> > > > >>>>>>>>>>>>> with
> > > > > > > > > >> > > > >>>>>>>>>>>>>> this approach is that it introduces
> > > > > > additional
> > > > > > > > > >> > complexity
> > > > > > > > > >> > > > >>> for
> > > > > > > > > >> > > > >>>>>>>>>>>>>> compatibility, as the new server
> > would
> > > > not
> > > > > > > > > >> immediately
> > > > > > > > > >> > add
> > > > > > > > > >> > > > >>> the
> > > > > > > > > >> > > > >>>>>>>>>> member
> > > > > > > > > >> > > > >>>>>>>>>>>> to
> > > > > > > > > >> > > > >>>>>>>>>>>>>> the group, while the old server
> > would.
> > > > This
> > > > > > > > > requires
> > > > > > > > > >> > > > >> clients
> > > > > > > > > >> > > > >>>> to
> > > > > > > > > >> > > > >>>>>>>>>>>>>> differentiate whether their
> memberId
> > > has
> > > > > been
> > > > > > > added
> > > > > > > > > >> to
> > > > > > > > > >> > the
> > > > > > > > > >> > > > >>>> group
> > > > > > > > > >> > > > >>>>>>>>>> or
> > > > > > > > > >> > > > >>>>>>>>>>>> not,
> > > > > > > > > >> > > > >>>>>>>>>>>>>> which could result in unexpected
> > logs.
> > > > > > > > > >> > > > >>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>> Best Regards,
> > > > > > > > > >> > > > >>>>>>>>>>>>>> TengYao
> > > > > > > > > >> > > > >>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>> Andrew Schofield <
> > > > > andrew_schofield@live.com>
> > > > > > 於
> > > > > > > > > >> > 2024年8月14日
> > > > > > > > > >> > > > >>> 週三
> > > > > > > > > >> > > > >>>>>>>>>>>> 上午12:29寫道：
> > > > > > > > > >> > > > >>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>>> Hi TengYao,
> > > > > > > > > >> > > > >>>>>>>>>>>>>>> Thanks for the KIP. I wonder if
> > > there’s
> > > > a
> > > > > > > > > different
> > > > > > > > > >> way
> > > > > > > > > >> > > > >> to
> > > > > > > > > >> > > > >>>>> close
> > > > > > > > > >> > > > >>>>>>>>>>> what
> > > > > > > > > >> > > > >>>>>>>>>>>>>>> is quite a small window.
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>>> AS1: It is true that the initial
> > > > heartbeat
> > > > > > is
> > > > > > > not
> > > > > > > > > >> > > > >>> idempotent,
> > > > > > > > > >> > > > >>>>> but
> > > > > > > > > >> > > > >>>>>>>>>>> this
> > > > > > > > > >> > > > >>>>>>>>>>>>>>> remains
> > > > > > > > > >> > > > >>>>>>>>>>>>>>> true with this KIP. It’s just
> > > > differently
> > > > > > not
> > > > > > > > > >> > idempotent.
> > > > > > > > > >> > > > >>> If
> > > > > > > > > >> > > > >>>>> the
> > > > > > > > > >> > > > >>>>>>>>>>>> client
> > > > > > > > > >> > > > >>>>>>>>>>>>>>> makes its
> > > > > > > > > >> > > > >>>>>>>>>>>>>>> own member ID, sends a request and
> > > dies,
> > > > > the
> > > > > > > GC
> > > > > > > > > will
> > > > > > > > > >> > > > >> still
> > > > > > > > > >> > > > >>>> have
> > > > > > > > > >> > > > >>>>>>>>>>> added
> > > > > > > > > >> > > > >>>>>>>>>>>>>>> the member to the group and it
> will
> > > hang
> > > > > > > around
> > > > > > > > > >> until
> > > > > > > > > >> > the
> > > > > > > > > >> > > > >>>>> session
> > > > > > > > > >> > > > >>>>>>>>>>>>> expires.
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>>> I wonder if the GC could still
> > > generate
> > > > > the
> > > > > > > member
> > > > > > > > > >> ID
> > > > > > > > > >> > in
> > > > > > > > > >> > > > >>>>>>>>>> response to
> > > > > > > > > >> > > > >>>>>>>>>>>> the
> > > > > > > > > >> > > > >>>>>>>>>>>>>>> first
> > > > > > > > > >> > > > >>>>>>>>>>>>>>> heartbeat, and put the member in a
> > > > special
> > > > > > > PENDING
> > > > > > > > > >> > state
> > > > > > > > > >> > > > >>> with
> > > > > > > > > >> > > > >>>>> no
> > > > > > > > > >> > > > >>>>>>>>>>>>>>> assignments until the client sends
> > the
> > > > > next
> > > > > > > > > >> heartbeat,
> > > > > > > > > >> > > > >> thus
> > > > > > > > > >> > > > >>>>>>>>>>> confirming
> > > > > > > > > >> > > > >>>>>>>>>>>>> it
> > > > > > > > > >> > > > >>>>>>>>>>>>>>> has received the member ID. This
> > would
> > > > not
> > > > > > be
> > > > > > > a
> > > > > > > > > >> > protocol
> > > > > > > > > >> > > > >>>> change
> > > > > > > > > >> > > > >>>>>>>>>> at
> > > > > > > > > >> > > > >>>>>>>>>>>> all,
> > > > > > > > > >> > > > >>>>>>>>>>>>>>> just
> > > > > > > > > >> > > > >>>>>>>>>>>>>>> a change to the GC to keep a new
> > > member
> > > > in
> > > > > > the
> > > > > > > > > lobby
> > > > > > > > > >> > > > >> until
> > > > > > > > > >> > > > >>>> it’s
> > > > > > > > > >> > > > >>>>>>>>>>>>> comfirmed
> > > > > > > > > >> > > > >>>>>>>>>>>>>>> it knows its member ID.
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>>> Thanks,
> > > > > > > > > >> > > > >>>>>>>>>>>>>>> Andrew
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>> On 13 Aug 2024, at 15:59, TengYao
> > > Chi <
> > > > > > > > > >> > > > >>> kitingiao@gmail.com>
> > > > > > > > > >> > > > >>>>>>>>>> wrote:
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>> Hi Chia-Ping,
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>> Thanks for review and
> suggestions.
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>> I have updated the content of KIP
> > > > > > > accordingly.
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>> Please take a look.
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>> Best regards,
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>> TengYao
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>> Chia-Ping Tsai <
> > chia7712@apache.org>
> > > 於
> > > > > > > > > 2024年8月13日
> > > > > > > > > >> 週二
> > > > > > > > > >> > > > >>>>> 下午9:45寫道：
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> hi TengYao
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> thanks for this KIP.
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> 1) could you please describe the
> > > > > > > before/after
> > > > > > > > > >> > behavior
> > > > > > > > > >> > > > >> in
> > > > > > > > > >> > > > >>>> the
> > > > > > > > > >> > > > >>>>>>>>>>>>> "Proposed
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> Changes" section? IIRC, current
> > RPC
> > > > > allows
> > > > > > > HB
> > > > > > > > > >> having
> > > > > > > > > >> > > > >>> member
> > > > > > > > > >> > > > >>>>> id
> > > > > > > > > >> > > > >>>>>>>>>>>>>>> generated by
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> client, right? If HB has no
> member
> > > ID,
> > > > > > > server
> > > > > > > > > will
> > > > > > > > > >> > > > >>> generate
> > > > > > > > > >> > > > >>>>> one
> > > > > > > > > >> > > > >>>>>>>>>>> and
> > > > > > > > > >> > > > >>>>>>>>>>>>> then
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> return. The new behavior will
> > > enforce
> > > > HB
> > > > > > > "must"
> > > > > > > > > >> have
> > > > > > > > > >> > > > >>> member
> > > > > > > > > >> > > > >>>>> ID.
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> 2) could you please write the
> > > version
> > > > > > number
> > > > > > > > > >> > explicitly
> > > > > > > > > >> > > > >>> in
> > > > > > > > > >> > > > >>>>> the
> > > > > > > > > >> > > > >>>>>>>>>> KIP
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> 3) how new client code handle
> the
> > > old
> > > > > HB?
> > > > > > > Does
> > > > > > > > > it
> > > > > > > > > >> > > > >> always
> > > > > > > > > >> > > > >>>>>>>>>> generate
> > > > > > > > > >> > > > >>>>>>>>>>>>> member
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> ID on client-side even though
> that
> > > is
> > > > > not
> > > > > > > > > >> restricted?
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> Best,
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> Chia-Ping
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>> On 2024/08/13 06:20:42 TengYao
> Chi
> > > > > wrote:
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> Hello everyone,
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> I would like to start a
> > discussion
> > > > > thread
> > > > > > > on
> > > > > > > > > >> > KIP-1082,
> > > > > > > > > >> > > > >>>> which
> > > > > > > > > >> > > > >>>>>>>>>>>> proposes
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> enabling id generation for
> > clients
> > > > over
> > > > > > the
> > > > > > > > > >> > > > >>>>>>>>>>> ConsumerGroupHeartbeat
> > > > > > > > > >> > > > >>>>>>>>>>>>> RPC.
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> Here is the KIP Link: KIP-1082
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> <
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>
> > > > > > > > > >> > > > >>>>>
> > > > > > > > > >> > > > >>>>
> > > > > > > > > >> > > > >>>
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> Please take a look and let me
> > know
> > > > what
> > > > > > you
> > > > > > > > > >> think,
> > > > > > > > > >> > > > >> and I
> > > > > > > > > >> > > > >>>>> would
> > > > > > > > > >> > > > >>>>>>>>>>>>>>> appreciate
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> any suggestions and feedback.
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> Best regards,
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> TengYao
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>>>
> > > > > > > > > >> > > > >>>>>>>
> > > > > > > > > >> > > > >>>>>>>
> > > > > > > > > >> > > > >>>>>
> > > > > > > > > >> > > > >>>>
> > > > > > > > > >> > > > >>>
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[Chia-Ping Tsai <ch...@gmail.com>]

Hi David

> I tend to believe that generating a uuid on the client should not be that
difficult so I am not too concerned about it. How do you see this,

You're right. It's not that difficult, and we should have more confidence
in third-party libraries.

hi TengYao

Could you please summarize the discussion on "duplicate member ID" and add
it to KIP-1082? Also, please remove "The member is also removed from the
group." from KIP-848

Best,
Chia-Ping

David Jacot <dj...@confluent.io.invalid> 於 2024年9月23日 週一 下午10:43寫道：

> Hi Chia-Ping,
>
> > However, my concern is that consumers are now required to generate the
> member ID, so as a responsible server, we need to handle the incorrect
> client behavior. At least, based on David's comment, we can create another
> member id when it is already used and it is rejoining the group (member
> epoch = 0).
>
> I don't disagree with the concern. However, I don't see how we could make
> it work in practice. Let's say that a member XYZ sends a heartbeat with
> epoch zero. The server cannot know whether it is a new member joining for
> the first time or an existing member rejoining because it was fenced (e.g.
> wrong epoch, session timed out, etc.). We cannot differentiate the two
> cases. Hence we cannot handle it on the server side. The only viable
> alternative is to keep the member id generation on the server side if we
> are concerned by clients not able to generate unique uuids on their side. I
> tend to believe that generating a uuid on the client should not be that
> difficult so I am not too concerned about it. How do you see
> this, Chia-Ping?
>
> Cheers,
> David
>
> On Mon, Sep 23, 2024 at 4:13 PM Chia-Ping Tsai <ch...@gmail.com> wrote:
>
> > hi David
> >
> > > For instance, we could check whether the member id is already used and
> we
> > could regenerate another one if it is. However, I tend to believe that
> the
> > risk of collision within a group is neglectable.
> >
> > I agree the risk is negligible when the member ID is generated by the
> > server, as we can always ensure the server behaves responsibly. 😀
> > Therefore, I don't believe the additional check for the member ID is
> > necessary if the member id is created by server.
> >
> > However, my concern is that consumers are now required to generate the
> > member ID, so as a responsible server, we need to handle the incorrect
> > client behavior. At least, based on David's comment, we can create
> another
> > member id when it is already used and it is rejoining the group (member
> > epoch = 0).
> >
> > By the way, another related issue is that a duplicate consumer may
> > inadvertently terminate another consumer, as it will always send a
> > LEAVE_GROUP request upon closing, even if it hasn't received a heartbeat
> > response. However, I tend to believe the risk is negligible 😀
> >
> > Best,
> > Chia-Ping
> >
> >
> > David Jacot <dj...@confluent.io.invalid> 於 2024年9月23日 週一 下午9:07寫道：
> >
> > > Hi Chia-Ping,
> > >
> > > > Checks whether the member epoch matches the member epoch in its
> current
> > > assignment. FENCED_MEMBER_EPOCH is returned otherwise. The member is
> also
> > > removed from the group.
> > >
> > > We did not implement it like this in the end. We keep the member and
> only
> > > request him to re-synchronise himself by returning the
> > FENCED_MEMBER_EPOCH
> > > error. We can remove it from the KIP.
> > >
> > > > Given that member id is the incarnation of consumer, it is hard to
> > detect
> > > the scenario of "two consumers have the same member id" (it may be
> caused
> > > by the lost response). I guess the best approach is the later -created
> > > consumer should be idle with fenced error, and we should add this to
> the
> > > KIP (maybe we should revise the error message carried by
> > > `FencedMemberEpochException`). BUT, we need to rethink the approach if
> > > above description "The member is also removed from the group" is true
> > >
> > > I don't think that we have a reliable way to detect whether two members
> > end
> > > up using the same member id. I don't think that it is possible if we
> > > generate the member id on the client side. By the way, we don't handle
> > this
> > > case either on the server side today. We blindly generate a member id
> and
> > > use it. However, we could do better on the server side. For instance,
> we
> > > could check whether the member id is already used and we could
> regenerate
> > > another one if it is. However, I tend to believe that the risk of
> > collision
> > > within a group is neglectable.
> > >
> > > Best,
> > > David
> > >
> > > On Fri, Sep 20, 2024 at 8:11 PM Chia-Ping Tsai <ch...@gmail.com>
> > wrote:
> > >
> > > > wait a minute. There is a description about fenced error in the
> KIP-848
> > > >
> > > > > Checks whether the member epoch matches the member epoch in its
> > current
> > > > assignment. FENCED_MEMBER_EPOCH is returned otherwise. The member is
> > also
> > > > removed from the group.
> > > >
> > > > I failed to find the code of implementing "The member is also removed
> > > from
> > > > the group.". I will appreciate it if someone can share the code link
> to
> > > me.
> > > >
> > > > Let's assume it will be implemented eventually, and I feel that will
> > > cause
> > > > a serious issue as the consumers having the same member id will
> remove
> > > the
> > > > existing  member and then create itself member?
> > > >
> > > > > Once the client receives this exception, it should consider the
> > > exception
> > > > as a serious error and stop the process, and the user will need to
> > > assign a
> > > > new memberId.
> > > >
> > > > Given that member id is the incarnation of consumer, it is hard to
> > detect
> > > > the scenario of "two consumers have the same member id" (it may be
> > caused
> > > > by the lost response). I guess the best approach is the later
> -created
> > > > consumer should be idle with fenced error, and we should add this to
> > the
> > > > KIP (maybe we should revise the error message carried by
> > > > `FencedMemberEpochException`). BUT, we need to rethink the approach
> if
> > > > above description "The member is also removed from the group" is true
> > > >
> > > > Best,
> > > > Chia-Ping
> > > >
> > > >
> > > >
> > > > TengYao Chi <ki...@gmail.com> 於 2024年9月20日 週五 下午8:17寫道：
> > > >
> > > > > Hi Chia-Ping,
> > > > >
> > > > > Thanks for pointing out this issue.
> > > > >
> > > > > I’m thinking that maybe we might need to define a new Exception to
> > > handle
> > > > > this scenario.
> > > > > Once the client receives this exception, it should consider the
> > > exception
> > > > > as a serious error and stop the process, and the user will need to
> > > > assign a
> > > > > new memberId.
> > > > > What do you think?
> > > > >
> > > > > Best Regards,
> > > > > TengYao
> > > > >
> > > > > Chia-Ping Tsai <ch...@apache.org> 於 2024年9月20日 週五 下午7:00寫道：
> > > > >
> > > > > > > This part is not clear either. It basically says that if a
> member
> > > > joins
> > > > > > with an existing member id but a different epoch, it will be
> > fenced.
> > > > Then
> > > > > > it must rejoin with the same member id and epoch zero. This is
> > > already
> > > > > the
> > > > > > current behavior and it does not help with detecting duplicates,
> > > right?
> > > > > >
> > > > > > The duplicates issue is interesting. The member id is generated
> on
> > > > > > server-side by UUID before, so it seems to me the fenced member
> > epoch
> > > > > > happens only if the client miss the response with bumped epoch,
> and
> > > > this
> > > > > > case is recoverable.
> > > > > >
> > > > > > however, v1 brings a different fenced epoch scenario. Users, now,
> > > have
> > > > > > responsibility to generate unique member id, hence clients may
> > > > encounter
> > > > > > infinite fenced error if there are >1 clients having same member
> id
> > > due
> > > > > to
> > > > > > config error (or other engineer error)
> > > > > >
> > > > > > Maybe we can highlight this scenario if we start to requires
> > clients
> > > to
> > > > > > generate unique member id.
> > > > > >
> > > > > > Best,
> > > > > > Chia-Ping
> > > > > >
> > > > > > On 2024/09/19 18:36:48 David Jacot wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > Thanks for the update. I have a few nits:
> > > > > > >
> > > > > > > > If the member ID is null or empty, the server will reject the
> > > > request
> > > > > > > with an InvalidRequestException.
> > > > > > > We should clarify that this should only apply to version >= 1.
> > > > > > >
> > > > > > > > The consumer instance must generate a member ID, and this ID
> > > should
> > > > > > > remain consistent for the duration of the consumer's session.
> > > Here, a
> > > > > > > "session" is defined as the period from the consumer's first
> > > > heartbeat
> > > > > > > until it leaves the group, either through a graceful shutdown,
> a
> > > > > > heartbeat
> > > > > > > timeout, or the process stopping or dying. The consumer
> instance
> > > > should
> > > > > > > reuse the same member ID for all heartbeats and rejoin attempts
> > to
> > > > > > maintain
> > > > > > > continuity within the group.
> > > > > > >
> > > > > > > This part is not clear to me. When the member leaves the group,
> > it
> > > > > should
> > > > > > > not reset the member id. I would rather say that the member
> must
> > > > > generate
> > > > > > > its member id when it starts and it must keep it until the
> > process
> > > > > stops.
> > > > > > > It is basically an incarnation of the process.
> > > > > > >
> > > > > > > > If a conflict arises where the member ID generated by the
> > client
> > > is
> > > > > > > detected to be a duplicate within the same group (for example,
> > the
> > > > same
> > > > > > > member ID is associated with another active member in the
> group),
> > > the
> > > > > > > server will handle this by comparing the memberEpoch values of
> > the
> > > > > > > conflicting members. The member with the lower memberEpoch is
> > > > > considered
> > > > > > > outdated and will be fenced off by the server. When this
> occurs,
> > > the
> > > > > > server
> > > > > > > responds with a FENCED_MEMBER_EPOCH error to the client,
> > signaling
> > > it
> > > > > to
> > > > > > > rejoin the group with the same member ID while resetting the
> > > > > memberEpoch
> > > > > > to
> > > > > > > zero. This ensures that the client properly resynchronizes and
> > > > > maintains
> > > > > > > the continuity and consistency of the group membership.
> > > > > > >
> > > > > > > This part is not clear either. It basically says that if a
> member
> > > > joins
> > > > > > > with an existing member id but a different epoch, it will be
> > > fenced.
> > > > > Then
> > > > > > > it must rejoin with the same member id and epoch zero. This is
> > > > already
> > > > > > the
> > > > > > > current behavior and it does not help with detecting
> duplicates,
> > > > right?
> > > > > > > Should we just remove the paragraph?
> > > > > > >
> > > > > > > > A member ID mismatch occurs within a session: If the server
> > > > detects a
> > > > > > > mismatch between the provided member ID and the expected member
> > ID
> > > > for
> > > > > an
> > > > > > > ongoing session, it should return a UNKNOWN_MEMBER_ID  error.
> > > > > > >
> > > > > > > How could we detect a mismatch between the provided and the
> > > expected
> > > > > > member
> > > > > > > id? My understanding is that we can only know whether the
> > provided
> > > > > member
> > > > > > > id exists or not. This is already implemented.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > David
> > > > > > >
> > > > > > > On Sat, Sep 14, 2024 at 9:31 AM TengYao Chi <
> kitingiao@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Hello everyone,
> > > > > > > >
> > > > > > > > Since this KIP has been fully discussed, I will initiate a
> vote
> > > for
> > > > > it
> > > > > > next
> > > > > > > > Monday.
> > > > > > > > Thank you and have a nice weekend.
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > TengYao
> > > > > > > >
> > > > > > > > TengYao Chi <ki...@gmail.com> 於 2024年9月5日 週四 下午2:19寫道：
> > > > > > > >
> > > > > > > > > Hello everyone,
> > > > > > > > >
> > > > > > > > > KT2: It looks like everyone who has expressed an opinion
> > > supports
> > > > > the
> > > > > > > > > second option: “Document a recommendation for clients to
> use
> > > > UUIDs
> > > > > as
> > > > > > > > > member IDs, without strictly enforcing it.”
> > > > > > > > > I have updated the KIP accordingly.
> > > > > > > > > Please take a look, and let me know if you have any
> thoughts
> > or
> > > > > > feedback.
> > > > > > > > >
> > > > > > > > > Thank you!
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > TengYao
> > > > > > > > >
> > > > > > > > > Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月30日 週五
> > 下午9:56寫道：
> > > > > > > > >
> > > > > > > > >> hi TengYao
> > > > > > > > >>
> > > > > > > > >> KT2: +1 to second approach
> > > > > > > > >>
> > > > > > > > >> Best,
> > > > > > > > >> Chia-Ping
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> David Jacot <dj...@confluent.io.invalid> 於 2024年8月30日 週五
> > > > > 下午9:15寫道：
> > > > > > > > >>
> > > > > > > > >> > Hi TengYao,
> > > > > > > > >> >
> > > > > > > > >> > KT2: I don't think that we can realistically validate
> the
> > > uuid
> > > > > on
> > > > > > the
> > > > > > > > >> > server. It is basically a string of chars. So I lean
> > towards
> > > > > > having a
> > > > > > > > >> good
> > > > > > > > >> > recommendation in the KIP and in the document of the
> field
> > > in
> > > > > the
> > > > > > > > RPC's
> > > > > > > > >> > definition.
> > > > > > > > >> >
> > > > > > > > >> > Best,
> > > > > > > > >> > David
> > > > > > > > >> >
> > > > > > > > >> > On Fri, Aug 30, 2024 at 3:02 PM TengYao Chi <
> > > > > kitingiao@gmail.com>
> > > > > > > > >> wrote:
> > > > > > > > >> >
> > > > > > > > >> > > Hello Kirk !
> > > > > > > > >> > >
> > > > > > > > >> > > Thank you for your comments !
> > > > > > > > >> > >
> > > > > > > > >> > > KT1: Yes, you are correct. The issue is not unique to
> > the
> > > > > > initial
> > > > > > > > >> > > heartbeat; there can always be cases where the broker
> > > might
> > > > > lose
> > > > > > > > >> > connection
> > > > > > > > >> > > with a member.
> > > > > > > > >> > >
> > > > > > > > >> > > KT2: Currently, if the client doesn't have a member ID
> > and
> > > > the
> > > > > > > > >> > memberEpoch
> > > > > > > > >> > > equals 0, the coordinator will generate a UUID as the
> > > member
> > > > > ID
> > > > > > for
> > > > > > > > >> the
> > > > > > > > >> > > client. However, at the RPC level, the member ID is
> sent
> > > as
> > > > a
> > > > > > > > literal
> > > > > > > > >> > > string, meaning there are no restrictions on the
> format
> > at
> > > > > this
> > > > > > > > level.
> > > > > > > > >> > > This also reminds me that we haven't reached a final
> > > > > conclusion
> > > > > > on
> > > > > > > > >> how to
> > > > > > > > >> > > enforce the use of UUIDs.
> > > > > > > > >> > > From our previous discussions, I recall two possible
> > > > > approaches:
> > > > > > > > >> > > The first is to validate the UUID on the server side,
> > and
> > > if
> > > > > > it's
> > > > > > > > not
> > > > > > > > >> > > valid, throw an exception to the client.
> > > > > > > > >> > > The second is to document a recommendation for clients
> > to
> > > > use
> > > > > > UUIDs
> > > > > > > > as
> > > > > > > > >> > > member IDs, without strictly enforcing it.
> > > > > > > > >> > > I think it's time to decide on the approach we want to
> > > take.
> > > > > > > > >> > >
> > > > > > > > >> > > KT3: Yes, "session" can be considered synonymous with
> > > > > > "membership"
> > > > > > > > in
> > > > > > > > >> > this
> > > > > > > > >> > > context.
> > > > > > > > >> > >
> > > > > > > > >> > > KT4: Thank you for pointing that out. I will update
> the
> > > > > wording
> > > > > > to
> > > > > > > > >> > > specifically say this behavior is for consumers.
> > > > > > > > >> > >
> > > > > > > > >> > > Thanks again for your comments.
> > > > > > > > >> > >
> > > > > > > > >> > > Best regards,
> > > > > > > > >> > > TengYao
> > > > > > > > >> > >
> > > > > > > > >> > > Kirk True <ki...@kirktrue.pro> 於 2024年8月30日 週五
> > 上午12:39寫道：
> > > > > > > > >> > >
> > > > > > > > >> > > > Hi TengYao!
> > > > > > > > >> > > >
> > > > > > > > >> > > > Sorry for being late to the discussion...
> > > > > > > > >> > > >
> > > > > > > > >> > > > After reading the thread and then the KIP, I had a
> few
> > > > > > > > >> > > questions/comments:
> > > > > > > > >> > > >
> > > > > > > > >> > > > KT1: In Motivation, it states: "This scenario can
> > result
> > > > in
> > > > > > the
> > > > > > > > >> broker
> > > > > > > > >> > > > registering a new member for which it will never
> > > receive a
> > > > > > proper
> > > > > > > > >> leave
> > > > > > > > >> > > > request.” Just to be clear, the broker will always
> > have
> > > > > cases
> > > > > > > > where
> > > > > > > > >> it
> > > > > > > > >> > > > might lose connection with a member. That’s not
> unique
> > > to
> > > > > the
> > > > > > > > >> initial
> > > > > > > > >> > > > heartbeat, right?
> > > > > > > > >> > > >
> > > > > > > > >> > > > KT2: There was a bit of back and forth about format
> of
> > > the
> > > > > > member
> > > > > > > > >> ID.
> > > > > > > > >> > > From
> > > > > > > > >> > > > what I gathered in the thread, the member ID is
> still
> > > > > defined
> > > > > > in
> > > > > > > > the
> > > > > > > > >> > RPC
> > > > > > > > >> > > as
> > > > > > > > >> > > > a string and not a UUID, right? The KIP states that
> > the
> > > > > > “client
> > > > > > > > must
> > > > > > > > >> > > > generate a UUID as the member ID” and that the
> “server
> > > > will
> > > > > > > > validate
> > > > > > > > >> > > that a
> > > > > > > > >> > > > valid UUID is provided.” Is that a change for the
> > > server,
> > > > or
> > > > > > is it
> > > > > > > > >> > > already
> > > > > > > > >> > > > enforced as a UUID?
> > > > > > > > >> > > >
> > > > > > > > >> > > > KT3: Lianet mentioned some confusion over the use of
> > the
> > > > > word
> > > > > > > > >> > “session.”
> > > > > > > > >> > > > Isn’t “session” synonymous with “membership?”
> > > > > > > > >> > > >
> > > > > > > > >> > > > KT4: Under “Member ID Lifecycle,” it states: "The
> > client
> > > > > > should
> > > > > > > > >> reuse
> > > > > > > > >> > the
> > > > > > > > >> > > > same UUID as the member ID for all heartbeats and
> > rejoin
> > > > > > attempts
> > > > > > > > to
> > > > > > > > >> > > > maintain continuity within the group.” Could we
> change
> > > the
> > > > > > first
> > > > > > > > >> part
> > > > > > > > >> > of
> > > > > > > > >> > > > that to “The Consumer instance should…” We do have
> > > > lifetimes
> > > > > > that
> > > > > > > > >> > extend
> > > > > > > > >> > > > past the lifetime of a client instance (such as the
> > > > > > transaction
> > > > > > > > ID).
> > > > > > > > >> > > >
> > > > > > > > >> > > > Thanks,
> > > > > > > > >> > > > Kirk
> > > > > > > > >> > > >
> > > > > > > > >> > > > > On Aug 29, 2024, at 1:28 AM, TengYao Chi <
> > > > > > kitingiao@gmail.com>
> > > > > > > > >> > wrote:
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Hi David,
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Thank you for pointing that out.
> > > > > > > > >> > > > > I have updated the content of the KIP based on
> > > Lianet's
> > > > > and
> > > > > > your
> > > > > > > > >> > > > feedback.
> > > > > > > > >> > > > > Please take a look and let me know your thoughts.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Best regards,
> > > > > > > > >> > > > > TengYao
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > David Jacot <dj...@confluent.io.invalid> 於
> > > 2024年8月29日
> > > > 週四
> > > > > > > > >> 下午3:20寫道：
> > > > > > > > >> > > > >
> > > > > > > > >> > > > >> Hi TengYao,
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >> Thanks for the update. I haven't fully read it
> yet
> > > but
> > > > I
> > > > > > will
> > > > > > > > >> soon.
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >> LM4: This is incorrect. The consumer must keep
> its
> > > > member
> > > > > > id
> > > > > > > > >> during
> > > > > > > > >> > > its
> > > > > > > > >> > > > >> entire lifetime (until the process stops or
> dies).
> > > The
> > > > > > protocol
> > > > > > > > >> > > > stipulates
> > > > > > > > >> > > > >> that a member must rejoin with the same member id
> > and
> > > > the
> > > > > > > > member
> > > > > > > > >> > epoch
> > > > > > > > >> > > > set
> > > > > > > > >> > > > >> to zero when an FENCED_MEMBER_EPOCH occurs. This
> > > allows
> > > > > the
> > > > > > > > >> member
> > > > > > > > >> > to
> > > > > > > > >> > > > >> resynchronize itself. We should not change this
> > > > > behavior. I
> > > > > > > > think
> > > > > > > > >> > that
> > > > > > > > >> > > > we
> > > > > > > > >> > > > >> should see the client side generation id as an
> > > > > incarnation
> > > > > > id
> > > > > > > > of
> > > > > > > > >> the
> > > > > > > > >> > > > >> application. It is generated once and kept until
> it
> > > > stops
> > > > > > or
> > > > > > > > >> dies.
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >> Best,
> > > > > > > > >> > > > >> David
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >> On Thu, Aug 29, 2024 at 6:21 AM TengYao Chi <
> > > > > > > > kitingiao@gmail.com
> > > > > > > > >> >
> > > > > > > > >> > > > wrote:
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >>> Hello Lianet !
> > > > > > > > >> > > > >>>
> > > > > > > > >> > > > >>> Thanks for the reviews and suggestions!
> > > > > > > > >> > > > >>>
> > > > > > > > >> > > > >>> LM1: Indeed, we plan to enforce client-side ID
> > > > > generation
> > > > > > in
> > > > > > > > the
> > > > > > > > >> > > > future,
> > > > > > > > >> > > > >>> and it is not an alternative. I will change the
> > > title
> > > > > > > > >> accordingly.
> > > > > > > > >> > > > >>>
> > > > > > > > >> > > > >>> LM2: Yes, that's the expectation. I will add
> that
> > > > > > statement to
> > > > > > > > >> the
> > > > > > > > >> > > > public
> > > > > > > > >> > > > >>> interface section.
> > > > > > > > >> > > > >>>
> > > > > > > > >> > > > >>> LM3: Thank you for the high-level perspective
> > > review.
> > > > I
> > > > > > think
> > > > > > > > >> > you're
> > > > > > > > >> > > > >> right;
> > > > > > > > >> > > > >>> our intention isn't very clear since it was
> placed
> > > at
> > > > > the
> > > > > > end
> > > > > > > > of
> > > > > > > > >> > the
> > > > > > > > >> > > > >>> section. I will try to rephrase that section to
> > make
> > > > it
> > > > > > more
> > > > > > > > >> > obvious.
> > > > > > > > >> > > > >>>
> > > > > > > > >> > > > >>> LM4: Regarding the definition of "session" in
> this
> > > > KIP,
> > > > > I
> > > > > > > > >> believe
> > > > > > > > >> > it
> > > > > > > > >> > > > >> refers
> > > > > > > > >> > > > >>> to the period between the *first-time heartbeat*
> > and
> > > > > when
> > > > > > the
> > > > > > > > >> > > *consumer
> > > > > > > > >> > > > >>> leaves the group* (whether through a graceful
> > > shutdown
> > > > > or
> > > > > > a
> > > > > > > > >> > heartbeat
> > > > > > > > >> > > > >>> timeout). The consumer should reuse its UUID if
> it
> > > has
> > > > > > been
> > > > > > > > >> > generated
> > > > > > > > >> > > > >>> before. The only situation in which it will
> > > regenerate
> > > > > the
> > > > > > > > UUID
> > > > > > > > >> is
> > > > > > > > >> > if
> > > > > > > > >> > > > the
> > > > > > > > >> > > > >>> coordinator finds that there is already a
> consumer
> > > > with
> > > > > > the
> > > > > > > > same
> > > > > > > > >> > > UUID.
> > > > > > > > >> > > > >>> IIRC, the coordinator should compare the member
> > > > epochs,
> > > > > > and
> > > > > > > > the
> > > > > > > > >> > > > >>> later-joined consumer should be fenced off by
> the
> > > > > > coordinator
> > > > > > > > >> due
> > > > > > > > >> > to
> > > > > > > > >> > > > >> having
> > > > > > > > >> > > > >>> a lower member epoch. Once the consumer
> receives a
> > > > > > > > >> > > > `FENCED_MEMBER_EPOCH`
> > > > > > > > >> > > > >>> error, it will generate a new UUID and attempt
> to
> > > > > rejoin.
> > > > > > I
> > > > > > > > will
> > > > > > > > >> > > > clarify
> > > > > > > > >> > > > >>> this in the KIP.
> > > > > > > > >> > > > >>>
> > > > > > > > >> > > > >>> Thanks again for your reviews, I really
> appreciate
> > > it.
> > > > > > > > >> > > > >>>
> > > > > > > > >> > > > >>> Best regards,
> > > > > > > > >> > > > >>> TengYao
> > > > > > > > >> > > > >>>
> > > > > > > > >> > > > >>> Lianet M. <li...@gmail.com> 於 2024年8月28日 週三
> > > > > 下午7:12寫道：
> > > > > > > > >> > > > >>>
> > > > > > > > >> > > > >>>> Hello TengYao! Thanks for taking on this issue,
> > > we've
> > > > > > been
> > > > > > > > >> going
> > > > > > > > >> > > > around
> > > > > > > > >> > > > >>> it
> > > > > > > > >> > > > >>>> for a while.
> > > > > > > > >> > > > >>>>
> > > > > > > > >> > > > >>>> LM1: About the title of the KIP: "Enable ID
> > > > Generation
> > > > > > for
> > > > > > > > >> Clients
> > > > > > > > >> > > > over
> > > > > > > > >> > > > >>> the
> > > > > > > > >> > > > >>>> ConsumerGroupHeartbeat RPC". I find it
> confusing
> > > > > because
> > > > > > it
> > > > > > > > >> hints
> > > > > > > > >> > > that
> > > > > > > > >> > > > >>>> we're adding it as an alternative (which was
> > > > discussed
> > > > > > and
> > > > > > > > >> > > discarded,
> > > > > > > > >> > > > >> in
> > > > > > > > >> > > > >>>> favour of really enforcing it). It's also
> missing
> > > the
> > > > > > core
> > > > > > > > >> change
> > > > > > > > >> > > imo,
> > > > > > > > >> > > > >>>> which is "where" the generation happens. So,
> > maybe
> > > > more
> > > > > > to
> > > > > > > > the
> > > > > > > > >> > point
> > > > > > > > >> > > > >> with
> > > > > > > > >> > > > >>>> something along the lines of "Client-side
> > generated
> > > > ID
> > > > > > for
> > > > > > > > >> clients
> > > > > > > > >> > > > over
> > > > > > > > >> > > > >>>> ConsumerGroupHeartbeat RPC"?
> > > > > > > > >> > > > >>>>
> > > > > > > > >> > > > >>>> LM2: On the public interfaces section, the KIP
> > > states
> > > > > > that
> > > > > > > > "the
> > > > > > > > >> > > server
> > > > > > > > >> > > > >>> will
> > > > > > > > >> > > > >>>> reject the request", but we should agree on the
> > > > > specific
> > > > > > > > error
> > > > > > > > >> > > type. I
> > > > > > > > >> > > > >>>> expect it should fail with an
> > > > InvalidRequestException,
> > > > > is
> > > > > > > > that
> > > > > > > > >> the
> > > > > > > > >> > > > >>>> intention? (This was also suggested in the
> > > discussion
> > > > > > thread
> > > > > > > > >> > before
> > > > > > > > >> > > > but
> > > > > > > > >> > > > >>> is
> > > > > > > > >> > > > >>>> not in the KIP).
> > > > > > > > >> > > > >>>>
> > > > > > > > >> > > > >>>> LM3. Related to my previous point, I find that
> to
> > > be
> > > > > the
> > > > > > true
> > > > > > > > >> > > > >>> public-facing
> > > > > > > > >> > > > >>>> change (member ID mandatory at the protocol
> > level),
> > > > but
> > > > > > it's
> > > > > > > > >> only
> > > > > > > > >> > at
> > > > > > > > >> > > > >> the
> > > > > > > > >> > > > >>>> end of the Public interfaces changes, kind of
> > lost
> > > > > among
> > > > > > > > >> details
> > > > > > > > >> > of
> > > > > > > > >> > > > how
> > > > > > > > >> > > > >>>> we're going to do it. Should we rephrase that
> > > section
> > > > > > with
> > > > > > > > the
> > > > > > > > >> > > actual
> > > > > > > > >> > > > >>>> change first, and the hows after (ex. Bumping
> the
> > > > > > version is
> > > > > > > > >> not
> > > > > > > > >> > the
> > > > > > > > >> > > > >>>> public-facing change in this case, it's just
> the
> > > > > > mechanism to
> > > > > > > > >> > > properly
> > > > > > > > >> > > > >>>> introduce our change)
> > > > > > > > >> > > > >>>>
> > > > > > > > >> > > > >>>> LM4. Regarding the lifetime of the UUID: the
> KIP
> > > > states
> > > > > > we
> > > > > > > > will
> > > > > > > > >> > > > "Verify
> > > > > > > > >> > > > >>>> that the UUID remains consistent across all
> > > > subsequent
> > > > > > > > >> heartbeats
> > > > > > > > >> > > > >> during
> > > > > > > > >> > > > >>>> the session". What is this "session" referring
> to
> > > > > here? I
> > > > > > > > would
> > > > > > > > >> > > expect
> > > > > > > > >> > > > >>> that
> > > > > > > > >> > > > >>>> the UUID is associated to a consumer instance
> > > > > (generated
> > > > > > for
> > > > > > > > >> the
> > > > > > > > >> > > > >> consumer
> > > > > > > > >> > > > >>>> the first time it needs to send a HB if it
> > doesn't
> > > > have
> > > > > > the
> > > > > > > > >> UUID
> > > > > > > > >> > > yet.
> > > > > > > > >> > > > >>> From
> > > > > > > > >> > > > >>>> there on, every time it needs to send a "first
> > HB"
> > > > > > again, it
> > > > > > > > >> will
> > > > > > > > >> > > > reuse
> > > > > > > > >> > > > >>> its
> > > > > > > > >> > > > >>>> UUID, is that the intention? Note that we
> should
> > > > > consider
> > > > > > > > that
> > > > > > > > >> the
> > > > > > > > >> > > > same
> > > > > > > > >> > > > >>>> consumer instance may have many "first
> > heartbeats",
> > > > > > meaning
> > > > > > > > >> > > heartbeats
> > > > > > > > >> > > > >> to
> > > > > > > > >> > > > >>>> join the group when it's not part of it (ex.
> > > consumer
> > > > > > > > >> unsubscribe
> > > > > > > > >> > +
> > > > > > > > >> > > > >>>> subscribe, fenced, stale). Is this the
> intention
> > or
> > > > are
> > > > > > you
> > > > > > > > >> > > > considering
> > > > > > > > >> > > > >>> the
> > > > > > > > >> > > > >>>> lifetime differently? We should clarify it in
> the
> > > > KIP.
> > > > > > > > >> > > > >>>>
> > > > > > > > >> > > > >>>> Thanks!
> > > > > > > > >> > > > >>>>
> > > > > > > > >> > > > >>>> Lianet
> > > > > > > > >> > > > >>>>
> > > > > > > > >> > > > >>>> On Tue, Aug 27, 2024 at 2:27 AM TengYao Chi <
> > > > > > > > >> kitingiao@gmail.com>
> > > > > > > > >> > > > >> wrote:
> > > > > > > > >> > > > >>>>
> > > > > > > > >> > > > >>>>> Hi everyone,
> > > > > > > > >> > > > >>>>>
> > > > > > > > >> > > > >>>>> I have revised this KIP multiple times based
> on
> > > the
> > > > > > feedback
> > > > > > > > >> from
> > > > > > > > >> > > our
> > > > > > > > >> > > > >>>>> discussions.
> > > > > > > > >> > > > >>>>> I would greatly appreciate it if you could
> > review
> > > it
> > > > > > when
> > > > > > > > you
> > > > > > > > >> > have
> > > > > > > > >> > > > >> the
> > > > > > > > >> > > > >>>>> time.
> > > > > > > > >> > > > >>>>> If there are no further comments or
> > suggestions, I
> > > > > plan
> > > > > > to
> > > > > > > > >> > proceed
> > > > > > > > >> > > > >> with
> > > > > > > > >> > > > >>>>> initiating a vote soon.
> > > > > > > > >> > > > >>>>>
> > > > > > > > >> > > > >>>>> Best regards,
> > > > > > > > >> > > > >>>>> TengYao
> > > > > > > > >> > > > >>>>>
> > > > > > > > >> > > > >>>>> TengYao Chi <ki...@gmail.com> 於
> 2024年8月23日
> > 週五
> > > > > > 下午2:43寫道：
> > > > > > > > >> > > > >>>>>
> > > > > > > > >> > > > >>>>>> Hi Andrew,
> > > > > > > > >> > > > >>>>>> Thank you for your previous feedback and
> > > insights.
> > > > > > > > >> > > > >>>>>> Your contributions have added valuable
> > > perspectives
> > > > > to
> > > > > > the
> > > > > > > > >> > > > >>> discussions.
> > > > > > > > >> > > > >>>>>> And we also benefit from the comparison of
> > > > different
> > > > > > > > >> solutions.
> > > > > > > > >> > > > >>>>>> I’m also looking forward to seeing an initial
> > > > version
> > > > > > in
> > > > > > > > >> > KIP-932,
> > > > > > > > >> > > > >> as
> > > > > > > > >> > > > >>> it
> > > > > > > > >> > > > >>>>>> will provide a good reference for future
> > > > > > implementations.
> > > > > > > > >> > > > >>>>>>
> > > > > > > > >> > > > >>>>>> Regarding your comment on AS2, I wanted to
> > > clarify
> > > > > > that my
> > > > > > > > >> > > > >>>> specification
> > > > > > > > >> > > > >>>>>> references org.apache.kafka.common.Uuid.
> > > > > > > > >> > > > >>>>>> I believe we’re referring to the same class,
> > and
> > > it
> > > > > > might
> > > > > > > > >> just
> > > > > > > > >> > be
> > > > > > > > >> > > a
> > > > > > > > >> > > > >>>> small
> > > > > > > > >> > > > >>>>>> oversight due to the busy schedule.
> > > > > > > > >> > > > >>>>>>
> > > > > > > > >> > > > >>>>>> I want to express my gratitude once again for
> > > your
> > > > > many
> > > > > > > > >> > insightful
> > > > > > > > >> > > > >>>>>> comments, which have helped the discussion
> > > progress
> > > > > > > > smoothly.
> > > > > > > > >> > > > >>>>>>
> > > > > > > > >> > > > >>>>>> Best regards,
> > > > > > > > >> > > > >>>>>> TengYao
> > > > > > > > >> > > > >>>>>>
> > > > > > > > >> > > > >>>>>>
> > > > > > > > >> > > > >>>>>> Andrew Schofield <an...@live.com>
> 於
> > > > > > 2024年8月22日
> > > > > > > > 週四
> > > > > > > > >> > > > >>>> 下午11:28寫道：
> > > > > > > > >> > > > >>>>>>
> > > > > > > > >> > > > >>>>>>> Hi TengYao,
> > > > > > > > >> > > > >>>>>>> I’ve been reading through the comments and
> I’m
> > > > happy
> > > > > > that
> > > > > > > > >> the
> > > > > > > > >> > > > >> lobby
> > > > > > > > >> > > > >>>>>>> approach has not gained support.
> > > > > > > > >> > > > >>>>>>>
> > > > > > > > >> > > > >>>>>>> Assuming that this KIP is voted, I will be
> > happy
> > > > to
> > > > > > change
> > > > > > > > >> > > KIP-932
> > > > > > > > >> > > > >>> so
> > > > > > > > >> > > > >>>>>>> that it only supports client-side member ID
> > > > > > generation.
> > > > > > > > >> Because
> > > > > > > > >> > > > >> that
> > > > > > > > >> > > > >>>> KIP
> > > > > > > > >> > > > >>>>>>> is still
> > > > > > > > >> > > > >>>>>>> under development, I can do this in the
> first
> > > > > version
> > > > > > of
> > > > > > > > >> > > > >>>>>>> ShareGroupHeartbeat.
> > > > > > > > >> > > > >>>>>>>
> > > > > > > > >> > > > >>>>>>> AS2: For the encoding section, I suppose the
> > > > > specific
> > > > > > > > >> encoding
> > > > > > > > >> > > > >> which
> > > > > > > > >> > > > >>>>>>> is used is what org.apache.kafka.utils.Uuid
> > > uses.
> > > > > > > > >> > > > >>>>>>>
> > > > > > > > >> > > > >>>>>>> Thanks,
> > > > > > > > >> > > > >>>>>>> Andrew
> > > > > > > > >> > > > >>>>>>>
> > > > > > > > >> > > > >>>>>>>> On 14 Aug 2024, at 17:00, TengYao Chi <
> > > > > > > > kitingiao@gmail.com
> > > > > > > > >> >
> > > > > > > > >> > > > >>> wrote:
> > > > > > > > >> > > > >>>>>>>>
> > > > > > > > >> > > > >>>>>>>> Hello Apoorv,
> > > > > > > > >> > > > >>>>>>>> Thank you for your feedback.
> > > > > > > > >> > > > >>>>>>>> Regarding the questions you raised,
> > > > unfortunately,
> > > > > > this
> > > > > > > > KIP
> > > > > > > > >> > > > >> cannot
> > > > > > > > >> > > > >>>>>>>> guarantee the order of heartbeats. As with
> > many
> > > > > > classic
> > > > > > > > >> > > > >>> distributed
> > > > > > > > >> > > > >>>>>>> system
> > > > > > > > >> > > > >>>>>>>> challenges, what we can do is make our best
> > > > effort
> > > > > to
> > > > > > > > >> ensure
> > > > > > > > >> > > > >> that
> > > > > > > > >> > > > >>>>> there
> > > > > > > > >> > > > >>>>>>> are
> > > > > > > > >> > > > >>>>>>>> no idle members or stale assignments under
> > > normal
> > > > > > > > >> > circumstances.
> > > > > > > > >> > > > >>>>>>>>
> > > > > > > > >> > > > >>>>>>>> As for the lobby approach, I’m not a fan of
> > it
> > > > > > because it
> > > > > > > > >> > > > >> requires
> > > > > > > > >> > > > >>>>>>> adding a
> > > > > > > > >> > > > >>>>>>>> mechanism to maintain client state within
> the
> > > > > > > > >> ConsumerGroup,
> > > > > > > > >> > > > >>> which,
> > > > > > > > >> > > > >>>> in
> > > > > > > > >> > > > >>>>>>> my
> > > > > > > > >> > > > >>>>>>>> view, resembles something like a two-phase
> > > > commit.
> > > > > > This
> > > > > > > > >> would
> > > > > > > > >> > > > >>>>> introduce
> > > > > > > > >> > > > >>>>>>>> more complexity than the proposal in this
> > KIP,
> > > > > which
> > > > > > is
> > > > > > > > >> > > > >> something
> > > > > > > > >> > > > >>> we
> > > > > > > > >> > > > >>>>>>> want
> > > > > > > > >> > > > >>>>>>>> to avoid. KIP-848 aims to simplify the
> > existing
> > > > > > protocol,
> > > > > > > > >> and
> > > > > > > > >> > > > >>> while
> > > > > > > > >> > > > >>>>> the
> > > > > > > > >> > > > >>>>>>>> lobby approach is a good one, I believe it
> is
> > > not
> > > > > the
> > > > > > > > right
> > > > > > > > >> > fit
> > > > > > > > >> > > > >>> for
> > > > > > > > >> > > > >>>>> this
> > > > > > > > >> > > > >>>>>>>> particular situation.
> > > > > > > > >> > > > >>>>>>>>
> > > > > > > > >> > > > >>>>>>>> Best regards,
> > > > > > > > >> > > > >>>>>>>> TengYao
> > > > > > > > >> > > > >>>>>>>>
> > > > > > > > >> > > > >>>>>>>> TengYao Chi <ki...@gmail.com> 於
> > 2024年8月14日
> > > > 週三
> > > > > > > > >> 下午11:45寫道：
> > > > > > > > >> > > > >>>>>>>>
> > > > > > > > >> > > > >>>>>>>>> Hi David,
> > > > > > > > >> > > > >>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>> I really appreciate your review and
> > > suggestions.
> > > > > As
> > > > > > I am
> > > > > > > > >> > still
> > > > > > > > >> > > > >>>>> gaining
> > > > > > > > >> > > > >>>>>>>>> experience in writing KIPs, your input has
> > > been
> > > > > > > > incredibly
> > > > > > > > >> > > > >>>> helpful. I
> > > > > > > > >> > > > >>>>>>> am
> > > > > > > > >> > > > >>>>>>>>> currently applying your suggestions to the
> > KIP
> > > > and
> > > > > > will
> > > > > > > > >> > > > >> complete
> > > > > > > > >> > > > >>> it
> > > > > > > > >> > > > >>>>> as
> > > > > > > > >> > > > >>>>>>> soon
> > > > > > > > >> > > > >>>>>>>>> as possible.
> > > > > > > > >> > > > >>>>>>>>> Regarding the UUID part, I think we
> haven’t
> > > > > reached
> > > > > > a
> > > > > > > > >> > > > >> conclusion
> > > > > > > > >> > > > >>>>>>> yet.(So
> > > > > > > > >> > > > >>>>>>>>> far according to this thread)
> > > > > > > > >> > > > >>>>>>>>> However, I will review the current
> > > > implementation
> > > > > > in the
> > > > > > > > >> > Kafka
> > > > > > > > >> > > > >>>> `Uuid`
> > > > > > > > >> > > > >>>>>>>>> class and include a brief specification in
> > the
> > > > > KIP.
> > > > > > > > >> > > > >>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>> Once again, thank you so much for your
> help.
> > > > > > > > >> > > > >>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>> Best regards,
> > > > > > > > >> > > > >>>>>>>>> TengYao
> > > > > > > > >> > > > >>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> > > > 2024年8月14日
> > > > > 週三
> > > > > > > > >> > 下午11:14寫道：
> > > > > > > > >> > > > >>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>> hi Apoorv
> > > > > > > > >> > > > >>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>> As the memberId is now known to the
> > client,
> > > > and
> > > > > > client
> > > > > > > > >> > might
> > > > > > > > >> > > > >>> send
> > > > > > > > >> > > > >>>>> the
> > > > > > > > >> > > > >>>>>>>>>> leave
> > > > > > > > >> > > > >>>>>>>>>> group heartbeat on shutdown prior to
> > > receiving
> > > > > the
> > > > > > > > >> initial
> > > > > > > > >> > > > >>>> heartbeat
> > > > > > > > >> > > > >>>>>>>>>> response. If that's true then how do we
> > > > guarantee
> > > > > > that
> > > > > > > > >> the 2
> > > > > > > > >> > > > >>>>> requests
> > > > > > > > >> > > > >>>>>>> to
> > > > > > > > >> > > > >>>>>>>>>> join and leave will be processed in
> order,
> > > > which
> > > > > > could
> > > > > > > > >> still
> > > > > > > > >> > > > >>> leave
> > > > > > > > >> > > > >>>>>>> stale
> > > > > > > > >> > > > >>>>>>>>>> members or throw unknown member id
> > > exceptions?
> > > > > > > > >> > > > >>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>> This is definitely a good question. the
> > short
> > > > > > answer:
> > > > > > > > no
> > > > > > > > >> > > > >>> guarantee
> > > > > > > > >> > > > >>>>> but
> > > > > > > > >> > > > >>>>>>>>>> best
> > > > > > > > >> > > > >>>>>>>>>> efforts
> > > > > > > > >> > > > >>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>> Please notice the root cause is "we have
> no
> > > > > enough
> > > > > > time
> > > > > > > > >> to
> > > > > > > > >> > > > >> wait
> > > > > > > > >> > > > >>>>>>> member id
> > > > > > > > >> > > > >>>>>>>>>> (response) when closing consumer". Sadly,
> > we
> > > > can'
> > > > > > > > >> guarantee
> > > > > > > > >> > > > >> the
> > > > > > > > >> > > > >>>>>>> request
> > > > > > > > >> > > > >>>>>>>>>> order due to the same reason.
> > > > > > > > >> > > > >>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>> However, in contrast to previous
> behavior,
> > > > there
> > > > > > is one
> > > > > > > > >> big
> > > > > > > > >> > > > >>>> benefit
> > > > > > > > >> > > > >>>>>>> of new
> > > > > > > > >> > > > >>>>>>>>>> approach - we can try STONITH because we
> > know
> > > > the
> > > > > > > > member
> > > > > > > > >> id
> > > > > > > > >> > > > >>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>> Best,
> > > > > > > > >> > > > >>>>>>>>>> Chia-Ping
> > > > > > > > >> > > > >>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>> Apoorv Mittal <ap...@gmail.com>
> 於
> > > > > > 2024年8月14日
> > > > > > > > 週三
> > > > > > > > >> > > > >>>> 下午8:55寫道：
> > > > > > > > >> > > > >>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>> Hi TengYao,
> > > > > > > > >> > > > >>>>>>>>>>> Thanks for the KIP. Continuing on the
> > point
> > > > > which
> > > > > > > > Andrew
> > > > > > > > >> > > > >>>> mentioned
> > > > > > > > >> > > > >>>>> as
> > > > > > > > >> > > > >>>>>>>>>> AS1.
> > > > > > > > >> > > > >>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>> As the memberId is now known to the
> > client,
> > > > and
> > > > > > client
> > > > > > > > >> > might
> > > > > > > > >> > > > >>> send
> > > > > > > > >> > > > >>>>> the
> > > > > > > > >> > > > >>>>>>>>>> leave
> > > > > > > > >> > > > >>>>>>>>>>> group heartbeat on shutdown prior to
> > > receiving
> > > > > the
> > > > > > > > >> initial
> > > > > > > > >> > > > >>>>> heartbeat
> > > > > > > > >> > > > >>>>>>>>>>> response. If that's true then how do we
> > > > > guarantee
> > > > > > that
> > > > > > > > >> the
> > > > > > > > >> > 2
> > > > > > > > >> > > > >>>>>>> requests to
> > > > > > > > >> > > > >>>>>>>>>>> join and leave will be processed in
> order,
> > > > which
> > > > > > could
> > > > > > > > >> > still
> > > > > > > > >> > > > >>>> leave
> > > > > > > > >> > > > >>>>>>> stale
> > > > > > > > >> > > > >>>>>>>>>>> members or throw unknown member id
> > > exceptions?
> > > > > > > > >> > > > >>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>> Though the client side member id
> > generation
> > > is
> > > > > > helpful
> > > > > > > > >> > which
> > > > > > > > >> > > > >>> will
> > > > > > > > >> > > > >>>>>>>>>> represent
> > > > > > > > >> > > > >>>>>>>>>>> the same group perspective as from
> client
> > > and
> > > > > > broker's
> > > > > > > > >> end.
> > > > > > > > >> > > > >>> But I
> > > > > > > > >> > > > >>>>>>> think
> > > > > > > > >> > > > >>>>>>>>>> the
> > > > > > > > >> > > > >>>>>>>>>>> major concern we want to solve here is
> > Stale
> > > > > > Partition
> > > > > > > > >> > > > >>>> Assignments
> > > > > > > > >> > > > >>>>>>> which
> > > > > > > > >> > > > >>>>>>>>>>> might still exist with the new
> approach. I
> > > am
> > > > > > leaning
> > > > > > > > >> > towards
> > > > > > > > >> > > > >>> the
> > > > > > > > >> > > > >>>>>>>>>>> suggestion mentioned by Andrew where
> > > partition
> > > > > > > > >> assignment
> > > > > > > > >> > > > >>>> triggers
> > > > > > > > >> > > > >>>>> on
> > > > > > > > >> > > > >>>>>>>>>>> subsequent heartbeat when client
> > > acknowledges
> > > > > the
> > > > > > > > >> initial
> > > > > > > > >> > > > >>>>> heartbeat,
> > > > > > > > >> > > > >>>>>>>>>>> delayed partition assignment.
> > > > > > > > >> > > > >>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>> Though on a separate note, I have a
> > > different
> > > > > > > > question.
> > > > > > > > >> > What
> > > > > > > > >> > > > >>>>> happens
> > > > > > > > >> > > > >>>>>>>>>> when
> > > > > > > > >> > > > >>>>>>>>>>> there is an issue with the client which
> > > sends
> > > > > the
> > > > > > > > >> initial
> > > > > > > > >> > > > >>>> heartbeat
> > > > > > > > >> > > > >>>>>>>>>> without
> > > > > > > > >> > > > >>>>>>>>>>> memberId, then crashes and restarts? I
> > think
> > > > we
> > > > > > must
> > > > > > > > be
> > > > > > > > >> > > > >>>>> re-triggering
> > > > > > > > >> > > > >>>>>>>>>>> assignments and expiring members only
> > after
> > > > the
> > > > > > > > >> heartbeat
> > > > > > > > >> > > > >>> session
> > > > > > > > >> > > > >>>>>>>>>> timeout?
> > > > > > > > >> > > > >>>>>>>>>>> If that's true then shall delayed
> > partition
> > > > > > assignment
> > > > > > > > >> can
> > > > > > > > >> > > > >> help
> > > > > > > > >> > > > >>>>>>> benefit
> > > > > > > > >> > > > >>>>>>>>>> us
> > > > > > > > >> > > > >>>>>>>>>>> from this situation as well?
> > > > > > > > >> > > > >>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>> Regards,
> > > > > > > > >> > > > >>>>>>>>>>> Apoorv Mittal
> > > > > > > > >> > > > >>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>> On Wed, Aug 14, 2024 at 12:51 PM David
> > Jacot
> > > > > > > > >> > > > >>>>>>>>>> <dj...@confluent.io.invalid>
> > > > > > > > >> > > > >>>>>>>>>>> wrote:
> > > > > > > > >> > > > >>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>> Hi Andrew,
> > > > > > > > >> > > > >>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>> Personally, I don't like the lobby
> > > approach.
> > > > It
> > > > > > makes
> > > > > > > > >> > things
> > > > > > > > >> > > > >>>> more
> > > > > > > > >> > > > >>>>>>>>>>>> complicated and it would require
> changing
> > > the
> > > > > > records
> > > > > > > > >> on
> > > > > > > > >> > the
> > > > > > > > >> > > > >>>>> server
> > > > > > > > >> > > > >>>>>>>>>> too.
> > > > > > > > >> > > > >>>>>>>>>>>> This is why I initially suggested the
> > > > rejected
> > > > > > > > >> alternative
> > > > > > > > >> > > > >> #2
> > > > > > > > >> > > > >>>>> which
> > > > > > > > >> > > > >>>>>>> is
> > > > > > > > >> > > > >>>>>>>>>>>> pretty close but also not perfect.
> > > > > > > > >> > > > >>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>> I'd like to clarify one thing. The
> > > > > > > > >> ConsumerGroupHeartbeat
> > > > > > > > >> > > > >> API
> > > > > > > > >> > > > >>>>>>> already
> > > > > > > > >> > > > >>>>>>>>>>>> supports generating the member id on
> the
> > > > client
> > > > > > so we
> > > > > > > > >> > don't
> > > > > > > > >> > > > >>> need
> > > > > > > > >> > > > >>>>> any
> > > > > > > > >> > > > >>>>>>>>>>>> conditional logic on the client side.
> > This
> > > is
> > > > > > > > actually
> > > > > > > > >> > what
> > > > > > > > >> > > > >> we
> > > > > > > > >> > > > >>>>>>> wanted
> > > > > > > > >> > > > >>>>>>>>>> to
> > > > > > > > >> > > > >>>>>>>>>>> do
> > > > > > > > >> > > > >>>>>>>>>>>> in the first place but the idea got
> > pushed
> > > > back
> > > > > > by
> > > > > > > > >> Magnus
> > > > > > > > >> > > > >> back
> > > > > > > > >> > > > >>>>> then
> > > > > > > > >> > > > >>>>>>>>>>> because
> > > > > > > > >> > > > >>>>>>>>>>>> generating uuid from librdkafka
> required
> > a
> > > > new
> > > > > > > > >> dependency.
> > > > > > > > >> > > > >> It
> > > > > > > > >> > > > >>>>> turns
> > > > > > > > >> > > > >>>>>>>>>> out
> > > > > > > > >> > > > >>>>>>>>>>>> that librdkafka has that dependency
> > today.
> > > In
> > > > > > > > >> retrospect,
> > > > > > > > >> > we
> > > > > > > > >> > > > >>>>> should
> > > > > > > > >> > > > >>>>>>>>>> have
> > > > > > > > >> > > > >>>>>>>>>>>> pushed back on this. Long story short,
> we
> > > can
> > > > > > just do
> > > > > > > > >> it.
> > > > > > > > >> > > > >> The
> > > > > > > > >> > > > >>>>>>>>>> proposal in
> > > > > > > > >> > > > >>>>>>>>>>>> this KIP is to make the member id
> > required
> > > in
> > > > > > future
> > > > > > > > >> > > > >> versions.
> > > > > > > > >> > > > >>>> We
> > > > > > > > >> > > > >>>>>>>>>> could
> > > > > > > > >> > > > >>>>>>>>>>>> also decide not to do it and to keep
> > > > supporting
> > > > > > both
> > > > > > > > >> > > > >>>> approaches. I
> > > > > > > > >> > > > >>>>>>>>>> would
> > > > > > > > >> > > > >>>>>>>>>>>> also be fine with this.
> > > > > > > > >> > > > >>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>> Best,
> > > > > > > > >> > > > >>>>>>>>>>>> David
> > > > > > > > >> > > > >>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>> On Wed, Aug 14, 2024 at 12:30 PM Andrew
> > > > > > Schofield <
> > > > > > > > >> > > > >>>>>>>>>>>> andrew_schofield@live.com>
> > > > > > > > >> > > > >>>>>>>>>>>> wrote:
> > > > > > > > >> > > > >>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>> Hi TengYao,
> > > > > > > > >> > > > >>>>>>>>>>>>> Thanks for your response. I’ll have
> just
> > > one
> > > > > > more
> > > > > > > > try
> > > > > > > > >> to
> > > > > > > > >> > > > >>>>> persuade.
> > > > > > > > >> > > > >>>>>>>>>>>>> I feel that I will need to follow the
> > > > approach
> > > > > > with
> > > > > > > > >> > KIP-932
> > > > > > > > >> > > > >>>> when
> > > > > > > > >> > > > >>>>>>>>>> we’ve
> > > > > > > > >> > > > >>>>>>>>>>>>> made a decision, so I do have more
> than
> > a
> > > > > > passing
> > > > > > > > >> > interest
> > > > > > > > >> > > > >> in
> > > > > > > > >> > > > >>>>> this.
> > > > > > > > >> > > > >>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>> A group member in the lobby is in the
> > > group,
> > > > > > but it
> > > > > > > > >> does
> > > > > > > > >> > > > >> not
> > > > > > > > >> > > > >>>> have
> > > > > > > > >> > > > >>>>>>>>>> any
> > > > > > > > >> > > > >>>>>>>>>>>>> assignments. A member of a consumer
> > group
> > > > can
> > > > > > have
> > > > > > > > no
> > > > > > > > >> > > > >>> assigned
> > > > > > > > >> > > > >>>>>>>>>>>>> partitions (such as 5 CG members
> > > subscribed
> > > > > to a
> > > > > > > > topic
> > > > > > > > >> > > > >> with 4
> > > > > > > > >> > > > >>>>>>>>>>>> partitions),
> > > > > > > > >> > > > >>>>>>>>>>>>> so it’s a situation that consumer
> group
> > > > > members
> > > > > > > > >> already
> > > > > > > > >> > > > >>> expect.
> > > > > > > > >> > > > >>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>> One of Kafka’s strengths is the way
> that
> > > we
> > > > > > handle
> > > > > > > > API
> > > > > > > > >> > > > >>>>> versioning.
> > > > > > > > >> > > > >>>>>>>>>>>>> But, there is a cost - the behaviour
> is
> > > > > > different
> > > > > > > > >> > depending
> > > > > > > > >> > > > >>> on
> > > > > > > > >> > > > >>>>> the
> > > > > > > > >> > > > >>>>>>>>>> RPC
> > > > > > > > >> > > > >>>>>>>>>>>>> version. KIP-848 is on the cusp of
> > > > completion,
> > > > > > but
> > > > > > > > >> we’re
> > > > > > > > >> > > > >>>> already
> > > > > > > > >> > > > >>>>>>>>>> adding
> > > > > > > > >> > > > >>>>>>>>>>>>> conditional logic for v0/v1 for
> > > > > > > > >> ConsumerGroupHeartbeat.
> > > > > > > > >> > > > >>> That’s
> > > > > > > > >> > > > >>>> a
> > > > > > > > >> > > > >>>>>>>>>> pity.
> > > > > > > > >> > > > >>>>>>>>>>>>> Only a minor issue, but it’s
> > unfortunate.
> > > > > > > > >> > > > >>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>> Thanks,
> > > > > > > > >> > > > >>>>>>>>>>>>> Andrew
> > > > > > > > >> > > > >>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>> On 14 Aug 2024, at 08:47, TengYao
> Chi <
> > > > > > > > >> > > > >> kitingiao@gmail.com>
> > > > > > > > >> > > > >>>>>>>>>> wrote:
> > > > > > > > >> > > > >>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>> Hello Andrew
> > > > > > > > >> > > > >>>>>>>>>>>>>> Thank you for your thoughtful
> > suggestions
> > > > and
> > > > > > > > getting
> > > > > > > > >> > the
> > > > > > > > >> > > > >>>>>>>>>> discussion
> > > > > > > > >> > > > >>>>>>>>>>>>> going.
> > > > > > > > >> > > > >>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>> To AS1:
> > > > > > > > >> > > > >>>>>>>>>>>>>> In the current scenario where the
> > server
> > > > > > generates
> > > > > > > > >> the
> > > > > > > > >> > > > >> UUID,
> > > > > > > > >> > > > >>>> if
> > > > > > > > >> > > > >>>>>>>>>> the
> > > > > > > > >> > > > >>>>>>>>>>>>> client
> > > > > > > > >> > > > >>>>>>>>>>>>>> shuts down before receiving the
> > memberId
> > > > > > generated
> > > > > > > > by
> > > > > > > > >> > the
> > > > > > > > >> > > > >> GC
> > > > > > > > >> > > > >>>>>>>>>>>> (regardless
> > > > > > > > >> > > > >>>>>>>>>>>>> of
> > > > > > > > >> > > > >>>>>>>>>>>>>> whether it’s a graceful shutdown or
> > not),
> > > > the
> > > > > > GC
> > > > > > > > will
> > > > > > > > >> > > > >> still
> > > > > > > > >> > > > >>>> have
> > > > > > > > >> > > > >>>>>>>>>> to
> > > > > > > > >> > > > >>>>>>>>>>>> wait
> > > > > > > > >> > > > >>>>>>>>>>>>>> for the heartbeat timeout because the
> > > > client
> > > > > > > > doesn’t
> > > > > > > > >> > know
> > > > > > > > >> > > > >>> its
> > > > > > > > >> > > > >>>>>>>>>>> memberId.
> > > > > > > > >> > > > >>>>>>>>>>>>>> This KIP indeed cannot completely
> > resolve
> > > > the
> > > > > > > > >> > idempotency
> > > > > > > > >> > > > >>>> issue,
> > > > > > > > >> > > > >>>>>>>>>> but
> > > > > > > > >> > > > >>>>>>>>>>> it
> > > > > > > > >> > > > >>>>>>>>>>>>> can
> > > > > > > > >> > > > >>>>>>>>>>>>>> better handle shutdown scenarios
> under
> > > > normal
> > > > > > > > >> > > > >> circumstances
> > > > > > > > >> > > > >>>>>>>>>> because
> > > > > > > > >> > > > >>>>>>>>>>> the
> > > > > > > > >> > > > >>>>>>>>>>>>>> client always knows its memberId.
> Even
> > if
> > > > the
> > > > > > > > client
> > > > > > > > >> > shuts
> > > > > > > > >> > > > >>>> down
> > > > > > > > >> > > > >>>>>>>>>>>>> immediately
> > > > > > > > >> > > > >>>>>>>>>>>>>> after the initial heartbeat, as long
> as
> > > it
> > > > > > > > performs a
> > > > > > > > >> > > > >>> graceful
> > > > > > > > >> > > > >>>>>>>>>>> shutdown
> > > > > > > > >> > > > >>>>>>>>>>>>> and
> > > > > > > > >> > > > >>>>>>>>>>>>>> sends a leave heartbeat, the GC can
> > > manage
> > > > > the
> > > > > > > > >> situation
> > > > > > > > >> > > > >> and
> > > > > > > > >> > > > >>>>>>>>>> remove
> > > > > > > > >> > > > >>>>>>>>>>> the
> > > > > > > > >> > > > >>>>>>>>>>>>>> member. Therefore, the goal of this
> KIP
> > > is
> > > > to
> > > > > > > > address
> > > > > > > > >> > the
> > > > > > > > >> > > > >>>> issue
> > > > > > > > >> > > > >>>>>>>>>> where
> > > > > > > > >> > > > >>>>>>>>>>>> the
> > > > > > > > >> > > > >>>>>>>>>>>>>> GC has to wait for the heartbeat
> > timeout
> > > > due
> > > > > > to the
> > > > > > > > >> > client
> > > > > > > > >> > > > >>>>> leaving
> > > > > > > > >> > > > >>>>>>>>>>>>> without
> > > > > > > > >> > > > >>>>>>>>>>>>>> knowing its memberId, which leads to
> > > > reduced
> > > > > > > > >> throughput
> > > > > > > > >> > > > >> and
> > > > > > > > >> > > > >>>>>>>>>> limited
> > > > > > > > >> > > > >>>>>>>>>>>>>> scalability.
> > > > > > > > >> > > > >>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>> The solution you suggest has also
> been
> > > > > > proposed by
> > > > > > > > >> > David.
> > > > > > > > >> > > > >>> The
> > > > > > > > >> > > > >>>>>>>>>> concern
> > > > > > > > >> > > > >>>>>>>>>>>>> with
> > > > > > > > >> > > > >>>>>>>>>>>>>> this approach is that it introduces
> > > > > additional
> > > > > > > > >> > complexity
> > > > > > > > >> > > > >>> for
> > > > > > > > >> > > > >>>>>>>>>>>>>> compatibility, as the new server
> would
> > > not
> > > > > > > > >> immediately
> > > > > > > > >> > add
> > > > > > > > >> > > > >>> the
> > > > > > > > >> > > > >>>>>>>>>> member
> > > > > > > > >> > > > >>>>>>>>>>>> to
> > > > > > > > >> > > > >>>>>>>>>>>>>> the group, while the old server
> would.
> > > This
> > > > > > > > requires
> > > > > > > > >> > > > >> clients
> > > > > > > > >> > > > >>>> to
> > > > > > > > >> > > > >>>>>>>>>>>>>> differentiate whether their memberId
> > has
> > > > been
> > > > > > added
> > > > > > > > >> to
> > > > > > > > >> > the
> > > > > > > > >> > > > >>>> group
> > > > > > > > >> > > > >>>>>>>>>> or
> > > > > > > > >> > > > >>>>>>>>>>>> not,
> > > > > > > > >> > > > >>>>>>>>>>>>>> which could result in unexpected
> logs.
> > > > > > > > >> > > > >>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>> Best Regards,
> > > > > > > > >> > > > >>>>>>>>>>>>>> TengYao
> > > > > > > > >> > > > >>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>> Andrew Schofield <
> > > > andrew_schofield@live.com>
> > > > > 於
> > > > > > > > >> > 2024年8月14日
> > > > > > > > >> > > > >>> 週三
> > > > > > > > >> > > > >>>>>>>>>>>> 上午12:29寫道：
> > > > > > > > >> > > > >>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>>> Hi TengYao,
> > > > > > > > >> > > > >>>>>>>>>>>>>>> Thanks for the KIP. I wonder if
> > there’s
> > > a
> > > > > > > > different
> > > > > > > > >> way
> > > > > > > > >> > > > >> to
> > > > > > > > >> > > > >>>>> close
> > > > > > > > >> > > > >>>>>>>>>>> what
> > > > > > > > >> > > > >>>>>>>>>>>>>>> is quite a small window.
> > > > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>>> AS1: It is true that the initial
> > > heartbeat
> > > > > is
> > > > > > not
> > > > > > > > >> > > > >>> idempotent,
> > > > > > > > >> > > > >>>>> but
> > > > > > > > >> > > > >>>>>>>>>>> this
> > > > > > > > >> > > > >>>>>>>>>>>>>>> remains
> > > > > > > > >> > > > >>>>>>>>>>>>>>> true with this KIP. It’s just
> > > differently
> > > > > not
> > > > > > > > >> > idempotent.
> > > > > > > > >> > > > >>> If
> > > > > > > > >> > > > >>>>> the
> > > > > > > > >> > > > >>>>>>>>>>>> client
> > > > > > > > >> > > > >>>>>>>>>>>>>>> makes its
> > > > > > > > >> > > > >>>>>>>>>>>>>>> own member ID, sends a request and
> > dies,
> > > > the
> > > > > > GC
> > > > > > > > will
> > > > > > > > >> > > > >> still
> > > > > > > > >> > > > >>>> have
> > > > > > > > >> > > > >>>>>>>>>>> added
> > > > > > > > >> > > > >>>>>>>>>>>>>>> the member to the group and it will
> > hang
> > > > > > around
> > > > > > > > >> until
> > > > > > > > >> > the
> > > > > > > > >> > > > >>>>> session
> > > > > > > > >> > > > >>>>>>>>>>>>> expires.
> > > > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>>> I wonder if the GC could still
> > generate
> > > > the
> > > > > > member
> > > > > > > > >> ID
> > > > > > > > >> > in
> > > > > > > > >> > > > >>>>>>>>>> response to
> > > > > > > > >> > > > >>>>>>>>>>>> the
> > > > > > > > >> > > > >>>>>>>>>>>>>>> first
> > > > > > > > >> > > > >>>>>>>>>>>>>>> heartbeat, and put the member in a
> > > special
> > > > > > PENDING
> > > > > > > > >> > state
> > > > > > > > >> > > > >>> with
> > > > > > > > >> > > > >>>>> no
> > > > > > > > >> > > > >>>>>>>>>>>>>>> assignments until the client sends
> the
> > > > next
> > > > > > > > >> heartbeat,
> > > > > > > > >> > > > >> thus
> > > > > > > > >> > > > >>>>>>>>>>> confirming
> > > > > > > > >> > > > >>>>>>>>>>>>> it
> > > > > > > > >> > > > >>>>>>>>>>>>>>> has received the member ID. This
> would
> > > not
> > > > > be
> > > > > > a
> > > > > > > > >> > protocol
> > > > > > > > >> > > > >>>> change
> > > > > > > > >> > > > >>>>>>>>>> at
> > > > > > > > >> > > > >>>>>>>>>>>> all,
> > > > > > > > >> > > > >>>>>>>>>>>>>>> just
> > > > > > > > >> > > > >>>>>>>>>>>>>>> a change to the GC to keep a new
> > member
> > > in
> > > > > the
> > > > > > > > lobby
> > > > > > > > >> > > > >> until
> > > > > > > > >> > > > >>>> it’s
> > > > > > > > >> > > > >>>>>>>>>>>>> comfirmed
> > > > > > > > >> > > > >>>>>>>>>>>>>>> it knows its member ID.
> > > > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>>> Thanks,
> > > > > > > > >> > > > >>>>>>>>>>>>>>> Andrew
> > > > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>>>> On 13 Aug 2024, at 15:59, TengYao
> > Chi <
> > > > > > > > >> > > > >>> kitingiao@gmail.com>
> > > > > > > > >> > > > >>>>>>>>>> wrote:
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>>>> Hi Chia-Ping,
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>>>> Thanks for review and suggestions.
> > > > > > > > >> > > > >>>>>>>>>>>>>>>> I have updated the content of KIP
> > > > > > accordingly.
> > > > > > > > >> > > > >>>>>>>>>>>>>>>> Please take a look.
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>>>> Best regards,
> > > > > > > > >> > > > >>>>>>>>>>>>>>>> TengYao
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>>>> Chia-Ping Tsai <
> chia7712@apache.org>
> > 於
> > > > > > > > 2024年8月13日
> > > > > > > > >> 週二
> > > > > > > > >> > > > >>>>> 下午9:45寫道：
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>> hi TengYao
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>> thanks for this KIP.
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>> 1) could you please describe the
> > > > > > before/after
> > > > > > > > >> > behavior
> > > > > > > > >> > > > >> in
> > > > > > > > >> > > > >>>> the
> > > > > > > > >> > > > >>>>>>>>>>>>> "Proposed
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>> Changes" section? IIRC, current
> RPC
> > > > allows
> > > > > > HB
> > > > > > > > >> having
> > > > > > > > >> > > > >>> member
> > > > > > > > >> > > > >>>>> id
> > > > > > > > >> > > > >>>>>>>>>>>>>>> generated by
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>> client, right? If HB has no member
> > ID,
> > > > > > server
> > > > > > > > will
> > > > > > > > >> > > > >>> generate
> > > > > > > > >> > > > >>>>> one
> > > > > > > > >> > > > >>>>>>>>>>> and
> > > > > > > > >> > > > >>>>>>>>>>>>> then
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>> return. The new behavior will
> > enforce
> > > HB
> > > > > > "must"
> > > > > > > > >> have
> > > > > > > > >> > > > >>> member
> > > > > > > > >> > > > >>>>> ID.
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>> 2) could you please write the
> > version
> > > > > number
> > > > > > > > >> > explicitly
> > > > > > > > >> > > > >>> in
> > > > > > > > >> > > > >>>>> the
> > > > > > > > >> > > > >>>>>>>>>> KIP
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>> 3) how new client code handle the
> > old
> > > > HB?
> > > > > > Does
> > > > > > > > it
> > > > > > > > >> > > > >> always
> > > > > > > > >> > > > >>>>>>>>>> generate
> > > > > > > > >> > > > >>>>>>>>>>>>> member
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>> ID on client-side even though that
> > is
> > > > not
> > > > > > > > >> restricted?
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>> Best,
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>> Chia-Ping
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>> On 2024/08/13 06:20:42 TengYao Chi
> > > > wrote:
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> Hello everyone,
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> I would like to start a
> discussion
> > > > thread
> > > > > > on
> > > > > > > > >> > KIP-1082,
> > > > > > > > >> > > > >>>> which
> > > > > > > > >> > > > >>>>>>>>>>>> proposes
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> enabling id generation for
> clients
> > > over
> > > > > the
> > > > > > > > >> > > > >>>>>>>>>>> ConsumerGroupHeartbeat
> > > > > > > > >> > > > >>>>>>>>>>>>> RPC.
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> Here is the KIP Link: KIP-1082
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> <
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>
> > > > > > > > >> > > > >>>>>
> > > > > > > > >> > > > >>>>
> > > > > > > > >> > > > >>>
> > > > > > > > >> > > > >>
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> Please take a look and let me
> know
> > > what
> > > > > you
> > > > > > > > >> think,
> > > > > > > > >> > > > >> and I
> > > > > > > > >> > > > >>>>> would
> > > > > > > > >> > > > >>>>>>>>>>>>>>> appreciate
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> any suggestions and feedback.
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> Best regards,
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>>> TengYao
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>>
> > > > > > > > >> > > > >>>>>>>>>
> > > > > > > > >> > > > >>>>>>>
> > > > > > > > >> > > > >>>>>>>
> > > > > > > > >> > > > >>>>>
> > > > > > > > >> > > > >>>>
> > > > > > > > >> > > > >>>
> > > > > > > > >> > > > >>
> > > > > > > > >> > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[David Jacot <dj...@confluent.io.INVALID>]

Hi Chia-Ping,

> However, my concern is that consumers are now required to generate the
member ID, so as a responsible server, we need to handle the incorrect
client behavior. At least, based on David's comment, we can create another
member id when it is already used and it is rejoining the group (member
epoch = 0).

I don't disagree with the concern. However, I don't see how we could make
it work in practice. Let's say that a member XYZ sends a heartbeat with
epoch zero. The server cannot know whether it is a new member joining for
the first time or an existing member rejoining because it was fenced (e.g.
wrong epoch, session timed out, etc.). We cannot differentiate the two
cases. Hence we cannot handle it on the server side. The only viable
alternative is to keep the member id generation on the server side if we
are concerned by clients not able to generate unique uuids on their side. I
tend to believe that generating a uuid on the client should not be that
difficult so I am not too concerned about it. How do you see
this, Chia-Ping?

Cheers,
David

On Mon, Sep 23, 2024 at 4:13 PM Chia-Ping Tsai <ch...@gmail.com> wrote:

> hi David
>
> > For instance, we could check whether the member id is already used and we
> could regenerate another one if it is. However, I tend to believe that the
> risk of collision within a group is neglectable.
>
> I agree the risk is negligible when the member ID is generated by the
> server, as we can always ensure the server behaves responsibly. 😀
> Therefore, I don't believe the additional check for the member ID is
> necessary if the member id is created by server.
>
> However, my concern is that consumers are now required to generate the
> member ID, so as a responsible server, we need to handle the incorrect
> client behavior. At least, based on David's comment, we can create another
> member id when it is already used and it is rejoining the group (member
> epoch = 0).
>
> By the way, another related issue is that a duplicate consumer may
> inadvertently terminate another consumer, as it will always send a
> LEAVE_GROUP request upon closing, even if it hasn't received a heartbeat
> response. However, I tend to believe the risk is negligible 😀
>
> Best,
> Chia-Ping
>
>
> David Jacot <dj...@confluent.io.invalid> 於 2024年9月23日 週一 下午9:07寫道：
>
> > Hi Chia-Ping,
> >
> > > Checks whether the member epoch matches the member epoch in its current
> > assignment. FENCED_MEMBER_EPOCH is returned otherwise. The member is also
> > removed from the group.
> >
> > We did not implement it like this in the end. We keep the member and only
> > request him to re-synchronise himself by returning the
> FENCED_MEMBER_EPOCH
> > error. We can remove it from the KIP.
> >
> > > Given that member id is the incarnation of consumer, it is hard to
> detect
> > the scenario of "two consumers have the same member id" (it may be caused
> > by the lost response). I guess the best approach is the later -created
> > consumer should be idle with fenced error, and we should add this to the
> > KIP (maybe we should revise the error message carried by
> > `FencedMemberEpochException`). BUT, we need to rethink the approach if
> > above description "The member is also removed from the group" is true
> >
> > I don't think that we have a reliable way to detect whether two members
> end
> > up using the same member id. I don't think that it is possible if we
> > generate the member id on the client side. By the way, we don't handle
> this
> > case either on the server side today. We blindly generate a member id and
> > use it. However, we could do better on the server side. For instance, we
> > could check whether the member id is already used and we could regenerate
> > another one if it is. However, I tend to believe that the risk of
> collision
> > within a group is neglectable.
> >
> > Best,
> > David
> >
> > On Fri, Sep 20, 2024 at 8:11 PM Chia-Ping Tsai <ch...@gmail.com>
> wrote:
> >
> > > wait a minute. There is a description about fenced error in the KIP-848
> > >
> > > > Checks whether the member epoch matches the member epoch in its
> current
> > > assignment. FENCED_MEMBER_EPOCH is returned otherwise. The member is
> also
> > > removed from the group.
> > >
> > > I failed to find the code of implementing "The member is also removed
> > from
> > > the group.". I will appreciate it if someone can share the code link to
> > me.
> > >
> > > Let's assume it will be implemented eventually, and I feel that will
> > cause
> > > a serious issue as the consumers having the same member id will remove
> > the
> > > existing  member and then create itself member?
> > >
> > > > Once the client receives this exception, it should consider the
> > exception
> > > as a serious error and stop the process, and the user will need to
> > assign a
> > > new memberId.
> > >
> > > Given that member id is the incarnation of consumer, it is hard to
> detect
> > > the scenario of "two consumers have the same member id" (it may be
> caused
> > > by the lost response). I guess the best approach is the later -created
> > > consumer should be idle with fenced error, and we should add this to
> the
> > > KIP (maybe we should revise the error message carried by
> > > `FencedMemberEpochException`). BUT, we need to rethink the approach if
> > > above description "The member is also removed from the group" is true
> > >
> > > Best,
> > > Chia-Ping
> > >
> > >
> > >
> > > TengYao Chi <ki...@gmail.com> 於 2024年9月20日 週五 下午8:17寫道：
> > >
> > > > Hi Chia-Ping,
> > > >
> > > > Thanks for pointing out this issue.
> > > >
> > > > I’m thinking that maybe we might need to define a new Exception to
> > handle
> > > > this scenario.
> > > > Once the client receives this exception, it should consider the
> > exception
> > > > as a serious error and stop the process, and the user will need to
> > > assign a
> > > > new memberId.
> > > > What do you think?
> > > >
> > > > Best Regards,
> > > > TengYao
> > > >
> > > > Chia-Ping Tsai <ch...@apache.org> 於 2024年9月20日 週五 下午7:00寫道：
> > > >
> > > > > > This part is not clear either. It basically says that if a member
> > > joins
> > > > > with an existing member id but a different epoch, it will be
> fenced.
> > > Then
> > > > > it must rejoin with the same member id and epoch zero. This is
> > already
> > > > the
> > > > > current behavior and it does not help with detecting duplicates,
> > right?
> > > > >
> > > > > The duplicates issue is interesting. The member id is generated on
> > > > > server-side by UUID before, so it seems to me the fenced member
> epoch
> > > > > happens only if the client miss the response with bumped epoch, and
> > > this
> > > > > case is recoverable.
> > > > >
> > > > > however, v1 brings a different fenced epoch scenario. Users, now,
> > have
> > > > > responsibility to generate unique member id, hence clients may
> > > encounter
> > > > > infinite fenced error if there are >1 clients having same member id
> > due
> > > > to
> > > > > config error (or other engineer error)
> > > > >
> > > > > Maybe we can highlight this scenario if we start to requires
> clients
> > to
> > > > > generate unique member id.
> > > > >
> > > > > Best,
> > > > > Chia-Ping
> > > > >
> > > > > On 2024/09/19 18:36:48 David Jacot wrote:
> > > > > > Hi,
> > > > > >
> > > > > > Thanks for the update. I have a few nits:
> > > > > >
> > > > > > > If the member ID is null or empty, the server will reject the
> > > request
> > > > > > with an InvalidRequestException.
> > > > > > We should clarify that this should only apply to version >= 1.
> > > > > >
> > > > > > > The consumer instance must generate a member ID, and this ID
> > should
> > > > > > remain consistent for the duration of the consumer's session.
> > Here, a
> > > > > > "session" is defined as the period from the consumer's first
> > > heartbeat
> > > > > > until it leaves the group, either through a graceful shutdown, a
> > > > > heartbeat
> > > > > > timeout, or the process stopping or dying. The consumer instance
> > > should
> > > > > > reuse the same member ID for all heartbeats and rejoin attempts
> to
> > > > > maintain
> > > > > > continuity within the group.
> > > > > >
> > > > > > This part is not clear to me. When the member leaves the group,
> it
> > > > should
> > > > > > not reset the member id. I would rather say that the member must
> > > > generate
> > > > > > its member id when it starts and it must keep it until the
> process
> > > > stops.
> > > > > > It is basically an incarnation of the process.
> > > > > >
> > > > > > > If a conflict arises where the member ID generated by the
> client
> > is
> > > > > > detected to be a duplicate within the same group (for example,
> the
> > > same
> > > > > > member ID is associated with another active member in the group),
> > the
> > > > > > server will handle this by comparing the memberEpoch values of
> the
> > > > > > conflicting members. The member with the lower memberEpoch is
> > > > considered
> > > > > > outdated and will be fenced off by the server. When this occurs,
> > the
> > > > > server
> > > > > > responds with a FENCED_MEMBER_EPOCH error to the client,
> signaling
> > it
> > > > to
> > > > > > rejoin the group with the same member ID while resetting the
> > > > memberEpoch
> > > > > to
> > > > > > zero. This ensures that the client properly resynchronizes and
> > > > maintains
> > > > > > the continuity and consistency of the group membership.
> > > > > >
> > > > > > This part is not clear either. It basically says that if a member
> > > joins
> > > > > > with an existing member id but a different epoch, it will be
> > fenced.
> > > > Then
> > > > > > it must rejoin with the same member id and epoch zero. This is
> > > already
> > > > > the
> > > > > > current behavior and it does not help with detecting duplicates,
> > > right?
> > > > > > Should we just remove the paragraph?
> > > > > >
> > > > > > > A member ID mismatch occurs within a session: If the server
> > > detects a
> > > > > > mismatch between the provided member ID and the expected member
> ID
> > > for
> > > > an
> > > > > > ongoing session, it should return a UNKNOWN_MEMBER_ID  error.
> > > > > >
> > > > > > How could we detect a mismatch between the provided and the
> > expected
> > > > > member
> > > > > > id? My understanding is that we can only know whether the
> provided
> > > > member
> > > > > > id exists or not. This is already implemented.
> > > > > >
> > > > > > Thanks,
> > > > > > David
> > > > > >
> > > > > > On Sat, Sep 14, 2024 at 9:31 AM TengYao Chi <kitingiao@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Hello everyone,
> > > > > > >
> > > > > > > Since this KIP has been fully discussed, I will initiate a vote
> > for
> > > > it
> > > > > next
> > > > > > > Monday.
> > > > > > > Thank you and have a nice weekend.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > TengYao
> > > > > > >
> > > > > > > TengYao Chi <ki...@gmail.com> 於 2024年9月5日 週四 下午2:19寫道：
> > > > > > >
> > > > > > > > Hello everyone,
> > > > > > > >
> > > > > > > > KT2: It looks like everyone who has expressed an opinion
> > supports
> > > > the
> > > > > > > > second option: “Document a recommendation for clients to use
> > > UUIDs
> > > > as
> > > > > > > > member IDs, without strictly enforcing it.”
> > > > > > > > I have updated the KIP accordingly.
> > > > > > > > Please take a look, and let me know if you have any thoughts
> or
> > > > > feedback.
> > > > > > > >
> > > > > > > > Thank you!
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > TengYao
> > > > > > > >
> > > > > > > > Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月30日 週五
> 下午9:56寫道：
> > > > > > > >
> > > > > > > >> hi TengYao
> > > > > > > >>
> > > > > > > >> KT2: +1 to second approach
> > > > > > > >>
> > > > > > > >> Best,
> > > > > > > >> Chia-Ping
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> David Jacot <dj...@confluent.io.invalid> 於 2024年8月30日 週五
> > > > 下午9:15寫道：
> > > > > > > >>
> > > > > > > >> > Hi TengYao,
> > > > > > > >> >
> > > > > > > >> > KT2: I don't think that we can realistically validate the
> > uuid
> > > > on
> > > > > the
> > > > > > > >> > server. It is basically a string of chars. So I lean
> towards
> > > > > having a
> > > > > > > >> good
> > > > > > > >> > recommendation in the KIP and in the document of the field
> > in
> > > > the
> > > > > > > RPC's
> > > > > > > >> > definition.
> > > > > > > >> >
> > > > > > > >> > Best,
> > > > > > > >> > David
> > > > > > > >> >
> > > > > > > >> > On Fri, Aug 30, 2024 at 3:02 PM TengYao Chi <
> > > > kitingiao@gmail.com>
> > > > > > > >> wrote:
> > > > > > > >> >
> > > > > > > >> > > Hello Kirk !
> > > > > > > >> > >
> > > > > > > >> > > Thank you for your comments !
> > > > > > > >> > >
> > > > > > > >> > > KT1: Yes, you are correct. The issue is not unique to
> the
> > > > > initial
> > > > > > > >> > > heartbeat; there can always be cases where the broker
> > might
> > > > lose
> > > > > > > >> > connection
> > > > > > > >> > > with a member.
> > > > > > > >> > >
> > > > > > > >> > > KT2: Currently, if the client doesn't have a member ID
> and
> > > the
> > > > > > > >> > memberEpoch
> > > > > > > >> > > equals 0, the coordinator will generate a UUID as the
> > member
> > > > ID
> > > > > for
> > > > > > > >> the
> > > > > > > >> > > client. However, at the RPC level, the member ID is sent
> > as
> > > a
> > > > > > > literal
> > > > > > > >> > > string, meaning there are no restrictions on the format
> at
> > > > this
> > > > > > > level.
> > > > > > > >> > > This also reminds me that we haven't reached a final
> > > > conclusion
> > > > > on
> > > > > > > >> how to
> > > > > > > >> > > enforce the use of UUIDs.
> > > > > > > >> > > From our previous discussions, I recall two possible
> > > > approaches:
> > > > > > > >> > > The first is to validate the UUID on the server side,
> and
> > if
> > > > > it's
> > > > > > > not
> > > > > > > >> > > valid, throw an exception to the client.
> > > > > > > >> > > The second is to document a recommendation for clients
> to
> > > use
> > > > > UUIDs
> > > > > > > as
> > > > > > > >> > > member IDs, without strictly enforcing it.
> > > > > > > >> > > I think it's time to decide on the approach we want to
> > take.
> > > > > > > >> > >
> > > > > > > >> > > KT3: Yes, "session" can be considered synonymous with
> > > > > "membership"
> > > > > > > in
> > > > > > > >> > this
> > > > > > > >> > > context.
> > > > > > > >> > >
> > > > > > > >> > > KT4: Thank you for pointing that out. I will update the
> > > > wording
> > > > > to
> > > > > > > >> > > specifically say this behavior is for consumers.
> > > > > > > >> > >
> > > > > > > >> > > Thanks again for your comments.
> > > > > > > >> > >
> > > > > > > >> > > Best regards,
> > > > > > > >> > > TengYao
> > > > > > > >> > >
> > > > > > > >> > > Kirk True <ki...@kirktrue.pro> 於 2024年8月30日 週五
> 上午12:39寫道：
> > > > > > > >> > >
> > > > > > > >> > > > Hi TengYao!
> > > > > > > >> > > >
> > > > > > > >> > > > Sorry for being late to the discussion...
> > > > > > > >> > > >
> > > > > > > >> > > > After reading the thread and then the KIP, I had a few
> > > > > > > >> > > questions/comments:
> > > > > > > >> > > >
> > > > > > > >> > > > KT1: In Motivation, it states: "This scenario can
> result
> > > in
> > > > > the
> > > > > > > >> broker
> > > > > > > >> > > > registering a new member for which it will never
> > receive a
> > > > > proper
> > > > > > > >> leave
> > > > > > > >> > > > request.” Just to be clear, the broker will always
> have
> > > > cases
> > > > > > > where
> > > > > > > >> it
> > > > > > > >> > > > might lose connection with a member. That’s not unique
> > to
> > > > the
> > > > > > > >> initial
> > > > > > > >> > > > heartbeat, right?
> > > > > > > >> > > >
> > > > > > > >> > > > KT2: There was a bit of back and forth about format of
> > the
> > > > > member
> > > > > > > >> ID.
> > > > > > > >> > > From
> > > > > > > >> > > > what I gathered in the thread, the member ID is still
> > > > defined
> > > > > in
> > > > > > > the
> > > > > > > >> > RPC
> > > > > > > >> > > as
> > > > > > > >> > > > a string and not a UUID, right? The KIP states that
> the
> > > > > “client
> > > > > > > must
> > > > > > > >> > > > generate a UUID as the member ID” and that the “server
> > > will
> > > > > > > validate
> > > > > > > >> > > that a
> > > > > > > >> > > > valid UUID is provided.” Is that a change for the
> > server,
> > > or
> > > > > is it
> > > > > > > >> > > already
> > > > > > > >> > > > enforced as a UUID?
> > > > > > > >> > > >
> > > > > > > >> > > > KT3: Lianet mentioned some confusion over the use of
> the
> > > > word
> > > > > > > >> > “session.”
> > > > > > > >> > > > Isn’t “session” synonymous with “membership?”
> > > > > > > >> > > >
> > > > > > > >> > > > KT4: Under “Member ID Lifecycle,” it states: "The
> client
> > > > > should
> > > > > > > >> reuse
> > > > > > > >> > the
> > > > > > > >> > > > same UUID as the member ID for all heartbeats and
> rejoin
> > > > > attempts
> > > > > > > to
> > > > > > > >> > > > maintain continuity within the group.” Could we change
> > the
> > > > > first
> > > > > > > >> part
> > > > > > > >> > of
> > > > > > > >> > > > that to “The Consumer instance should…” We do have
> > > lifetimes
> > > > > that
> > > > > > > >> > extend
> > > > > > > >> > > > past the lifetime of a client instance (such as the
> > > > > transaction
> > > > > > > ID).
> > > > > > > >> > > >
> > > > > > > >> > > > Thanks,
> > > > > > > >> > > > Kirk
> > > > > > > >> > > >
> > > > > > > >> > > > > On Aug 29, 2024, at 1:28 AM, TengYao Chi <
> > > > > kitingiao@gmail.com>
> > > > > > > >> > wrote:
> > > > > > > >> > > > >
> > > > > > > >> > > > > Hi David,
> > > > > > > >> > > > >
> > > > > > > >> > > > > Thank you for pointing that out.
> > > > > > > >> > > > > I have updated the content of the KIP based on
> > Lianet's
> > > > and
> > > > > your
> > > > > > > >> > > > feedback.
> > > > > > > >> > > > > Please take a look and let me know your thoughts.
> > > > > > > >> > > > >
> > > > > > > >> > > > > Best regards,
> > > > > > > >> > > > > TengYao
> > > > > > > >> > > > >
> > > > > > > >> > > > > David Jacot <dj...@confluent.io.invalid> 於
> > 2024年8月29日
> > > 週四
> > > > > > > >> 下午3:20寫道：
> > > > > > > >> > > > >
> > > > > > > >> > > > >> Hi TengYao,
> > > > > > > >> > > > >>
> > > > > > > >> > > > >> Thanks for the update. I haven't fully read it yet
> > but
> > > I
> > > > > will
> > > > > > > >> soon.
> > > > > > > >> > > > >>
> > > > > > > >> > > > >> LM4: This is incorrect. The consumer must keep its
> > > member
> > > > > id
> > > > > > > >> during
> > > > > > > >> > > its
> > > > > > > >> > > > >> entire lifetime (until the process stops or dies).
> > The
> > > > > protocol
> > > > > > > >> > > > stipulates
> > > > > > > >> > > > >> that a member must rejoin with the same member id
> and
> > > the
> > > > > > > member
> > > > > > > >> > epoch
> > > > > > > >> > > > set
> > > > > > > >> > > > >> to zero when an FENCED_MEMBER_EPOCH occurs. This
> > allows
> > > > the
> > > > > > > >> member
> > > > > > > >> > to
> > > > > > > >> > > > >> resynchronize itself. We should not change this
> > > > behavior. I
> > > > > > > think
> > > > > > > >> > that
> > > > > > > >> > > > we
> > > > > > > >> > > > >> should see the client side generation id as an
> > > > incarnation
> > > > > id
> > > > > > > of
> > > > > > > >> the
> > > > > > > >> > > > >> application. It is generated once and kept until it
> > > stops
> > > > > or
> > > > > > > >> dies.
> > > > > > > >> > > > >>
> > > > > > > >> > > > >> Best,
> > > > > > > >> > > > >> David
> > > > > > > >> > > > >>
> > > > > > > >> > > > >> On Thu, Aug 29, 2024 at 6:21 AM TengYao Chi <
> > > > > > > kitingiao@gmail.com
> > > > > > > >> >
> > > > > > > >> > > > wrote:
> > > > > > > >> > > > >>
> > > > > > > >> > > > >>> Hello Lianet !
> > > > > > > >> > > > >>>
> > > > > > > >> > > > >>> Thanks for the reviews and suggestions!
> > > > > > > >> > > > >>>
> > > > > > > >> > > > >>> LM1: Indeed, we plan to enforce client-side ID
> > > > generation
> > > > > in
> > > > > > > the
> > > > > > > >> > > > future,
> > > > > > > >> > > > >>> and it is not an alternative. I will change the
> > title
> > > > > > > >> accordingly.
> > > > > > > >> > > > >>>
> > > > > > > >> > > > >>> LM2: Yes, that's the expectation. I will add that
> > > > > statement to
> > > > > > > >> the
> > > > > > > >> > > > public
> > > > > > > >> > > > >>> interface section.
> > > > > > > >> > > > >>>
> > > > > > > >> > > > >>> LM3: Thank you for the high-level perspective
> > review.
> > > I
> > > > > think
> > > > > > > >> > you're
> > > > > > > >> > > > >> right;
> > > > > > > >> > > > >>> our intention isn't very clear since it was placed
> > at
> > > > the
> > > > > end
> > > > > > > of
> > > > > > > >> > the
> > > > > > > >> > > > >>> section. I will try to rephrase that section to
> make
> > > it
> > > > > more
> > > > > > > >> > obvious.
> > > > > > > >> > > > >>>
> > > > > > > >> > > > >>> LM4: Regarding the definition of "session" in this
> > > KIP,
> > > > I
> > > > > > > >> believe
> > > > > > > >> > it
> > > > > > > >> > > > >> refers
> > > > > > > >> > > > >>> to the period between the *first-time heartbeat*
> and
> > > > when
> > > > > the
> > > > > > > >> > > *consumer
> > > > > > > >> > > > >>> leaves the group* (whether through a graceful
> > shutdown
> > > > or
> > > > > a
> > > > > > > >> > heartbeat
> > > > > > > >> > > > >>> timeout). The consumer should reuse its UUID if it
> > has
> > > > > been
> > > > > > > >> > generated
> > > > > > > >> > > > >>> before. The only situation in which it will
> > regenerate
> > > > the
> > > > > > > UUID
> > > > > > > >> is
> > > > > > > >> > if
> > > > > > > >> > > > the
> > > > > > > >> > > > >>> coordinator finds that there is already a consumer
> > > with
> > > > > the
> > > > > > > same
> > > > > > > >> > > UUID.
> > > > > > > >> > > > >>> IIRC, the coordinator should compare the member
> > > epochs,
> > > > > and
> > > > > > > the
> > > > > > > >> > > > >>> later-joined consumer should be fenced off by the
> > > > > coordinator
> > > > > > > >> due
> > > > > > > >> > to
> > > > > > > >> > > > >> having
> > > > > > > >> > > > >>> a lower member epoch. Once the consumer receives a
> > > > > > > >> > > > `FENCED_MEMBER_EPOCH`
> > > > > > > >> > > > >>> error, it will generate a new UUID and attempt to
> > > > rejoin.
> > > > > I
> > > > > > > will
> > > > > > > >> > > > clarify
> > > > > > > >> > > > >>> this in the KIP.
> > > > > > > >> > > > >>>
> > > > > > > >> > > > >>> Thanks again for your reviews, I really appreciate
> > it.
> > > > > > > >> > > > >>>
> > > > > > > >> > > > >>> Best regards,
> > > > > > > >> > > > >>> TengYao
> > > > > > > >> > > > >>>
> > > > > > > >> > > > >>> Lianet M. <li...@gmail.com> 於 2024年8月28日 週三
> > > > 下午7:12寫道：
> > > > > > > >> > > > >>>
> > > > > > > >> > > > >>>> Hello TengYao! Thanks for taking on this issue,
> > we've
> > > > > been
> > > > > > > >> going
> > > > > > > >> > > > around
> > > > > > > >> > > > >>> it
> > > > > > > >> > > > >>>> for a while.
> > > > > > > >> > > > >>>>
> > > > > > > >> > > > >>>> LM1: About the title of the KIP: "Enable ID
> > > Generation
> > > > > for
> > > > > > > >> Clients
> > > > > > > >> > > > over
> > > > > > > >> > > > >>> the
> > > > > > > >> > > > >>>> ConsumerGroupHeartbeat RPC". I find it confusing
> > > > because
> > > > > it
> > > > > > > >> hints
> > > > > > > >> > > that
> > > > > > > >> > > > >>>> we're adding it as an alternative (which was
> > > discussed
> > > > > and
> > > > > > > >> > > discarded,
> > > > > > > >> > > > >> in
> > > > > > > >> > > > >>>> favour of really enforcing it). It's also missing
> > the
> > > > > core
> > > > > > > >> change
> > > > > > > >> > > imo,
> > > > > > > >> > > > >>>> which is "where" the generation happens. So,
> maybe
> > > more
> > > > > to
> > > > > > > the
> > > > > > > >> > point
> > > > > > > >> > > > >> with
> > > > > > > >> > > > >>>> something along the lines of "Client-side
> generated
> > > ID
> > > > > for
> > > > > > > >> clients
> > > > > > > >> > > > over
> > > > > > > >> > > > >>>> ConsumerGroupHeartbeat RPC"?
> > > > > > > >> > > > >>>>
> > > > > > > >> > > > >>>> LM2: On the public interfaces section, the KIP
> > states
> > > > > that
> > > > > > > "the
> > > > > > > >> > > server
> > > > > > > >> > > > >>> will
> > > > > > > >> > > > >>>> reject the request", but we should agree on the
> > > > specific
> > > > > > > error
> > > > > > > >> > > type. I
> > > > > > > >> > > > >>>> expect it should fail with an
> > > InvalidRequestException,
> > > > is
> > > > > > > that
> > > > > > > >> the
> > > > > > > >> > > > >>>> intention? (This was also suggested in the
> > discussion
> > > > > thread
> > > > > > > >> > before
> > > > > > > >> > > > but
> > > > > > > >> > > > >>> is
> > > > > > > >> > > > >>>> not in the KIP).
> > > > > > > >> > > > >>>>
> > > > > > > >> > > > >>>> LM3. Related to my previous point, I find that to
> > be
> > > > the
> > > > > true
> > > > > > > >> > > > >>> public-facing
> > > > > > > >> > > > >>>> change (member ID mandatory at the protocol
> level),
> > > but
> > > > > it's
> > > > > > > >> only
> > > > > > > >> > at
> > > > > > > >> > > > >> the
> > > > > > > >> > > > >>>> end of the Public interfaces changes, kind of
> lost
> > > > among
> > > > > > > >> details
> > > > > > > >> > of
> > > > > > > >> > > > how
> > > > > > > >> > > > >>>> we're going to do it. Should we rephrase that
> > section
> > > > > with
> > > > > > > the
> > > > > > > >> > > actual
> > > > > > > >> > > > >>>> change first, and the hows after (ex. Bumping the
> > > > > version is
> > > > > > > >> not
> > > > > > > >> > the
> > > > > > > >> > > > >>>> public-facing change in this case, it's just the
> > > > > mechanism to
> > > > > > > >> > > properly
> > > > > > > >> > > > >>>> introduce our change)
> > > > > > > >> > > > >>>>
> > > > > > > >> > > > >>>> LM4. Regarding the lifetime of the UUID: the KIP
> > > states
> > > > > we
> > > > > > > will
> > > > > > > >> > > > "Verify
> > > > > > > >> > > > >>>> that the UUID remains consistent across all
> > > subsequent
> > > > > > > >> heartbeats
> > > > > > > >> > > > >> during
> > > > > > > >> > > > >>>> the session". What is this "session" referring to
> > > > here? I
> > > > > > > would
> > > > > > > >> > > expect
> > > > > > > >> > > > >>> that
> > > > > > > >> > > > >>>> the UUID is associated to a consumer instance
> > > > (generated
> > > > > for
> > > > > > > >> the
> > > > > > > >> > > > >> consumer
> > > > > > > >> > > > >>>> the first time it needs to send a HB if it
> doesn't
> > > have
> > > > > the
> > > > > > > >> UUID
> > > > > > > >> > > yet.
> > > > > > > >> > > > >>> From
> > > > > > > >> > > > >>>> there on, every time it needs to send a "first
> HB"
> > > > > again, it
> > > > > > > >> will
> > > > > > > >> > > > reuse
> > > > > > > >> > > > >>> its
> > > > > > > >> > > > >>>> UUID, is that the intention? Note that we should
> > > > consider
> > > > > > > that
> > > > > > > >> the
> > > > > > > >> > > > same
> > > > > > > >> > > > >>>> consumer instance may have many "first
> heartbeats",
> > > > > meaning
> > > > > > > >> > > heartbeats
> > > > > > > >> > > > >> to
> > > > > > > >> > > > >>>> join the group when it's not part of it (ex.
> > consumer
> > > > > > > >> unsubscribe
> > > > > > > >> > +
> > > > > > > >> > > > >>>> subscribe, fenced, stale). Is this the intention
> or
> > > are
> > > > > you
> > > > > > > >> > > > considering
> > > > > > > >> > > > >>> the
> > > > > > > >> > > > >>>> lifetime differently? We should clarify it in the
> > > KIP.
> > > > > > > >> > > > >>>>
> > > > > > > >> > > > >>>> Thanks!
> > > > > > > >> > > > >>>>
> > > > > > > >> > > > >>>> Lianet
> > > > > > > >> > > > >>>>
> > > > > > > >> > > > >>>> On Tue, Aug 27, 2024 at 2:27 AM TengYao Chi <
> > > > > > > >> kitingiao@gmail.com>
> > > > > > > >> > > > >> wrote:
> > > > > > > >> > > > >>>>
> > > > > > > >> > > > >>>>> Hi everyone,
> > > > > > > >> > > > >>>>>
> > > > > > > >> > > > >>>>> I have revised this KIP multiple times based on
> > the
> > > > > feedback
> > > > > > > >> from
> > > > > > > >> > > our
> > > > > > > >> > > > >>>>> discussions.
> > > > > > > >> > > > >>>>> I would greatly appreciate it if you could
> review
> > it
> > > > > when
> > > > > > > you
> > > > > > > >> > have
> > > > > > > >> > > > >> the
> > > > > > > >> > > > >>>>> time.
> > > > > > > >> > > > >>>>> If there are no further comments or
> suggestions, I
> > > > plan
> > > > > to
> > > > > > > >> > proceed
> > > > > > > >> > > > >> with
> > > > > > > >> > > > >>>>> initiating a vote soon.
> > > > > > > >> > > > >>>>>
> > > > > > > >> > > > >>>>> Best regards,
> > > > > > > >> > > > >>>>> TengYao
> > > > > > > >> > > > >>>>>
> > > > > > > >> > > > >>>>> TengYao Chi <ki...@gmail.com> 於 2024年8月23日
> 週五
> > > > > 下午2:43寫道：
> > > > > > > >> > > > >>>>>
> > > > > > > >> > > > >>>>>> Hi Andrew,
> > > > > > > >> > > > >>>>>> Thank you for your previous feedback and
> > insights.
> > > > > > > >> > > > >>>>>> Your contributions have added valuable
> > perspectives
> > > > to
> > > > > the
> > > > > > > >> > > > >>> discussions.
> > > > > > > >> > > > >>>>>> And we also benefit from the comparison of
> > > different
> > > > > > > >> solutions.
> > > > > > > >> > > > >>>>>> I’m also looking forward to seeing an initial
> > > version
> > > > > in
> > > > > > > >> > KIP-932,
> > > > > > > >> > > > >> as
> > > > > > > >> > > > >>> it
> > > > > > > >> > > > >>>>>> will provide a good reference for future
> > > > > implementations.
> > > > > > > >> > > > >>>>>>
> > > > > > > >> > > > >>>>>> Regarding your comment on AS2, I wanted to
> > clarify
> > > > > that my
> > > > > > > >> > > > >>>> specification
> > > > > > > >> > > > >>>>>> references org.apache.kafka.common.Uuid.
> > > > > > > >> > > > >>>>>> I believe we’re referring to the same class,
> and
> > it
> > > > > might
> > > > > > > >> just
> > > > > > > >> > be
> > > > > > > >> > > a
> > > > > > > >> > > > >>>> small
> > > > > > > >> > > > >>>>>> oversight due to the busy schedule.
> > > > > > > >> > > > >>>>>>
> > > > > > > >> > > > >>>>>> I want to express my gratitude once again for
> > your
> > > > many
> > > > > > > >> > insightful
> > > > > > > >> > > > >>>>>> comments, which have helped the discussion
> > progress
> > > > > > > smoothly.
> > > > > > > >> > > > >>>>>>
> > > > > > > >> > > > >>>>>> Best regards,
> > > > > > > >> > > > >>>>>> TengYao
> > > > > > > >> > > > >>>>>>
> > > > > > > >> > > > >>>>>>
> > > > > > > >> > > > >>>>>> Andrew Schofield <an...@live.com> 於
> > > > > 2024年8月22日
> > > > > > > 週四
> > > > > > > >> > > > >>>> 下午11:28寫道：
> > > > > > > >> > > > >>>>>>
> > > > > > > >> > > > >>>>>>> Hi TengYao,
> > > > > > > >> > > > >>>>>>> I’ve been reading through the comments and I’m
> > > happy
> > > > > that
> > > > > > > >> the
> > > > > > > >> > > > >> lobby
> > > > > > > >> > > > >>>>>>> approach has not gained support.
> > > > > > > >> > > > >>>>>>>
> > > > > > > >> > > > >>>>>>> Assuming that this KIP is voted, I will be
> happy
> > > to
> > > > > change
> > > > > > > >> > > KIP-932
> > > > > > > >> > > > >>> so
> > > > > > > >> > > > >>>>>>> that it only supports client-side member ID
> > > > > generation.
> > > > > > > >> Because
> > > > > > > >> > > > >> that
> > > > > > > >> > > > >>>> KIP
> > > > > > > >> > > > >>>>>>> is still
> > > > > > > >> > > > >>>>>>> under development, I can do this in the first
> > > > version
> > > > > of
> > > > > > > >> > > > >>>>>>> ShareGroupHeartbeat.
> > > > > > > >> > > > >>>>>>>
> > > > > > > >> > > > >>>>>>> AS2: For the encoding section, I suppose the
> > > > specific
> > > > > > > >> encoding
> > > > > > > >> > > > >> which
> > > > > > > >> > > > >>>>>>> is used is what org.apache.kafka.utils.Uuid
> > uses.
> > > > > > > >> > > > >>>>>>>
> > > > > > > >> > > > >>>>>>> Thanks,
> > > > > > > >> > > > >>>>>>> Andrew
> > > > > > > >> > > > >>>>>>>
> > > > > > > >> > > > >>>>>>>> On 14 Aug 2024, at 17:00, TengYao Chi <
> > > > > > > kitingiao@gmail.com
> > > > > > > >> >
> > > > > > > >> > > > >>> wrote:
> > > > > > > >> > > > >>>>>>>>
> > > > > > > >> > > > >>>>>>>> Hello Apoorv,
> > > > > > > >> > > > >>>>>>>> Thank you for your feedback.
> > > > > > > >> > > > >>>>>>>> Regarding the questions you raised,
> > > unfortunately,
> > > > > this
> > > > > > > KIP
> > > > > > > >> > > > >> cannot
> > > > > > > >> > > > >>>>>>>> guarantee the order of heartbeats. As with
> many
> > > > > classic
> > > > > > > >> > > > >>> distributed
> > > > > > > >> > > > >>>>>>> system
> > > > > > > >> > > > >>>>>>>> challenges, what we can do is make our best
> > > effort
> > > > to
> > > > > > > >> ensure
> > > > > > > >> > > > >> that
> > > > > > > >> > > > >>>>> there
> > > > > > > >> > > > >>>>>>> are
> > > > > > > >> > > > >>>>>>>> no idle members or stale assignments under
> > normal
> > > > > > > >> > circumstances.
> > > > > > > >> > > > >>>>>>>>
> > > > > > > >> > > > >>>>>>>> As for the lobby approach, I’m not a fan of
> it
> > > > > because it
> > > > > > > >> > > > >> requires
> > > > > > > >> > > > >>>>>>> adding a
> > > > > > > >> > > > >>>>>>>> mechanism to maintain client state within the
> > > > > > > >> ConsumerGroup,
> > > > > > > >> > > > >>> which,
> > > > > > > >> > > > >>>> in
> > > > > > > >> > > > >>>>>>> my
> > > > > > > >> > > > >>>>>>>> view, resembles something like a two-phase
> > > commit.
> > > > > This
> > > > > > > >> would
> > > > > > > >> > > > >>>>> introduce
> > > > > > > >> > > > >>>>>>>> more complexity than the proposal in this
> KIP,
> > > > which
> > > > > is
> > > > > > > >> > > > >> something
> > > > > > > >> > > > >>> we
> > > > > > > >> > > > >>>>>>> want
> > > > > > > >> > > > >>>>>>>> to avoid. KIP-848 aims to simplify the
> existing
> > > > > protocol,
> > > > > > > >> and
> > > > > > > >> > > > >>> while
> > > > > > > >> > > > >>>>> the
> > > > > > > >> > > > >>>>>>>> lobby approach is a good one, I believe it is
> > not
> > > > the
> > > > > > > right
> > > > > > > >> > fit
> > > > > > > >> > > > >>> for
> > > > > > > >> > > > >>>>> this
> > > > > > > >> > > > >>>>>>>> particular situation.
> > > > > > > >> > > > >>>>>>>>
> > > > > > > >> > > > >>>>>>>> Best regards,
> > > > > > > >> > > > >>>>>>>> TengYao
> > > > > > > >> > > > >>>>>>>>
> > > > > > > >> > > > >>>>>>>> TengYao Chi <ki...@gmail.com> 於
> 2024年8月14日
> > > 週三
> > > > > > > >> 下午11:45寫道：
> > > > > > > >> > > > >>>>>>>>
> > > > > > > >> > > > >>>>>>>>> Hi David,
> > > > > > > >> > > > >>>>>>>>>
> > > > > > > >> > > > >>>>>>>>> I really appreciate your review and
> > suggestions.
> > > > As
> > > > > I am
> > > > > > > >> > still
> > > > > > > >> > > > >>>>> gaining
> > > > > > > >> > > > >>>>>>>>> experience in writing KIPs, your input has
> > been
> > > > > > > incredibly
> > > > > > > >> > > > >>>> helpful. I
> > > > > > > >> > > > >>>>>>> am
> > > > > > > >> > > > >>>>>>>>> currently applying your suggestions to the
> KIP
> > > and
> > > > > will
> > > > > > > >> > > > >> complete
> > > > > > > >> > > > >>> it
> > > > > > > >> > > > >>>>> as
> > > > > > > >> > > > >>>>>>> soon
> > > > > > > >> > > > >>>>>>>>> as possible.
> > > > > > > >> > > > >>>>>>>>> Regarding the UUID part, I think we haven’t
> > > > reached
> > > > > a
> > > > > > > >> > > > >> conclusion
> > > > > > > >> > > > >>>>>>> yet.(So
> > > > > > > >> > > > >>>>>>>>> far according to this thread)
> > > > > > > >> > > > >>>>>>>>> However, I will review the current
> > > implementation
> > > > > in the
> > > > > > > >> > Kafka
> > > > > > > >> > > > >>>> `Uuid`
> > > > > > > >> > > > >>>>>>>>> class and include a brief specification in
> the
> > > > KIP.
> > > > > > > >> > > > >>>>>>>>>
> > > > > > > >> > > > >>>>>>>>> Once again, thank you so much for your help.
> > > > > > > >> > > > >>>>>>>>>
> > > > > > > >> > > > >>>>>>>>> Best regards,
> > > > > > > >> > > > >>>>>>>>> TengYao
> > > > > > > >> > > > >>>>>>>>>
> > > > > > > >> > > > >>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> > > 2024年8月14日
> > > > 週三
> > > > > > > >> > 下午11:14寫道：
> > > > > > > >> > > > >>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>> hi Apoorv
> > > > > > > >> > > > >>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>> As the memberId is now known to the
> client,
> > > and
> > > > > client
> > > > > > > >> > might
> > > > > > > >> > > > >>> send
> > > > > > > >> > > > >>>>> the
> > > > > > > >> > > > >>>>>>>>>> leave
> > > > > > > >> > > > >>>>>>>>>> group heartbeat on shutdown prior to
> > receiving
> > > > the
> > > > > > > >> initial
> > > > > > > >> > > > >>>> heartbeat
> > > > > > > >> > > > >>>>>>>>>> response. If that's true then how do we
> > > guarantee
> > > > > that
> > > > > > > >> the 2
> > > > > > > >> > > > >>>>> requests
> > > > > > > >> > > > >>>>>>> to
> > > > > > > >> > > > >>>>>>>>>> join and leave will be processed in order,
> > > which
> > > > > could
> > > > > > > >> still
> > > > > > > >> > > > >>> leave
> > > > > > > >> > > > >>>>>>> stale
> > > > > > > >> > > > >>>>>>>>>> members or throw unknown member id
> > exceptions?
> > > > > > > >> > > > >>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>> This is definitely a good question. the
> short
> > > > > answer:
> > > > > > > no
> > > > > > > >> > > > >>> guarantee
> > > > > > > >> > > > >>>>> but
> > > > > > > >> > > > >>>>>>>>>> best
> > > > > > > >> > > > >>>>>>>>>> efforts
> > > > > > > >> > > > >>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>> Please notice the root cause is "we have no
> > > > enough
> > > > > time
> > > > > > > >> to
> > > > > > > >> > > > >> wait
> > > > > > > >> > > > >>>>>>> member id
> > > > > > > >> > > > >>>>>>>>>> (response) when closing consumer". Sadly,
> we
> > > can'
> > > > > > > >> guarantee
> > > > > > > >> > > > >> the
> > > > > > > >> > > > >>>>>>> request
> > > > > > > >> > > > >>>>>>>>>> order due to the same reason.
> > > > > > > >> > > > >>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>> However, in contrast to previous behavior,
> > > there
> > > > > is one
> > > > > > > >> big
> > > > > > > >> > > > >>>> benefit
> > > > > > > >> > > > >>>>>>> of new
> > > > > > > >> > > > >>>>>>>>>> approach - we can try STONITH because we
> know
> > > the
> > > > > > > member
> > > > > > > >> id
> > > > > > > >> > > > >>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>> Best,
> > > > > > > >> > > > >>>>>>>>>> Chia-Ping
> > > > > > > >> > > > >>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>> Apoorv Mittal <ap...@gmail.com> 於
> > > > > 2024年8月14日
> > > > > > > 週三
> > > > > > > >> > > > >>>> 下午8:55寫道：
> > > > > > > >> > > > >>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>> Hi TengYao,
> > > > > > > >> > > > >>>>>>>>>>> Thanks for the KIP. Continuing on the
> point
> > > > which
> > > > > > > Andrew
> > > > > > > >> > > > >>>> mentioned
> > > > > > > >> > > > >>>>> as
> > > > > > > >> > > > >>>>>>>>>> AS1.
> > > > > > > >> > > > >>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>> As the memberId is now known to the
> client,
> > > and
> > > > > client
> > > > > > > >> > might
> > > > > > > >> > > > >>> send
> > > > > > > >> > > > >>>>> the
> > > > > > > >> > > > >>>>>>>>>> leave
> > > > > > > >> > > > >>>>>>>>>>> group heartbeat on shutdown prior to
> > receiving
> > > > the
> > > > > > > >> initial
> > > > > > > >> > > > >>>>> heartbeat
> > > > > > > >> > > > >>>>>>>>>>> response. If that's true then how do we
> > > > guarantee
> > > > > that
> > > > > > > >> the
> > > > > > > >> > 2
> > > > > > > >> > > > >>>>>>> requests to
> > > > > > > >> > > > >>>>>>>>>>> join and leave will be processed in order,
> > > which
> > > > > could
> > > > > > > >> > still
> > > > > > > >> > > > >>>> leave
> > > > > > > >> > > > >>>>>>> stale
> > > > > > > >> > > > >>>>>>>>>>> members or throw unknown member id
> > exceptions?
> > > > > > > >> > > > >>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>> Though the client side member id
> generation
> > is
> > > > > helpful
> > > > > > > >> > which
> > > > > > > >> > > > >>> will
> > > > > > > >> > > > >>>>>>>>>> represent
> > > > > > > >> > > > >>>>>>>>>>> the same group perspective as from client
> > and
> > > > > broker's
> > > > > > > >> end.
> > > > > > > >> > > > >>> But I
> > > > > > > >> > > > >>>>>>> think
> > > > > > > >> > > > >>>>>>>>>> the
> > > > > > > >> > > > >>>>>>>>>>> major concern we want to solve here is
> Stale
> > > > > Partition
> > > > > > > >> > > > >>>> Assignments
> > > > > > > >> > > > >>>>>>> which
> > > > > > > >> > > > >>>>>>>>>>> might still exist with the new approach. I
> > am
> > > > > leaning
> > > > > > > >> > towards
> > > > > > > >> > > > >>> the
> > > > > > > >> > > > >>>>>>>>>>> suggestion mentioned by Andrew where
> > partition
> > > > > > > >> assignment
> > > > > > > >> > > > >>>> triggers
> > > > > > > >> > > > >>>>> on
> > > > > > > >> > > > >>>>>>>>>>> subsequent heartbeat when client
> > acknowledges
> > > > the
> > > > > > > >> initial
> > > > > > > >> > > > >>>>> heartbeat,
> > > > > > > >> > > > >>>>>>>>>>> delayed partition assignment.
> > > > > > > >> > > > >>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>> Though on a separate note, I have a
> > different
> > > > > > > question.
> > > > > > > >> > What
> > > > > > > >> > > > >>>>> happens
> > > > > > > >> > > > >>>>>>>>>> when
> > > > > > > >> > > > >>>>>>>>>>> there is an issue with the client which
> > sends
> > > > the
> > > > > > > >> initial
> > > > > > > >> > > > >>>> heartbeat
> > > > > > > >> > > > >>>>>>>>>> without
> > > > > > > >> > > > >>>>>>>>>>> memberId, then crashes and restarts? I
> think
> > > we
> > > > > must
> > > > > > > be
> > > > > > > >> > > > >>>>> re-triggering
> > > > > > > >> > > > >>>>>>>>>>> assignments and expiring members only
> after
> > > the
> > > > > > > >> heartbeat
> > > > > > > >> > > > >>> session
> > > > > > > >> > > > >>>>>>>>>> timeout?
> > > > > > > >> > > > >>>>>>>>>>> If that's true then shall delayed
> partition
> > > > > assignment
> > > > > > > >> can
> > > > > > > >> > > > >> help
> > > > > > > >> > > > >>>>>>> benefit
> > > > > > > >> > > > >>>>>>>>>> us
> > > > > > > >> > > > >>>>>>>>>>> from this situation as well?
> > > > > > > >> > > > >>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>> Regards,
> > > > > > > >> > > > >>>>>>>>>>> Apoorv Mittal
> > > > > > > >> > > > >>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>> On Wed, Aug 14, 2024 at 12:51 PM David
> Jacot
> > > > > > > >> > > > >>>>>>>>>> <dj...@confluent.io.invalid>
> > > > > > > >> > > > >>>>>>>>>>> wrote:
> > > > > > > >> > > > >>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>> Hi Andrew,
> > > > > > > >> > > > >>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>> Personally, I don't like the lobby
> > approach.
> > > It
> > > > > makes
> > > > > > > >> > things
> > > > > > > >> > > > >>>> more
> > > > > > > >> > > > >>>>>>>>>>>> complicated and it would require changing
> > the
> > > > > records
> > > > > > > >> on
> > > > > > > >> > the
> > > > > > > >> > > > >>>>> server
> > > > > > > >> > > > >>>>>>>>>> too.
> > > > > > > >> > > > >>>>>>>>>>>> This is why I initially suggested the
> > > rejected
> > > > > > > >> alternative
> > > > > > > >> > > > >> #2
> > > > > > > >> > > > >>>>> which
> > > > > > > >> > > > >>>>>>> is
> > > > > > > >> > > > >>>>>>>>>>>> pretty close but also not perfect.
> > > > > > > >> > > > >>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>> I'd like to clarify one thing. The
> > > > > > > >> ConsumerGroupHeartbeat
> > > > > > > >> > > > >> API
> > > > > > > >> > > > >>>>>>> already
> > > > > > > >> > > > >>>>>>>>>>>> supports generating the member id on the
> > > client
> > > > > so we
> > > > > > > >> > don't
> > > > > > > >> > > > >>> need
> > > > > > > >> > > > >>>>> any
> > > > > > > >> > > > >>>>>>>>>>>> conditional logic on the client side.
> This
> > is
> > > > > > > actually
> > > > > > > >> > what
> > > > > > > >> > > > >> we
> > > > > > > >> > > > >>>>>>> wanted
> > > > > > > >> > > > >>>>>>>>>> to
> > > > > > > >> > > > >>>>>>>>>>> do
> > > > > > > >> > > > >>>>>>>>>>>> in the first place but the idea got
> pushed
> > > back
> > > > > by
> > > > > > > >> Magnus
> > > > > > > >> > > > >> back
> > > > > > > >> > > > >>>>> then
> > > > > > > >> > > > >>>>>>>>>>> because
> > > > > > > >> > > > >>>>>>>>>>>> generating uuid from librdkafka required
> a
> > > new
> > > > > > > >> dependency.
> > > > > > > >> > > > >> It
> > > > > > > >> > > > >>>>> turns
> > > > > > > >> > > > >>>>>>>>>> out
> > > > > > > >> > > > >>>>>>>>>>>> that librdkafka has that dependency
> today.
> > In
> > > > > > > >> retrospect,
> > > > > > > >> > we
> > > > > > > >> > > > >>>>> should
> > > > > > > >> > > > >>>>>>>>>> have
> > > > > > > >> > > > >>>>>>>>>>>> pushed back on this. Long story short, we
> > can
> > > > > just do
> > > > > > > >> it.
> > > > > > > >> > > > >> The
> > > > > > > >> > > > >>>>>>>>>> proposal in
> > > > > > > >> > > > >>>>>>>>>>>> this KIP is to make the member id
> required
> > in
> > > > > future
> > > > > > > >> > > > >> versions.
> > > > > > > >> > > > >>>> We
> > > > > > > >> > > > >>>>>>>>>> could
> > > > > > > >> > > > >>>>>>>>>>>> also decide not to do it and to keep
> > > supporting
> > > > > both
> > > > > > > >> > > > >>>> approaches. I
> > > > > > > >> > > > >>>>>>>>>> would
> > > > > > > >> > > > >>>>>>>>>>>> also be fine with this.
> > > > > > > >> > > > >>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>> Best,
> > > > > > > >> > > > >>>>>>>>>>>> David
> > > > > > > >> > > > >>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>> On Wed, Aug 14, 2024 at 12:30 PM Andrew
> > > > > Schofield <
> > > > > > > >> > > > >>>>>>>>>>>> andrew_schofield@live.com>
> > > > > > > >> > > > >>>>>>>>>>>> wrote:
> > > > > > > >> > > > >>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>> Hi TengYao,
> > > > > > > >> > > > >>>>>>>>>>>>> Thanks for your response. I’ll have just
> > one
> > > > > more
> > > > > > > try
> > > > > > > >> to
> > > > > > > >> > > > >>>>> persuade.
> > > > > > > >> > > > >>>>>>>>>>>>> I feel that I will need to follow the
> > > approach
> > > > > with
> > > > > > > >> > KIP-932
> > > > > > > >> > > > >>>> when
> > > > > > > >> > > > >>>>>>>>>> we’ve
> > > > > > > >> > > > >>>>>>>>>>>>> made a decision, so I do have more than
> a
> > > > > passing
> > > > > > > >> > interest
> > > > > > > >> > > > >> in
> > > > > > > >> > > > >>>>> this.
> > > > > > > >> > > > >>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>> A group member in the lobby is in the
> > group,
> > > > > but it
> > > > > > > >> does
> > > > > > > >> > > > >> not
> > > > > > > >> > > > >>>> have
> > > > > > > >> > > > >>>>>>>>>> any
> > > > > > > >> > > > >>>>>>>>>>>>> assignments. A member of a consumer
> group
> > > can
> > > > > have
> > > > > > > no
> > > > > > > >> > > > >>> assigned
> > > > > > > >> > > > >>>>>>>>>>>>> partitions (such as 5 CG members
> > subscribed
> > > > to a
> > > > > > > topic
> > > > > > > >> > > > >> with 4
> > > > > > > >> > > > >>>>>>>>>>>> partitions),
> > > > > > > >> > > > >>>>>>>>>>>>> so it’s a situation that consumer group
> > > > members
> > > > > > > >> already
> > > > > > > >> > > > >>> expect.
> > > > > > > >> > > > >>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>> One of Kafka’s strengths is the way that
> > we
> > > > > handle
> > > > > > > API
> > > > > > > >> > > > >>>>> versioning.
> > > > > > > >> > > > >>>>>>>>>>>>> But, there is a cost - the behaviour is
> > > > > different
> > > > > > > >> > depending
> > > > > > > >> > > > >>> on
> > > > > > > >> > > > >>>>> the
> > > > > > > >> > > > >>>>>>>>>> RPC
> > > > > > > >> > > > >>>>>>>>>>>>> version. KIP-848 is on the cusp of
> > > completion,
> > > > > but
> > > > > > > >> we’re
> > > > > > > >> > > > >>>> already
> > > > > > > >> > > > >>>>>>>>>> adding
> > > > > > > >> > > > >>>>>>>>>>>>> conditional logic for v0/v1 for
> > > > > > > >> ConsumerGroupHeartbeat.
> > > > > > > >> > > > >>> That’s
> > > > > > > >> > > > >>>> a
> > > > > > > >> > > > >>>>>>>>>> pity.
> > > > > > > >> > > > >>>>>>>>>>>>> Only a minor issue, but it’s
> unfortunate.
> > > > > > > >> > > > >>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>> Thanks,
> > > > > > > >> > > > >>>>>>>>>>>>> Andrew
> > > > > > > >> > > > >>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>> On 14 Aug 2024, at 08:47, TengYao Chi <
> > > > > > > >> > > > >> kitingiao@gmail.com>
> > > > > > > >> > > > >>>>>>>>>> wrote:
> > > > > > > >> > > > >>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>> Hello Andrew
> > > > > > > >> > > > >>>>>>>>>>>>>> Thank you for your thoughtful
> suggestions
> > > and
> > > > > > > getting
> > > > > > > >> > the
> > > > > > > >> > > > >>>>>>>>>> discussion
> > > > > > > >> > > > >>>>>>>>>>>>> going.
> > > > > > > >> > > > >>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>> To AS1:
> > > > > > > >> > > > >>>>>>>>>>>>>> In the current scenario where the
> server
> > > > > generates
> > > > > > > >> the
> > > > > > > >> > > > >> UUID,
> > > > > > > >> > > > >>>> if
> > > > > > > >> > > > >>>>>>>>>> the
> > > > > > > >> > > > >>>>>>>>>>>>> client
> > > > > > > >> > > > >>>>>>>>>>>>>> shuts down before receiving the
> memberId
> > > > > generated
> > > > > > > by
> > > > > > > >> > the
> > > > > > > >> > > > >> GC
> > > > > > > >> > > > >>>>>>>>>>>> (regardless
> > > > > > > >> > > > >>>>>>>>>>>>> of
> > > > > > > >> > > > >>>>>>>>>>>>>> whether it’s a graceful shutdown or
> not),
> > > the
> > > > > GC
> > > > > > > will
> > > > > > > >> > > > >> still
> > > > > > > >> > > > >>>> have
> > > > > > > >> > > > >>>>>>>>>> to
> > > > > > > >> > > > >>>>>>>>>>>> wait
> > > > > > > >> > > > >>>>>>>>>>>>>> for the heartbeat timeout because the
> > > client
> > > > > > > doesn’t
> > > > > > > >> > know
> > > > > > > >> > > > >>> its
> > > > > > > >> > > > >>>>>>>>>>> memberId.
> > > > > > > >> > > > >>>>>>>>>>>>>> This KIP indeed cannot completely
> resolve
> > > the
> > > > > > > >> > idempotency
> > > > > > > >> > > > >>>> issue,
> > > > > > > >> > > > >>>>>>>>>> but
> > > > > > > >> > > > >>>>>>>>>>> it
> > > > > > > >> > > > >>>>>>>>>>>>> can
> > > > > > > >> > > > >>>>>>>>>>>>>> better handle shutdown scenarios under
> > > normal
> > > > > > > >> > > > >> circumstances
> > > > > > > >> > > > >>>>>>>>>> because
> > > > > > > >> > > > >>>>>>>>>>> the
> > > > > > > >> > > > >>>>>>>>>>>>>> client always knows its memberId. Even
> if
> > > the
> > > > > > > client
> > > > > > > >> > shuts
> > > > > > > >> > > > >>>> down
> > > > > > > >> > > > >>>>>>>>>>>>> immediately
> > > > > > > >> > > > >>>>>>>>>>>>>> after the initial heartbeat, as long as
> > it
> > > > > > > performs a
> > > > > > > >> > > > >>> graceful
> > > > > > > >> > > > >>>>>>>>>>> shutdown
> > > > > > > >> > > > >>>>>>>>>>>>> and
> > > > > > > >> > > > >>>>>>>>>>>>>> sends a leave heartbeat, the GC can
> > manage
> > > > the
> > > > > > > >> situation
> > > > > > > >> > > > >> and
> > > > > > > >> > > > >>>>>>>>>> remove
> > > > > > > >> > > > >>>>>>>>>>> the
> > > > > > > >> > > > >>>>>>>>>>>>>> member. Therefore, the goal of this KIP
> > is
> > > to
> > > > > > > address
> > > > > > > >> > the
> > > > > > > >> > > > >>>> issue
> > > > > > > >> > > > >>>>>>>>>> where
> > > > > > > >> > > > >>>>>>>>>>>> the
> > > > > > > >> > > > >>>>>>>>>>>>>> GC has to wait for the heartbeat
> timeout
> > > due
> > > > > to the
> > > > > > > >> > client
> > > > > > > >> > > > >>>>> leaving
> > > > > > > >> > > > >>>>>>>>>>>>> without
> > > > > > > >> > > > >>>>>>>>>>>>>> knowing its memberId, which leads to
> > > reduced
> > > > > > > >> throughput
> > > > > > > >> > > > >> and
> > > > > > > >> > > > >>>>>>>>>> limited
> > > > > > > >> > > > >>>>>>>>>>>>>> scalability.
> > > > > > > >> > > > >>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>> The solution you suggest has also been
> > > > > proposed by
> > > > > > > >> > David.
> > > > > > > >> > > > >>> The
> > > > > > > >> > > > >>>>>>>>>> concern
> > > > > > > >> > > > >>>>>>>>>>>>> with
> > > > > > > >> > > > >>>>>>>>>>>>>> this approach is that it introduces
> > > > additional
> > > > > > > >> > complexity
> > > > > > > >> > > > >>> for
> > > > > > > >> > > > >>>>>>>>>>>>>> compatibility, as the new server would
> > not
> > > > > > > >> immediately
> > > > > > > >> > add
> > > > > > > >> > > > >>> the
> > > > > > > >> > > > >>>>>>>>>> member
> > > > > > > >> > > > >>>>>>>>>>>> to
> > > > > > > >> > > > >>>>>>>>>>>>>> the group, while the old server would.
> > This
> > > > > > > requires
> > > > > > > >> > > > >> clients
> > > > > > > >> > > > >>>> to
> > > > > > > >> > > > >>>>>>>>>>>>>> differentiate whether their memberId
> has
> > > been
> > > > > added
> > > > > > > >> to
> > > > > > > >> > the
> > > > > > > >> > > > >>>> group
> > > > > > > >> > > > >>>>>>>>>> or
> > > > > > > >> > > > >>>>>>>>>>>> not,
> > > > > > > >> > > > >>>>>>>>>>>>>> which could result in unexpected logs.
> > > > > > > >> > > > >>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>> Best Regards,
> > > > > > > >> > > > >>>>>>>>>>>>>> TengYao
> > > > > > > >> > > > >>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>> Andrew Schofield <
> > > andrew_schofield@live.com>
> > > > 於
> > > > > > > >> > 2024年8月14日
> > > > > > > >> > > > >>> 週三
> > > > > > > >> > > > >>>>>>>>>>>> 上午12:29寫道：
> > > > > > > >> > > > >>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>>> Hi TengYao,
> > > > > > > >> > > > >>>>>>>>>>>>>>> Thanks for the KIP. I wonder if
> there’s
> > a
> > > > > > > different
> > > > > > > >> way
> > > > > > > >> > > > >> to
> > > > > > > >> > > > >>>>> close
> > > > > > > >> > > > >>>>>>>>>>> what
> > > > > > > >> > > > >>>>>>>>>>>>>>> is quite a small window.
> > > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>>> AS1: It is true that the initial
> > heartbeat
> > > > is
> > > > > not
> > > > > > > >> > > > >>> idempotent,
> > > > > > > >> > > > >>>>> but
> > > > > > > >> > > > >>>>>>>>>>> this
> > > > > > > >> > > > >>>>>>>>>>>>>>> remains
> > > > > > > >> > > > >>>>>>>>>>>>>>> true with this KIP. It’s just
> > differently
> > > > not
> > > > > > > >> > idempotent.
> > > > > > > >> > > > >>> If
> > > > > > > >> > > > >>>>> the
> > > > > > > >> > > > >>>>>>>>>>>> client
> > > > > > > >> > > > >>>>>>>>>>>>>>> makes its
> > > > > > > >> > > > >>>>>>>>>>>>>>> own member ID, sends a request and
> dies,
> > > the
> > > > > GC
> > > > > > > will
> > > > > > > >> > > > >> still
> > > > > > > >> > > > >>>> have
> > > > > > > >> > > > >>>>>>>>>>> added
> > > > > > > >> > > > >>>>>>>>>>>>>>> the member to the group and it will
> hang
> > > > > around
> > > > > > > >> until
> > > > > > > >> > the
> > > > > > > >> > > > >>>>> session
> > > > > > > >> > > > >>>>>>>>>>>>> expires.
> > > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>>> I wonder if the GC could still
> generate
> > > the
> > > > > member
> > > > > > > >> ID
> > > > > > > >> > in
> > > > > > > >> > > > >>>>>>>>>> response to
> > > > > > > >> > > > >>>>>>>>>>>> the
> > > > > > > >> > > > >>>>>>>>>>>>>>> first
> > > > > > > >> > > > >>>>>>>>>>>>>>> heartbeat, and put the member in a
> > special
> > > > > PENDING
> > > > > > > >> > state
> > > > > > > >> > > > >>> with
> > > > > > > >> > > > >>>>> no
> > > > > > > >> > > > >>>>>>>>>>>>>>> assignments until the client sends the
> > > next
> > > > > > > >> heartbeat,
> > > > > > > >> > > > >> thus
> > > > > > > >> > > > >>>>>>>>>>> confirming
> > > > > > > >> > > > >>>>>>>>>>>>> it
> > > > > > > >> > > > >>>>>>>>>>>>>>> has received the member ID. This would
> > not
> > > > be
> > > > > a
> > > > > > > >> > protocol
> > > > > > > >> > > > >>>> change
> > > > > > > >> > > > >>>>>>>>>> at
> > > > > > > >> > > > >>>>>>>>>>>> all,
> > > > > > > >> > > > >>>>>>>>>>>>>>> just
> > > > > > > >> > > > >>>>>>>>>>>>>>> a change to the GC to keep a new
> member
> > in
> > > > the
> > > > > > > lobby
> > > > > > > >> > > > >> until
> > > > > > > >> > > > >>>> it’s
> > > > > > > >> > > > >>>>>>>>>>>>> comfirmed
> > > > > > > >> > > > >>>>>>>>>>>>>>> it knows its member ID.
> > > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>>> Thanks,
> > > > > > > >> > > > >>>>>>>>>>>>>>> Andrew
> > > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>>>> On 13 Aug 2024, at 15:59, TengYao
> Chi <
> > > > > > > >> > > > >>> kitingiao@gmail.com>
> > > > > > > >> > > > >>>>>>>>>> wrote:
> > > > > > > >> > > > >>>>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>>>> Hi Chia-Ping,
> > > > > > > >> > > > >>>>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>>>> Thanks for review and suggestions.
> > > > > > > >> > > > >>>>>>>>>>>>>>>> I have updated the content of KIP
> > > > > accordingly.
> > > > > > > >> > > > >>>>>>>>>>>>>>>> Please take a look.
> > > > > > > >> > > > >>>>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>>>> Best regards,
> > > > > > > >> > > > >>>>>>>>>>>>>>>> TengYao
> > > > > > > >> > > > >>>>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org>
> 於
> > > > > > > 2024年8月13日
> > > > > > > >> 週二
> > > > > > > >> > > > >>>>> 下午9:45寫道：
> > > > > > > >> > > > >>>>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>>>>> hi TengYao
> > > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>>>>> thanks for this KIP.
> > > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>>>>> 1) could you please describe the
> > > > > before/after
> > > > > > > >> > behavior
> > > > > > > >> > > > >> in
> > > > > > > >> > > > >>>> the
> > > > > > > >> > > > >>>>>>>>>>>>> "Proposed
> > > > > > > >> > > > >>>>>>>>>>>>>>>>> Changes" section? IIRC, current RPC
> > > allows
> > > > > HB
> > > > > > > >> having
> > > > > > > >> > > > >>> member
> > > > > > > >> > > > >>>>> id
> > > > > > > >> > > > >>>>>>>>>>>>>>> generated by
> > > > > > > >> > > > >>>>>>>>>>>>>>>>> client, right? If HB has no member
> ID,
> > > > > server
> > > > > > > will
> > > > > > > >> > > > >>> generate
> > > > > > > >> > > > >>>>> one
> > > > > > > >> > > > >>>>>>>>>>> and
> > > > > > > >> > > > >>>>>>>>>>>>> then
> > > > > > > >> > > > >>>>>>>>>>>>>>>>> return. The new behavior will
> enforce
> > HB
> > > > > "must"
> > > > > > > >> have
> > > > > > > >> > > > >>> member
> > > > > > > >> > > > >>>>> ID.
> > > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>>>>> 2) could you please write the
> version
> > > > number
> > > > > > > >> > explicitly
> > > > > > > >> > > > >>> in
> > > > > > > >> > > > >>>>> the
> > > > > > > >> > > > >>>>>>>>>> KIP
> > > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>>>>> 3) how new client code handle the
> old
> > > HB?
> > > > > Does
> > > > > > > it
> > > > > > > >> > > > >> always
> > > > > > > >> > > > >>>>>>>>>> generate
> > > > > > > >> > > > >>>>>>>>>>>>> member
> > > > > > > >> > > > >>>>>>>>>>>>>>>>> ID on client-side even though that
> is
> > > not
> > > > > > > >> restricted?
> > > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>>>>> Best,
> > > > > > > >> > > > >>>>>>>>>>>>>>>>> Chia-Ping
> > > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>>>>> On 2024/08/13 06:20:42 TengYao Chi
> > > wrote:
> > > > > > > >> > > > >>>>>>>>>>>>>>>>>> Hello everyone,
> > > > > > > >> > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>>>>>> I would like to start a discussion
> > > thread
> > > > > on
> > > > > > > >> > KIP-1082,
> > > > > > > >> > > > >>>> which
> > > > > > > >> > > > >>>>>>>>>>>> proposes
> > > > > > > >> > > > >>>>>>>>>>>>>>>>>> enabling id generation for clients
> > over
> > > > the
> > > > > > > >> > > > >>>>>>>>>>> ConsumerGroupHeartbeat
> > > > > > > >> > > > >>>>>>>>>>>>> RPC.
> > > > > > > >> > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>>>>>> Here is the KIP Link: KIP-1082
> > > > > > > >> > > > >>>>>>>>>>>>>>>>>> <
> > > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>
> > > > > > > >> > > > >>>>>>>
> > > > > > > >> > > > >>>>>
> > > > > > > >> > > > >>>>
> > > > > > > >> > > > >>>
> > > > > > > >> > > > >>
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
> > > > > > > >> > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>>>>>> Please take a look and let me know
> > what
> > > > you
> > > > > > > >> think,
> > > > > > > >> > > > >> and I
> > > > > > > >> > > > >>>>> would
> > > > > > > >> > > > >>>>>>>>>>>>>>> appreciate
> > > > > > > >> > > > >>>>>>>>>>>>>>>>>> any suggestions and feedback.
> > > > > > > >> > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>>>>>> Best regards,
> > > > > > > >> > > > >>>>>>>>>>>>>>>>>> TengYao
> > > > > > > >> > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>>
> > > > > > > >> > > > >>>>>>>>>
> > > > > > > >> > > > >>>>>>>
> > > > > > > >> > > > >>>>>>>
> > > > > > > >> > > > >>>>>
> > > > > > > >> > > > >>>>
> > > > > > > >> > > > >>>
> > > > > > > >> > > > >>
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[Chia-Ping Tsai <ch...@gmail.com>]

hi David

> For instance, we could check whether the member id is already used and we
could regenerate another one if it is. However, I tend to believe that the
risk of collision within a group is neglectable.

I agree the risk is negligible when the member ID is generated by the
server, as we can always ensure the server behaves responsibly. 😀
Therefore, I don't believe the additional check for the member ID is
necessary if the member id is created by server.

However, my concern is that consumers are now required to generate the
member ID, so as a responsible server, we need to handle the incorrect
client behavior. At least, based on David's comment, we can create another
member id when it is already used and it is rejoining the group (member
epoch = 0).

By the way, another related issue is that a duplicate consumer may
inadvertently terminate another consumer, as it will always send a
LEAVE_GROUP request upon closing, even if it hasn't received a heartbeat
response. However, I tend to believe the risk is negligible 😀

Best,
Chia-Ping


David Jacot <dj...@confluent.io.invalid> 於 2024年9月23日 週一 下午9:07寫道：

> Hi Chia-Ping,
>
> > Checks whether the member epoch matches the member epoch in its current
> assignment. FENCED_MEMBER_EPOCH is returned otherwise. The member is also
> removed from the group.
>
> We did not implement it like this in the end. We keep the member and only
> request him to re-synchronise himself by returning the FENCED_MEMBER_EPOCH
> error. We can remove it from the KIP.
>
> > Given that member id is the incarnation of consumer, it is hard to detect
> the scenario of "two consumers have the same member id" (it may be caused
> by the lost response). I guess the best approach is the later -created
> consumer should be idle with fenced error, and we should add this to the
> KIP (maybe we should revise the error message carried by
> `FencedMemberEpochException`). BUT, we need to rethink the approach if
> above description "The member is also removed from the group" is true
>
> I don't think that we have a reliable way to detect whether two members end
> up using the same member id. I don't think that it is possible if we
> generate the member id on the client side. By the way, we don't handle this
> case either on the server side today. We blindly generate a member id and
> use it. However, we could do better on the server side. For instance, we
> could check whether the member id is already used and we could regenerate
> another one if it is. However, I tend to believe that the risk of collision
> within a group is neglectable.
>
> Best,
> David
>
> On Fri, Sep 20, 2024 at 8:11 PM Chia-Ping Tsai <ch...@gmail.com> wrote:
>
> > wait a minute. There is a description about fenced error in the KIP-848
> >
> > > Checks whether the member epoch matches the member epoch in its current
> > assignment. FENCED_MEMBER_EPOCH is returned otherwise. The member is also
> > removed from the group.
> >
> > I failed to find the code of implementing "The member is also removed
> from
> > the group.". I will appreciate it if someone can share the code link to
> me.
> >
> > Let's assume it will be implemented eventually, and I feel that will
> cause
> > a serious issue as the consumers having the same member id will remove
> the
> > existing  member and then create itself member?
> >
> > > Once the client receives this exception, it should consider the
> exception
> > as a serious error and stop the process, and the user will need to
> assign a
> > new memberId.
> >
> > Given that member id is the incarnation of consumer, it is hard to detect
> > the scenario of "two consumers have the same member id" (it may be caused
> > by the lost response). I guess the best approach is the later -created
> > consumer should be idle with fenced error, and we should add this to the
> > KIP (maybe we should revise the error message carried by
> > `FencedMemberEpochException`). BUT, we need to rethink the approach if
> > above description "The member is also removed from the group" is true
> >
> > Best,
> > Chia-Ping
> >
> >
> >
> > TengYao Chi <ki...@gmail.com> 於 2024年9月20日 週五 下午8:17寫道：
> >
> > > Hi Chia-Ping,
> > >
> > > Thanks for pointing out this issue.
> > >
> > > I’m thinking that maybe we might need to define a new Exception to
> handle
> > > this scenario.
> > > Once the client receives this exception, it should consider the
> exception
> > > as a serious error and stop the process, and the user will need to
> > assign a
> > > new memberId.
> > > What do you think?
> > >
> > > Best Regards,
> > > TengYao
> > >
> > > Chia-Ping Tsai <ch...@apache.org> 於 2024年9月20日 週五 下午7:00寫道：
> > >
> > > > > This part is not clear either. It basically says that if a member
> > joins
> > > > with an existing member id but a different epoch, it will be fenced.
> > Then
> > > > it must rejoin with the same member id and epoch zero. This is
> already
> > > the
> > > > current behavior and it does not help with detecting duplicates,
> right?
> > > >
> > > > The duplicates issue is interesting. The member id is generated on
> > > > server-side by UUID before, so it seems to me the fenced member epoch
> > > > happens only if the client miss the response with bumped epoch, and
> > this
> > > > case is recoverable.
> > > >
> > > > however, v1 brings a different fenced epoch scenario. Users, now,
> have
> > > > responsibility to generate unique member id, hence clients may
> > encounter
> > > > infinite fenced error if there are >1 clients having same member id
> due
> > > to
> > > > config error (or other engineer error)
> > > >
> > > > Maybe we can highlight this scenario if we start to requires clients
> to
> > > > generate unique member id.
> > > >
> > > > Best,
> > > > Chia-Ping
> > > >
> > > > On 2024/09/19 18:36:48 David Jacot wrote:
> > > > > Hi,
> > > > >
> > > > > Thanks for the update. I have a few nits:
> > > > >
> > > > > > If the member ID is null or empty, the server will reject the
> > request
> > > > > with an InvalidRequestException.
> > > > > We should clarify that this should only apply to version >= 1.
> > > > >
> > > > > > The consumer instance must generate a member ID, and this ID
> should
> > > > > remain consistent for the duration of the consumer's session.
> Here, a
> > > > > "session" is defined as the period from the consumer's first
> > heartbeat
> > > > > until it leaves the group, either through a graceful shutdown, a
> > > > heartbeat
> > > > > timeout, or the process stopping or dying. The consumer instance
> > should
> > > > > reuse the same member ID for all heartbeats and rejoin attempts to
> > > > maintain
> > > > > continuity within the group.
> > > > >
> > > > > This part is not clear to me. When the member leaves the group, it
> > > should
> > > > > not reset the member id. I would rather say that the member must
> > > generate
> > > > > its member id when it starts and it must keep it until the process
> > > stops.
> > > > > It is basically an incarnation of the process.
> > > > >
> > > > > > If a conflict arises where the member ID generated by the client
> is
> > > > > detected to be a duplicate within the same group (for example, the
> > same
> > > > > member ID is associated with another active member in the group),
> the
> > > > > server will handle this by comparing the memberEpoch values of the
> > > > > conflicting members. The member with the lower memberEpoch is
> > > considered
> > > > > outdated and will be fenced off by the server. When this occurs,
> the
> > > > server
> > > > > responds with a FENCED_MEMBER_EPOCH error to the client, signaling
> it
> > > to
> > > > > rejoin the group with the same member ID while resetting the
> > > memberEpoch
> > > > to
> > > > > zero. This ensures that the client properly resynchronizes and
> > > maintains
> > > > > the continuity and consistency of the group membership.
> > > > >
> > > > > This part is not clear either. It basically says that if a member
> > joins
> > > > > with an existing member id but a different epoch, it will be
> fenced.
> > > Then
> > > > > it must rejoin with the same member id and epoch zero. This is
> > already
> > > > the
> > > > > current behavior and it does not help with detecting duplicates,
> > right?
> > > > > Should we just remove the paragraph?
> > > > >
> > > > > > A member ID mismatch occurs within a session: If the server
> > detects a
> > > > > mismatch between the provided member ID and the expected member ID
> > for
> > > an
> > > > > ongoing session, it should return a UNKNOWN_MEMBER_ID  error.
> > > > >
> > > > > How could we detect a mismatch between the provided and the
> expected
> > > > member
> > > > > id? My understanding is that we can only know whether the provided
> > > member
> > > > > id exists or not. This is already implemented.
> > > > >
> > > > > Thanks,
> > > > > David
> > > > >
> > > > > On Sat, Sep 14, 2024 at 9:31 AM TengYao Chi <ki...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hello everyone,
> > > > > >
> > > > > > Since this KIP has been fully discussed, I will initiate a vote
> for
> > > it
> > > > next
> > > > > > Monday.
> > > > > > Thank you and have a nice weekend.
> > > > > >
> > > > > > Best regards,
> > > > > > TengYao
> > > > > >
> > > > > > TengYao Chi <ki...@gmail.com> 於 2024年9月5日 週四 下午2:19寫道：
> > > > > >
> > > > > > > Hello everyone,
> > > > > > >
> > > > > > > KT2: It looks like everyone who has expressed an opinion
> supports
> > > the
> > > > > > > second option: “Document a recommendation for clients to use
> > UUIDs
> > > as
> > > > > > > member IDs, without strictly enforcing it.”
> > > > > > > I have updated the KIP accordingly.
> > > > > > > Please take a look, and let me know if you have any thoughts or
> > > > feedback.
> > > > > > >
> > > > > > > Thank you!
> > > > > > >
> > > > > > > Best regards,
> > > > > > > TengYao
> > > > > > >
> > > > > > > Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月30日 週五 下午9:56寫道：
> > > > > > >
> > > > > > >> hi TengYao
> > > > > > >>
> > > > > > >> KT2: +1 to second approach
> > > > > > >>
> > > > > > >> Best,
> > > > > > >> Chia-Ping
> > > > > > >>
> > > > > > >>
> > > > > > >> David Jacot <dj...@confluent.io.invalid> 於 2024年8月30日 週五
> > > 下午9:15寫道：
> > > > > > >>
> > > > > > >> > Hi TengYao,
> > > > > > >> >
> > > > > > >> > KT2: I don't think that we can realistically validate the
> uuid
> > > on
> > > > the
> > > > > > >> > server. It is basically a string of chars. So I lean towards
> > > > having a
> > > > > > >> good
> > > > > > >> > recommendation in the KIP and in the document of the field
> in
> > > the
> > > > > > RPC's
> > > > > > >> > definition.
> > > > > > >> >
> > > > > > >> > Best,
> > > > > > >> > David
> > > > > > >> >
> > > > > > >> > On Fri, Aug 30, 2024 at 3:02 PM TengYao Chi <
> > > kitingiao@gmail.com>
> > > > > > >> wrote:
> > > > > > >> >
> > > > > > >> > > Hello Kirk !
> > > > > > >> > >
> > > > > > >> > > Thank you for your comments !
> > > > > > >> > >
> > > > > > >> > > KT1: Yes, you are correct. The issue is not unique to the
> > > > initial
> > > > > > >> > > heartbeat; there can always be cases where the broker
> might
> > > lose
> > > > > > >> > connection
> > > > > > >> > > with a member.
> > > > > > >> > >
> > > > > > >> > > KT2: Currently, if the client doesn't have a member ID and
> > the
> > > > > > >> > memberEpoch
> > > > > > >> > > equals 0, the coordinator will generate a UUID as the
> member
> > > ID
> > > > for
> > > > > > >> the
> > > > > > >> > > client. However, at the RPC level, the member ID is sent
> as
> > a
> > > > > > literal
> > > > > > >> > > string, meaning there are no restrictions on the format at
> > > this
> > > > > > level.
> > > > > > >> > > This also reminds me that we haven't reached a final
> > > conclusion
> > > > on
> > > > > > >> how to
> > > > > > >> > > enforce the use of UUIDs.
> > > > > > >> > > From our previous discussions, I recall two possible
> > > approaches:
> > > > > > >> > > The first is to validate the UUID on the server side, and
> if
> > > > it's
> > > > > > not
> > > > > > >> > > valid, throw an exception to the client.
> > > > > > >> > > The second is to document a recommendation for clients to
> > use
> > > > UUIDs
> > > > > > as
> > > > > > >> > > member IDs, without strictly enforcing it.
> > > > > > >> > > I think it's time to decide on the approach we want to
> take.
> > > > > > >> > >
> > > > > > >> > > KT3: Yes, "session" can be considered synonymous with
> > > > "membership"
> > > > > > in
> > > > > > >> > this
> > > > > > >> > > context.
> > > > > > >> > >
> > > > > > >> > > KT4: Thank you for pointing that out. I will update the
> > > wording
> > > > to
> > > > > > >> > > specifically say this behavior is for consumers.
> > > > > > >> > >
> > > > > > >> > > Thanks again for your comments.
> > > > > > >> > >
> > > > > > >> > > Best regards,
> > > > > > >> > > TengYao
> > > > > > >> > >
> > > > > > >> > > Kirk True <ki...@kirktrue.pro> 於 2024年8月30日 週五 上午12:39寫道：
> > > > > > >> > >
> > > > > > >> > > > Hi TengYao!
> > > > > > >> > > >
> > > > > > >> > > > Sorry for being late to the discussion...
> > > > > > >> > > >
> > > > > > >> > > > After reading the thread and then the KIP, I had a few
> > > > > > >> > > questions/comments:
> > > > > > >> > > >
> > > > > > >> > > > KT1: In Motivation, it states: "This scenario can result
> > in
> > > > the
> > > > > > >> broker
> > > > > > >> > > > registering a new member for which it will never
> receive a
> > > > proper
> > > > > > >> leave
> > > > > > >> > > > request.” Just to be clear, the broker will always have
> > > cases
> > > > > > where
> > > > > > >> it
> > > > > > >> > > > might lose connection with a member. That’s not unique
> to
> > > the
> > > > > > >> initial
> > > > > > >> > > > heartbeat, right?
> > > > > > >> > > >
> > > > > > >> > > > KT2: There was a bit of back and forth about format of
> the
> > > > member
> > > > > > >> ID.
> > > > > > >> > > From
> > > > > > >> > > > what I gathered in the thread, the member ID is still
> > > defined
> > > > in
> > > > > > the
> > > > > > >> > RPC
> > > > > > >> > > as
> > > > > > >> > > > a string and not a UUID, right? The KIP states that the
> > > > “client
> > > > > > must
> > > > > > >> > > > generate a UUID as the member ID” and that the “server
> > will
> > > > > > validate
> > > > > > >> > > that a
> > > > > > >> > > > valid UUID is provided.” Is that a change for the
> server,
> > or
> > > > is it
> > > > > > >> > > already
> > > > > > >> > > > enforced as a UUID?
> > > > > > >> > > >
> > > > > > >> > > > KT3: Lianet mentioned some confusion over the use of the
> > > word
> > > > > > >> > “session.”
> > > > > > >> > > > Isn’t “session” synonymous with “membership?”
> > > > > > >> > > >
> > > > > > >> > > > KT4: Under “Member ID Lifecycle,” it states: "The client
> > > > should
> > > > > > >> reuse
> > > > > > >> > the
> > > > > > >> > > > same UUID as the member ID for all heartbeats and rejoin
> > > > attempts
> > > > > > to
> > > > > > >> > > > maintain continuity within the group.” Could we change
> the
> > > > first
> > > > > > >> part
> > > > > > >> > of
> > > > > > >> > > > that to “The Consumer instance should…” We do have
> > lifetimes
> > > > that
> > > > > > >> > extend
> > > > > > >> > > > past the lifetime of a client instance (such as the
> > > > transaction
> > > > > > ID).
> > > > > > >> > > >
> > > > > > >> > > > Thanks,
> > > > > > >> > > > Kirk
> > > > > > >> > > >
> > > > > > >> > > > > On Aug 29, 2024, at 1:28 AM, TengYao Chi <
> > > > kitingiao@gmail.com>
> > > > > > >> > wrote:
> > > > > > >> > > > >
> > > > > > >> > > > > Hi David,
> > > > > > >> > > > >
> > > > > > >> > > > > Thank you for pointing that out.
> > > > > > >> > > > > I have updated the content of the KIP based on
> Lianet's
> > > and
> > > > your
> > > > > > >> > > > feedback.
> > > > > > >> > > > > Please take a look and let me know your thoughts.
> > > > > > >> > > > >
> > > > > > >> > > > > Best regards,
> > > > > > >> > > > > TengYao
> > > > > > >> > > > >
> > > > > > >> > > > > David Jacot <dj...@confluent.io.invalid> 於
> 2024年8月29日
> > 週四
> > > > > > >> 下午3:20寫道：
> > > > > > >> > > > >
> > > > > > >> > > > >> Hi TengYao,
> > > > > > >> > > > >>
> > > > > > >> > > > >> Thanks for the update. I haven't fully read it yet
> but
> > I
> > > > will
> > > > > > >> soon.
> > > > > > >> > > > >>
> > > > > > >> > > > >> LM4: This is incorrect. The consumer must keep its
> > member
> > > > id
> > > > > > >> during
> > > > > > >> > > its
> > > > > > >> > > > >> entire lifetime (until the process stops or dies).
> The
> > > > protocol
> > > > > > >> > > > stipulates
> > > > > > >> > > > >> that a member must rejoin with the same member id and
> > the
> > > > > > member
> > > > > > >> > epoch
> > > > > > >> > > > set
> > > > > > >> > > > >> to zero when an FENCED_MEMBER_EPOCH occurs. This
> allows
> > > the
> > > > > > >> member
> > > > > > >> > to
> > > > > > >> > > > >> resynchronize itself. We should not change this
> > > behavior. I
> > > > > > think
> > > > > > >> > that
> > > > > > >> > > > we
> > > > > > >> > > > >> should see the client side generation id as an
> > > incarnation
> > > > id
> > > > > > of
> > > > > > >> the
> > > > > > >> > > > >> application. It is generated once and kept until it
> > stops
> > > > or
> > > > > > >> dies.
> > > > > > >> > > > >>
> > > > > > >> > > > >> Best,
> > > > > > >> > > > >> David
> > > > > > >> > > > >>
> > > > > > >> > > > >> On Thu, Aug 29, 2024 at 6:21 AM TengYao Chi <
> > > > > > kitingiao@gmail.com
> > > > > > >> >
> > > > > > >> > > > wrote:
> > > > > > >> > > > >>
> > > > > > >> > > > >>> Hello Lianet !
> > > > > > >> > > > >>>
> > > > > > >> > > > >>> Thanks for the reviews and suggestions!
> > > > > > >> > > > >>>
> > > > > > >> > > > >>> LM1: Indeed, we plan to enforce client-side ID
> > > generation
> > > > in
> > > > > > the
> > > > > > >> > > > future,
> > > > > > >> > > > >>> and it is not an alternative. I will change the
> title
> > > > > > >> accordingly.
> > > > > > >> > > > >>>
> > > > > > >> > > > >>> LM2: Yes, that's the expectation. I will add that
> > > > statement to
> > > > > > >> the
> > > > > > >> > > > public
> > > > > > >> > > > >>> interface section.
> > > > > > >> > > > >>>
> > > > > > >> > > > >>> LM3: Thank you for the high-level perspective
> review.
> > I
> > > > think
> > > > > > >> > you're
> > > > > > >> > > > >> right;
> > > > > > >> > > > >>> our intention isn't very clear since it was placed
> at
> > > the
> > > > end
> > > > > > of
> > > > > > >> > the
> > > > > > >> > > > >>> section. I will try to rephrase that section to make
> > it
> > > > more
> > > > > > >> > obvious.
> > > > > > >> > > > >>>
> > > > > > >> > > > >>> LM4: Regarding the definition of "session" in this
> > KIP,
> > > I
> > > > > > >> believe
> > > > > > >> > it
> > > > > > >> > > > >> refers
> > > > > > >> > > > >>> to the period between the *first-time heartbeat* and
> > > when
> > > > the
> > > > > > >> > > *consumer
> > > > > > >> > > > >>> leaves the group* (whether through a graceful
> shutdown
> > > or
> > > > a
> > > > > > >> > heartbeat
> > > > > > >> > > > >>> timeout). The consumer should reuse its UUID if it
> has
> > > > been
> > > > > > >> > generated
> > > > > > >> > > > >>> before. The only situation in which it will
> regenerate
> > > the
> > > > > > UUID
> > > > > > >> is
> > > > > > >> > if
> > > > > > >> > > > the
> > > > > > >> > > > >>> coordinator finds that there is already a consumer
> > with
> > > > the
> > > > > > same
> > > > > > >> > > UUID.
> > > > > > >> > > > >>> IIRC, the coordinator should compare the member
> > epochs,
> > > > and
> > > > > > the
> > > > > > >> > > > >>> later-joined consumer should be fenced off by the
> > > > coordinator
> > > > > > >> due
> > > > > > >> > to
> > > > > > >> > > > >> having
> > > > > > >> > > > >>> a lower member epoch. Once the consumer receives a
> > > > > > >> > > > `FENCED_MEMBER_EPOCH`
> > > > > > >> > > > >>> error, it will generate a new UUID and attempt to
> > > rejoin.
> > > > I
> > > > > > will
> > > > > > >> > > > clarify
> > > > > > >> > > > >>> this in the KIP.
> > > > > > >> > > > >>>
> > > > > > >> > > > >>> Thanks again for your reviews, I really appreciate
> it.
> > > > > > >> > > > >>>
> > > > > > >> > > > >>> Best regards,
> > > > > > >> > > > >>> TengYao
> > > > > > >> > > > >>>
> > > > > > >> > > > >>> Lianet M. <li...@gmail.com> 於 2024年8月28日 週三
> > > 下午7:12寫道：
> > > > > > >> > > > >>>
> > > > > > >> > > > >>>> Hello TengYao! Thanks for taking on this issue,
> we've
> > > > been
> > > > > > >> going
> > > > > > >> > > > around
> > > > > > >> > > > >>> it
> > > > > > >> > > > >>>> for a while.
> > > > > > >> > > > >>>>
> > > > > > >> > > > >>>> LM1: About the title of the KIP: "Enable ID
> > Generation
> > > > for
> > > > > > >> Clients
> > > > > > >> > > > over
> > > > > > >> > > > >>> the
> > > > > > >> > > > >>>> ConsumerGroupHeartbeat RPC". I find it confusing
> > > because
> > > > it
> > > > > > >> hints
> > > > > > >> > > that
> > > > > > >> > > > >>>> we're adding it as an alternative (which was
> > discussed
> > > > and
> > > > > > >> > > discarded,
> > > > > > >> > > > >> in
> > > > > > >> > > > >>>> favour of really enforcing it). It's also missing
> the
> > > > core
> > > > > > >> change
> > > > > > >> > > imo,
> > > > > > >> > > > >>>> which is "where" the generation happens. So, maybe
> > more
> > > > to
> > > > > > the
> > > > > > >> > point
> > > > > > >> > > > >> with
> > > > > > >> > > > >>>> something along the lines of "Client-side generated
> > ID
> > > > for
> > > > > > >> clients
> > > > > > >> > > > over
> > > > > > >> > > > >>>> ConsumerGroupHeartbeat RPC"?
> > > > > > >> > > > >>>>
> > > > > > >> > > > >>>> LM2: On the public interfaces section, the KIP
> states
> > > > that
> > > > > > "the
> > > > > > >> > > server
> > > > > > >> > > > >>> will
> > > > > > >> > > > >>>> reject the request", but we should agree on the
> > > specific
> > > > > > error
> > > > > > >> > > type. I
> > > > > > >> > > > >>>> expect it should fail with an
> > InvalidRequestException,
> > > is
> > > > > > that
> > > > > > >> the
> > > > > > >> > > > >>>> intention? (This was also suggested in the
> discussion
> > > > thread
> > > > > > >> > before
> > > > > > >> > > > but
> > > > > > >> > > > >>> is
> > > > > > >> > > > >>>> not in the KIP).
> > > > > > >> > > > >>>>
> > > > > > >> > > > >>>> LM3. Related to my previous point, I find that to
> be
> > > the
> > > > true
> > > > > > >> > > > >>> public-facing
> > > > > > >> > > > >>>> change (member ID mandatory at the protocol level),
> > but
> > > > it's
> > > > > > >> only
> > > > > > >> > at
> > > > > > >> > > > >> the
> > > > > > >> > > > >>>> end of the Public interfaces changes, kind of lost
> > > among
> > > > > > >> details
> > > > > > >> > of
> > > > > > >> > > > how
> > > > > > >> > > > >>>> we're going to do it. Should we rephrase that
> section
> > > > with
> > > > > > the
> > > > > > >> > > actual
> > > > > > >> > > > >>>> change first, and the hows after (ex. Bumping the
> > > > version is
> > > > > > >> not
> > > > > > >> > the
> > > > > > >> > > > >>>> public-facing change in this case, it's just the
> > > > mechanism to
> > > > > > >> > > properly
> > > > > > >> > > > >>>> introduce our change)
> > > > > > >> > > > >>>>
> > > > > > >> > > > >>>> LM4. Regarding the lifetime of the UUID: the KIP
> > states
> > > > we
> > > > > > will
> > > > > > >> > > > "Verify
> > > > > > >> > > > >>>> that the UUID remains consistent across all
> > subsequent
> > > > > > >> heartbeats
> > > > > > >> > > > >> during
> > > > > > >> > > > >>>> the session". What is this "session" referring to
> > > here? I
> > > > > > would
> > > > > > >> > > expect
> > > > > > >> > > > >>> that
> > > > > > >> > > > >>>> the UUID is associated to a consumer instance
> > > (generated
> > > > for
> > > > > > >> the
> > > > > > >> > > > >> consumer
> > > > > > >> > > > >>>> the first time it needs to send a HB if it doesn't
> > have
> > > > the
> > > > > > >> UUID
> > > > > > >> > > yet.
> > > > > > >> > > > >>> From
> > > > > > >> > > > >>>> there on, every time it needs to send a "first HB"
> > > > again, it
> > > > > > >> will
> > > > > > >> > > > reuse
> > > > > > >> > > > >>> its
> > > > > > >> > > > >>>> UUID, is that the intention? Note that we should
> > > consider
> > > > > > that
> > > > > > >> the
> > > > > > >> > > > same
> > > > > > >> > > > >>>> consumer instance may have many "first heartbeats",
> > > > meaning
> > > > > > >> > > heartbeats
> > > > > > >> > > > >> to
> > > > > > >> > > > >>>> join the group when it's not part of it (ex.
> consumer
> > > > > > >> unsubscribe
> > > > > > >> > +
> > > > > > >> > > > >>>> subscribe, fenced, stale). Is this the intention or
> > are
> > > > you
> > > > > > >> > > > considering
> > > > > > >> > > > >>> the
> > > > > > >> > > > >>>> lifetime differently? We should clarify it in the
> > KIP.
> > > > > > >> > > > >>>>
> > > > > > >> > > > >>>> Thanks!
> > > > > > >> > > > >>>>
> > > > > > >> > > > >>>> Lianet
> > > > > > >> > > > >>>>
> > > > > > >> > > > >>>> On Tue, Aug 27, 2024 at 2:27 AM TengYao Chi <
> > > > > > >> kitingiao@gmail.com>
> > > > > > >> > > > >> wrote:
> > > > > > >> > > > >>>>
> > > > > > >> > > > >>>>> Hi everyone,
> > > > > > >> > > > >>>>>
> > > > > > >> > > > >>>>> I have revised this KIP multiple times based on
> the
> > > > feedback
> > > > > > >> from
> > > > > > >> > > our
> > > > > > >> > > > >>>>> discussions.
> > > > > > >> > > > >>>>> I would greatly appreciate it if you could review
> it
> > > > when
> > > > > > you
> > > > > > >> > have
> > > > > > >> > > > >> the
> > > > > > >> > > > >>>>> time.
> > > > > > >> > > > >>>>> If there are no further comments or suggestions, I
> > > plan
> > > > to
> > > > > > >> > proceed
> > > > > > >> > > > >> with
> > > > > > >> > > > >>>>> initiating a vote soon.
> > > > > > >> > > > >>>>>
> > > > > > >> > > > >>>>> Best regards,
> > > > > > >> > > > >>>>> TengYao
> > > > > > >> > > > >>>>>
> > > > > > >> > > > >>>>> TengYao Chi <ki...@gmail.com> 於 2024年8月23日 週五
> > > > 下午2:43寫道：
> > > > > > >> > > > >>>>>
> > > > > > >> > > > >>>>>> Hi Andrew,
> > > > > > >> > > > >>>>>> Thank you for your previous feedback and
> insights.
> > > > > > >> > > > >>>>>> Your contributions have added valuable
> perspectives
> > > to
> > > > the
> > > > > > >> > > > >>> discussions.
> > > > > > >> > > > >>>>>> And we also benefit from the comparison of
> > different
> > > > > > >> solutions.
> > > > > > >> > > > >>>>>> I’m also looking forward to seeing an initial
> > version
> > > > in
> > > > > > >> > KIP-932,
> > > > > > >> > > > >> as
> > > > > > >> > > > >>> it
> > > > > > >> > > > >>>>>> will provide a good reference for future
> > > > implementations.
> > > > > > >> > > > >>>>>>
> > > > > > >> > > > >>>>>> Regarding your comment on AS2, I wanted to
> clarify
> > > > that my
> > > > > > >> > > > >>>> specification
> > > > > > >> > > > >>>>>> references org.apache.kafka.common.Uuid.
> > > > > > >> > > > >>>>>> I believe we’re referring to the same class, and
> it
> > > > might
> > > > > > >> just
> > > > > > >> > be
> > > > > > >> > > a
> > > > > > >> > > > >>>> small
> > > > > > >> > > > >>>>>> oversight due to the busy schedule.
> > > > > > >> > > > >>>>>>
> > > > > > >> > > > >>>>>> I want to express my gratitude once again for
> your
> > > many
> > > > > > >> > insightful
> > > > > > >> > > > >>>>>> comments, which have helped the discussion
> progress
> > > > > > smoothly.
> > > > > > >> > > > >>>>>>
> > > > > > >> > > > >>>>>> Best regards,
> > > > > > >> > > > >>>>>> TengYao
> > > > > > >> > > > >>>>>>
> > > > > > >> > > > >>>>>>
> > > > > > >> > > > >>>>>> Andrew Schofield <an...@live.com> 於
> > > > 2024年8月22日
> > > > > > 週四
> > > > > > >> > > > >>>> 下午11:28寫道：
> > > > > > >> > > > >>>>>>
> > > > > > >> > > > >>>>>>> Hi TengYao,
> > > > > > >> > > > >>>>>>> I’ve been reading through the comments and I’m
> > happy
> > > > that
> > > > > > >> the
> > > > > > >> > > > >> lobby
> > > > > > >> > > > >>>>>>> approach has not gained support.
> > > > > > >> > > > >>>>>>>
> > > > > > >> > > > >>>>>>> Assuming that this KIP is voted, I will be happy
> > to
> > > > change
> > > > > > >> > > KIP-932
> > > > > > >> > > > >>> so
> > > > > > >> > > > >>>>>>> that it only supports client-side member ID
> > > > generation.
> > > > > > >> Because
> > > > > > >> > > > >> that
> > > > > > >> > > > >>>> KIP
> > > > > > >> > > > >>>>>>> is still
> > > > > > >> > > > >>>>>>> under development, I can do this in the first
> > > version
> > > > of
> > > > > > >> > > > >>>>>>> ShareGroupHeartbeat.
> > > > > > >> > > > >>>>>>>
> > > > > > >> > > > >>>>>>> AS2: For the encoding section, I suppose the
> > > specific
> > > > > > >> encoding
> > > > > > >> > > > >> which
> > > > > > >> > > > >>>>>>> is used is what org.apache.kafka.utils.Uuid
> uses.
> > > > > > >> > > > >>>>>>>
> > > > > > >> > > > >>>>>>> Thanks,
> > > > > > >> > > > >>>>>>> Andrew
> > > > > > >> > > > >>>>>>>
> > > > > > >> > > > >>>>>>>> On 14 Aug 2024, at 17:00, TengYao Chi <
> > > > > > kitingiao@gmail.com
> > > > > > >> >
> > > > > > >> > > > >>> wrote:
> > > > > > >> > > > >>>>>>>>
> > > > > > >> > > > >>>>>>>> Hello Apoorv,
> > > > > > >> > > > >>>>>>>> Thank you for your feedback.
> > > > > > >> > > > >>>>>>>> Regarding the questions you raised,
> > unfortunately,
> > > > this
> > > > > > KIP
> > > > > > >> > > > >> cannot
> > > > > > >> > > > >>>>>>>> guarantee the order of heartbeats. As with many
> > > > classic
> > > > > > >> > > > >>> distributed
> > > > > > >> > > > >>>>>>> system
> > > > > > >> > > > >>>>>>>> challenges, what we can do is make our best
> > effort
> > > to
> > > > > > >> ensure
> > > > > > >> > > > >> that
> > > > > > >> > > > >>>>> there
> > > > > > >> > > > >>>>>>> are
> > > > > > >> > > > >>>>>>>> no idle members or stale assignments under
> normal
> > > > > > >> > circumstances.
> > > > > > >> > > > >>>>>>>>
> > > > > > >> > > > >>>>>>>> As for the lobby approach, I’m not a fan of it
> > > > because it
> > > > > > >> > > > >> requires
> > > > > > >> > > > >>>>>>> adding a
> > > > > > >> > > > >>>>>>>> mechanism to maintain client state within the
> > > > > > >> ConsumerGroup,
> > > > > > >> > > > >>> which,
> > > > > > >> > > > >>>> in
> > > > > > >> > > > >>>>>>> my
> > > > > > >> > > > >>>>>>>> view, resembles something like a two-phase
> > commit.
> > > > This
> > > > > > >> would
> > > > > > >> > > > >>>>> introduce
> > > > > > >> > > > >>>>>>>> more complexity than the proposal in this KIP,
> > > which
> > > > is
> > > > > > >> > > > >> something
> > > > > > >> > > > >>> we
> > > > > > >> > > > >>>>>>> want
> > > > > > >> > > > >>>>>>>> to avoid. KIP-848 aims to simplify the existing
> > > > protocol,
> > > > > > >> and
> > > > > > >> > > > >>> while
> > > > > > >> > > > >>>>> the
> > > > > > >> > > > >>>>>>>> lobby approach is a good one, I believe it is
> not
> > > the
> > > > > > right
> > > > > > >> > fit
> > > > > > >> > > > >>> for
> > > > > > >> > > > >>>>> this
> > > > > > >> > > > >>>>>>>> particular situation.
> > > > > > >> > > > >>>>>>>>
> > > > > > >> > > > >>>>>>>> Best regards,
> > > > > > >> > > > >>>>>>>> TengYao
> > > > > > >> > > > >>>>>>>>
> > > > > > >> > > > >>>>>>>> TengYao Chi <ki...@gmail.com> 於 2024年8月14日
> > 週三
> > > > > > >> 下午11:45寫道：
> > > > > > >> > > > >>>>>>>>
> > > > > > >> > > > >>>>>>>>> Hi David,
> > > > > > >> > > > >>>>>>>>>
> > > > > > >> > > > >>>>>>>>> I really appreciate your review and
> suggestions.
> > > As
> > > > I am
> > > > > > >> > still
> > > > > > >> > > > >>>>> gaining
> > > > > > >> > > > >>>>>>>>> experience in writing KIPs, your input has
> been
> > > > > > incredibly
> > > > > > >> > > > >>>> helpful. I
> > > > > > >> > > > >>>>>>> am
> > > > > > >> > > > >>>>>>>>> currently applying your suggestions to the KIP
> > and
> > > > will
> > > > > > >> > > > >> complete
> > > > > > >> > > > >>> it
> > > > > > >> > > > >>>>> as
> > > > > > >> > > > >>>>>>> soon
> > > > > > >> > > > >>>>>>>>> as possible.
> > > > > > >> > > > >>>>>>>>> Regarding the UUID part, I think we haven’t
> > > reached
> > > > a
> > > > > > >> > > > >> conclusion
> > > > > > >> > > > >>>>>>> yet.(So
> > > > > > >> > > > >>>>>>>>> far according to this thread)
> > > > > > >> > > > >>>>>>>>> However, I will review the current
> > implementation
> > > > in the
> > > > > > >> > Kafka
> > > > > > >> > > > >>>> `Uuid`
> > > > > > >> > > > >>>>>>>>> class and include a brief specification in the
> > > KIP.
> > > > > > >> > > > >>>>>>>>>
> > > > > > >> > > > >>>>>>>>> Once again, thank you so much for your help.
> > > > > > >> > > > >>>>>>>>>
> > > > > > >> > > > >>>>>>>>> Best regards,
> > > > > > >> > > > >>>>>>>>> TengYao
> > > > > > >> > > > >>>>>>>>>
> > > > > > >> > > > >>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> > 2024年8月14日
> > > 週三
> > > > > > >> > 下午11:14寫道：
> > > > > > >> > > > >>>>>>>>>
> > > > > > >> > > > >>>>>>>>>> hi Apoorv
> > > > > > >> > > > >>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>> As the memberId is now known to the client,
> > and
> > > > client
> > > > > > >> > might
> > > > > > >> > > > >>> send
> > > > > > >> > > > >>>>> the
> > > > > > >> > > > >>>>>>>>>> leave
> > > > > > >> > > > >>>>>>>>>> group heartbeat on shutdown prior to
> receiving
> > > the
> > > > > > >> initial
> > > > > > >> > > > >>>> heartbeat
> > > > > > >> > > > >>>>>>>>>> response. If that's true then how do we
> > guarantee
> > > > that
> > > > > > >> the 2
> > > > > > >> > > > >>>>> requests
> > > > > > >> > > > >>>>>>> to
> > > > > > >> > > > >>>>>>>>>> join and leave will be processed in order,
> > which
> > > > could
> > > > > > >> still
> > > > > > >> > > > >>> leave
> > > > > > >> > > > >>>>>>> stale
> > > > > > >> > > > >>>>>>>>>> members or throw unknown member id
> exceptions?
> > > > > > >> > > > >>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>> This is definitely a good question. the short
> > > > answer:
> > > > > > no
> > > > > > >> > > > >>> guarantee
> > > > > > >> > > > >>>>> but
> > > > > > >> > > > >>>>>>>>>> best
> > > > > > >> > > > >>>>>>>>>> efforts
> > > > > > >> > > > >>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>> Please notice the root cause is "we have no
> > > enough
> > > > time
> > > > > > >> to
> > > > > > >> > > > >> wait
> > > > > > >> > > > >>>>>>> member id
> > > > > > >> > > > >>>>>>>>>> (response) when closing consumer". Sadly, we
> > can'
> > > > > > >> guarantee
> > > > > > >> > > > >> the
> > > > > > >> > > > >>>>>>> request
> > > > > > >> > > > >>>>>>>>>> order due to the same reason.
> > > > > > >> > > > >>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>> However, in contrast to previous behavior,
> > there
> > > > is one
> > > > > > >> big
> > > > > > >> > > > >>>> benefit
> > > > > > >> > > > >>>>>>> of new
> > > > > > >> > > > >>>>>>>>>> approach - we can try STONITH because we know
> > the
> > > > > > member
> > > > > > >> id
> > > > > > >> > > > >>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>> Best,
> > > > > > >> > > > >>>>>>>>>> Chia-Ping
> > > > > > >> > > > >>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>> Apoorv Mittal <ap...@gmail.com> 於
> > > > 2024年8月14日
> > > > > > 週三
> > > > > > >> > > > >>>> 下午8:55寫道：
> > > > > > >> > > > >>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>> Hi TengYao,
> > > > > > >> > > > >>>>>>>>>>> Thanks for the KIP. Continuing on the point
> > > which
> > > > > > Andrew
> > > > > > >> > > > >>>> mentioned
> > > > > > >> > > > >>>>> as
> > > > > > >> > > > >>>>>>>>>> AS1.
> > > > > > >> > > > >>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>> As the memberId is now known to the client,
> > and
> > > > client
> > > > > > >> > might
> > > > > > >> > > > >>> send
> > > > > > >> > > > >>>>> the
> > > > > > >> > > > >>>>>>>>>> leave
> > > > > > >> > > > >>>>>>>>>>> group heartbeat on shutdown prior to
> receiving
> > > the
> > > > > > >> initial
> > > > > > >> > > > >>>>> heartbeat
> > > > > > >> > > > >>>>>>>>>>> response. If that's true then how do we
> > > guarantee
> > > > that
> > > > > > >> the
> > > > > > >> > 2
> > > > > > >> > > > >>>>>>> requests to
> > > > > > >> > > > >>>>>>>>>>> join and leave will be processed in order,
> > which
> > > > could
> > > > > > >> > still
> > > > > > >> > > > >>>> leave
> > > > > > >> > > > >>>>>>> stale
> > > > > > >> > > > >>>>>>>>>>> members or throw unknown member id
> exceptions?
> > > > > > >> > > > >>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>> Though the client side member id generation
> is
> > > > helpful
> > > > > > >> > which
> > > > > > >> > > > >>> will
> > > > > > >> > > > >>>>>>>>>> represent
> > > > > > >> > > > >>>>>>>>>>> the same group perspective as from client
> and
> > > > broker's
> > > > > > >> end.
> > > > > > >> > > > >>> But I
> > > > > > >> > > > >>>>>>> think
> > > > > > >> > > > >>>>>>>>>> the
> > > > > > >> > > > >>>>>>>>>>> major concern we want to solve here is Stale
> > > > Partition
> > > > > > >> > > > >>>> Assignments
> > > > > > >> > > > >>>>>>> which
> > > > > > >> > > > >>>>>>>>>>> might still exist with the new approach. I
> am
> > > > leaning
> > > > > > >> > towards
> > > > > > >> > > > >>> the
> > > > > > >> > > > >>>>>>>>>>> suggestion mentioned by Andrew where
> partition
> > > > > > >> assignment
> > > > > > >> > > > >>>> triggers
> > > > > > >> > > > >>>>> on
> > > > > > >> > > > >>>>>>>>>>> subsequent heartbeat when client
> acknowledges
> > > the
> > > > > > >> initial
> > > > > > >> > > > >>>>> heartbeat,
> > > > > > >> > > > >>>>>>>>>>> delayed partition assignment.
> > > > > > >> > > > >>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>> Though on a separate note, I have a
> different
> > > > > > question.
> > > > > > >> > What
> > > > > > >> > > > >>>>> happens
> > > > > > >> > > > >>>>>>>>>> when
> > > > > > >> > > > >>>>>>>>>>> there is an issue with the client which
> sends
> > > the
> > > > > > >> initial
> > > > > > >> > > > >>>> heartbeat
> > > > > > >> > > > >>>>>>>>>> without
> > > > > > >> > > > >>>>>>>>>>> memberId, then crashes and restarts? I think
> > we
> > > > must
> > > > > > be
> > > > > > >> > > > >>>>> re-triggering
> > > > > > >> > > > >>>>>>>>>>> assignments and expiring members only after
> > the
> > > > > > >> heartbeat
> > > > > > >> > > > >>> session
> > > > > > >> > > > >>>>>>>>>> timeout?
> > > > > > >> > > > >>>>>>>>>>> If that's true then shall delayed partition
> > > > assignment
> > > > > > >> can
> > > > > > >> > > > >> help
> > > > > > >> > > > >>>>>>> benefit
> > > > > > >> > > > >>>>>>>>>> us
> > > > > > >> > > > >>>>>>>>>>> from this situation as well?
> > > > > > >> > > > >>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>> Regards,
> > > > > > >> > > > >>>>>>>>>>> Apoorv Mittal
> > > > > > >> > > > >>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>> On Wed, Aug 14, 2024 at 12:51 PM David Jacot
> > > > > > >> > > > >>>>>>>>>> <dj...@confluent.io.invalid>
> > > > > > >> > > > >>>>>>>>>>> wrote:
> > > > > > >> > > > >>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>> Hi Andrew,
> > > > > > >> > > > >>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>> Personally, I don't like the lobby
> approach.
> > It
> > > > makes
> > > > > > >> > things
> > > > > > >> > > > >>>> more
> > > > > > >> > > > >>>>>>>>>>>> complicated and it would require changing
> the
> > > > records
> > > > > > >> on
> > > > > > >> > the
> > > > > > >> > > > >>>>> server
> > > > > > >> > > > >>>>>>>>>> too.
> > > > > > >> > > > >>>>>>>>>>>> This is why I initially suggested the
> > rejected
> > > > > > >> alternative
> > > > > > >> > > > >> #2
> > > > > > >> > > > >>>>> which
> > > > > > >> > > > >>>>>>> is
> > > > > > >> > > > >>>>>>>>>>>> pretty close but also not perfect.
> > > > > > >> > > > >>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>> I'd like to clarify one thing. The
> > > > > > >> ConsumerGroupHeartbeat
> > > > > > >> > > > >> API
> > > > > > >> > > > >>>>>>> already
> > > > > > >> > > > >>>>>>>>>>>> supports generating the member id on the
> > client
> > > > so we
> > > > > > >> > don't
> > > > > > >> > > > >>> need
> > > > > > >> > > > >>>>> any
> > > > > > >> > > > >>>>>>>>>>>> conditional logic on the client side. This
> is
> > > > > > actually
> > > > > > >> > what
> > > > > > >> > > > >> we
> > > > > > >> > > > >>>>>>> wanted
> > > > > > >> > > > >>>>>>>>>> to
> > > > > > >> > > > >>>>>>>>>>> do
> > > > > > >> > > > >>>>>>>>>>>> in the first place but the idea got pushed
> > back
> > > > by
> > > > > > >> Magnus
> > > > > > >> > > > >> back
> > > > > > >> > > > >>>>> then
> > > > > > >> > > > >>>>>>>>>>> because
> > > > > > >> > > > >>>>>>>>>>>> generating uuid from librdkafka required a
> > new
> > > > > > >> dependency.
> > > > > > >> > > > >> It
> > > > > > >> > > > >>>>> turns
> > > > > > >> > > > >>>>>>>>>> out
> > > > > > >> > > > >>>>>>>>>>>> that librdkafka has that dependency today.
> In
> > > > > > >> retrospect,
> > > > > > >> > we
> > > > > > >> > > > >>>>> should
> > > > > > >> > > > >>>>>>>>>> have
> > > > > > >> > > > >>>>>>>>>>>> pushed back on this. Long story short, we
> can
> > > > just do
> > > > > > >> it.
> > > > > > >> > > > >> The
> > > > > > >> > > > >>>>>>>>>> proposal in
> > > > > > >> > > > >>>>>>>>>>>> this KIP is to make the member id required
> in
> > > > future
> > > > > > >> > > > >> versions.
> > > > > > >> > > > >>>> We
> > > > > > >> > > > >>>>>>>>>> could
> > > > > > >> > > > >>>>>>>>>>>> also decide not to do it and to keep
> > supporting
> > > > both
> > > > > > >> > > > >>>> approaches. I
> > > > > > >> > > > >>>>>>>>>> would
> > > > > > >> > > > >>>>>>>>>>>> also be fine with this.
> > > > > > >> > > > >>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>> Best,
> > > > > > >> > > > >>>>>>>>>>>> David
> > > > > > >> > > > >>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>> On Wed, Aug 14, 2024 at 12:30 PM Andrew
> > > > Schofield <
> > > > > > >> > > > >>>>>>>>>>>> andrew_schofield@live.com>
> > > > > > >> > > > >>>>>>>>>>>> wrote:
> > > > > > >> > > > >>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>> Hi TengYao,
> > > > > > >> > > > >>>>>>>>>>>>> Thanks for your response. I’ll have just
> one
> > > > more
> > > > > > try
> > > > > > >> to
> > > > > > >> > > > >>>>> persuade.
> > > > > > >> > > > >>>>>>>>>>>>> I feel that I will need to follow the
> > approach
> > > > with
> > > > > > >> > KIP-932
> > > > > > >> > > > >>>> when
> > > > > > >> > > > >>>>>>>>>> we’ve
> > > > > > >> > > > >>>>>>>>>>>>> made a decision, so I do have more than a
> > > > passing
> > > > > > >> > interest
> > > > > > >> > > > >> in
> > > > > > >> > > > >>>>> this.
> > > > > > >> > > > >>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>> A group member in the lobby is in the
> group,
> > > > but it
> > > > > > >> does
> > > > > > >> > > > >> not
> > > > > > >> > > > >>>> have
> > > > > > >> > > > >>>>>>>>>> any
> > > > > > >> > > > >>>>>>>>>>>>> assignments. A member of a consumer group
> > can
> > > > have
> > > > > > no
> > > > > > >> > > > >>> assigned
> > > > > > >> > > > >>>>>>>>>>>>> partitions (such as 5 CG members
> subscribed
> > > to a
> > > > > > topic
> > > > > > >> > > > >> with 4
> > > > > > >> > > > >>>>>>>>>>>> partitions),
> > > > > > >> > > > >>>>>>>>>>>>> so it’s a situation that consumer group
> > > members
> > > > > > >> already
> > > > > > >> > > > >>> expect.
> > > > > > >> > > > >>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>> One of Kafka’s strengths is the way that
> we
> > > > handle
> > > > > > API
> > > > > > >> > > > >>>>> versioning.
> > > > > > >> > > > >>>>>>>>>>>>> But, there is a cost - the behaviour is
> > > > different
> > > > > > >> > depending
> > > > > > >> > > > >>> on
> > > > > > >> > > > >>>>> the
> > > > > > >> > > > >>>>>>>>>> RPC
> > > > > > >> > > > >>>>>>>>>>>>> version. KIP-848 is on the cusp of
> > completion,
> > > > but
> > > > > > >> we’re
> > > > > > >> > > > >>>> already
> > > > > > >> > > > >>>>>>>>>> adding
> > > > > > >> > > > >>>>>>>>>>>>> conditional logic for v0/v1 for
> > > > > > >> ConsumerGroupHeartbeat.
> > > > > > >> > > > >>> That’s
> > > > > > >> > > > >>>> a
> > > > > > >> > > > >>>>>>>>>> pity.
> > > > > > >> > > > >>>>>>>>>>>>> Only a minor issue, but it’s unfortunate.
> > > > > > >> > > > >>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>> Thanks,
> > > > > > >> > > > >>>>>>>>>>>>> Andrew
> > > > > > >> > > > >>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>> On 14 Aug 2024, at 08:47, TengYao Chi <
> > > > > > >> > > > >> kitingiao@gmail.com>
> > > > > > >> > > > >>>>>>>>>> wrote:
> > > > > > >> > > > >>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>> Hello Andrew
> > > > > > >> > > > >>>>>>>>>>>>>> Thank you for your thoughtful suggestions
> > and
> > > > > > getting
> > > > > > >> > the
> > > > > > >> > > > >>>>>>>>>> discussion
> > > > > > >> > > > >>>>>>>>>>>>> going.
> > > > > > >> > > > >>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>> To AS1:
> > > > > > >> > > > >>>>>>>>>>>>>> In the current scenario where the server
> > > > generates
> > > > > > >> the
> > > > > > >> > > > >> UUID,
> > > > > > >> > > > >>>> if
> > > > > > >> > > > >>>>>>>>>> the
> > > > > > >> > > > >>>>>>>>>>>>> client
> > > > > > >> > > > >>>>>>>>>>>>>> shuts down before receiving the memberId
> > > > generated
> > > > > > by
> > > > > > >> > the
> > > > > > >> > > > >> GC
> > > > > > >> > > > >>>>>>>>>>>> (regardless
> > > > > > >> > > > >>>>>>>>>>>>> of
> > > > > > >> > > > >>>>>>>>>>>>>> whether it’s a graceful shutdown or not),
> > the
> > > > GC
> > > > > > will
> > > > > > >> > > > >> still
> > > > > > >> > > > >>>> have
> > > > > > >> > > > >>>>>>>>>> to
> > > > > > >> > > > >>>>>>>>>>>> wait
> > > > > > >> > > > >>>>>>>>>>>>>> for the heartbeat timeout because the
> > client
> > > > > > doesn’t
> > > > > > >> > know
> > > > > > >> > > > >>> its
> > > > > > >> > > > >>>>>>>>>>> memberId.
> > > > > > >> > > > >>>>>>>>>>>>>> This KIP indeed cannot completely resolve
> > the
> > > > > > >> > idempotency
> > > > > > >> > > > >>>> issue,
> > > > > > >> > > > >>>>>>>>>> but
> > > > > > >> > > > >>>>>>>>>>> it
> > > > > > >> > > > >>>>>>>>>>>>> can
> > > > > > >> > > > >>>>>>>>>>>>>> better handle shutdown scenarios under
> > normal
> > > > > > >> > > > >> circumstances
> > > > > > >> > > > >>>>>>>>>> because
> > > > > > >> > > > >>>>>>>>>>> the
> > > > > > >> > > > >>>>>>>>>>>>>> client always knows its memberId. Even if
> > the
> > > > > > client
> > > > > > >> > shuts
> > > > > > >> > > > >>>> down
> > > > > > >> > > > >>>>>>>>>>>>> immediately
> > > > > > >> > > > >>>>>>>>>>>>>> after the initial heartbeat, as long as
> it
> > > > > > performs a
> > > > > > >> > > > >>> graceful
> > > > > > >> > > > >>>>>>>>>>> shutdown
> > > > > > >> > > > >>>>>>>>>>>>> and
> > > > > > >> > > > >>>>>>>>>>>>>> sends a leave heartbeat, the GC can
> manage
> > > the
> > > > > > >> situation
> > > > > > >> > > > >> and
> > > > > > >> > > > >>>>>>>>>> remove
> > > > > > >> > > > >>>>>>>>>>> the
> > > > > > >> > > > >>>>>>>>>>>>>> member. Therefore, the goal of this KIP
> is
> > to
> > > > > > address
> > > > > > >> > the
> > > > > > >> > > > >>>> issue
> > > > > > >> > > > >>>>>>>>>> where
> > > > > > >> > > > >>>>>>>>>>>> the
> > > > > > >> > > > >>>>>>>>>>>>>> GC has to wait for the heartbeat timeout
> > due
> > > > to the
> > > > > > >> > client
> > > > > > >> > > > >>>>> leaving
> > > > > > >> > > > >>>>>>>>>>>>> without
> > > > > > >> > > > >>>>>>>>>>>>>> knowing its memberId, which leads to
> > reduced
> > > > > > >> throughput
> > > > > > >> > > > >> and
> > > > > > >> > > > >>>>>>>>>> limited
> > > > > > >> > > > >>>>>>>>>>>>>> scalability.
> > > > > > >> > > > >>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>> The solution you suggest has also been
> > > > proposed by
> > > > > > >> > David.
> > > > > > >> > > > >>> The
> > > > > > >> > > > >>>>>>>>>> concern
> > > > > > >> > > > >>>>>>>>>>>>> with
> > > > > > >> > > > >>>>>>>>>>>>>> this approach is that it introduces
> > > additional
> > > > > > >> > complexity
> > > > > > >> > > > >>> for
> > > > > > >> > > > >>>>>>>>>>>>>> compatibility, as the new server would
> not
> > > > > > >> immediately
> > > > > > >> > add
> > > > > > >> > > > >>> the
> > > > > > >> > > > >>>>>>>>>> member
> > > > > > >> > > > >>>>>>>>>>>> to
> > > > > > >> > > > >>>>>>>>>>>>>> the group, while the old server would.
> This
> > > > > > requires
> > > > > > >> > > > >> clients
> > > > > > >> > > > >>>> to
> > > > > > >> > > > >>>>>>>>>>>>>> differentiate whether their memberId has
> > been
> > > > added
> > > > > > >> to
> > > > > > >> > the
> > > > > > >> > > > >>>> group
> > > > > > >> > > > >>>>>>>>>> or
> > > > > > >> > > > >>>>>>>>>>>> not,
> > > > > > >> > > > >>>>>>>>>>>>>> which could result in unexpected logs.
> > > > > > >> > > > >>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>> Best Regards,
> > > > > > >> > > > >>>>>>>>>>>>>> TengYao
> > > > > > >> > > > >>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>> Andrew Schofield <
> > andrew_schofield@live.com>
> > > 於
> > > > > > >> > 2024年8月14日
> > > > > > >> > > > >>> 週三
> > > > > > >> > > > >>>>>>>>>>>> 上午12:29寫道：
> > > > > > >> > > > >>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>>> Hi TengYao,
> > > > > > >> > > > >>>>>>>>>>>>>>> Thanks for the KIP. I wonder if there’s
> a
> > > > > > different
> > > > > > >> way
> > > > > > >> > > > >> to
> > > > > > >> > > > >>>>> close
> > > > > > >> > > > >>>>>>>>>>> what
> > > > > > >> > > > >>>>>>>>>>>>>>> is quite a small window.
> > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>>> AS1: It is true that the initial
> heartbeat
> > > is
> > > > not
> > > > > > >> > > > >>> idempotent,
> > > > > > >> > > > >>>>> but
> > > > > > >> > > > >>>>>>>>>>> this
> > > > > > >> > > > >>>>>>>>>>>>>>> remains
> > > > > > >> > > > >>>>>>>>>>>>>>> true with this KIP. It’s just
> differently
> > > not
> > > > > > >> > idempotent.
> > > > > > >> > > > >>> If
> > > > > > >> > > > >>>>> the
> > > > > > >> > > > >>>>>>>>>>>> client
> > > > > > >> > > > >>>>>>>>>>>>>>> makes its
> > > > > > >> > > > >>>>>>>>>>>>>>> own member ID, sends a request and dies,
> > the
> > > > GC
> > > > > > will
> > > > > > >> > > > >> still
> > > > > > >> > > > >>>> have
> > > > > > >> > > > >>>>>>>>>>> added
> > > > > > >> > > > >>>>>>>>>>>>>>> the member to the group and it will hang
> > > > around
> > > > > > >> until
> > > > > > >> > the
> > > > > > >> > > > >>>>> session
> > > > > > >> > > > >>>>>>>>>>>>> expires.
> > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>>> I wonder if the GC could still generate
> > the
> > > > member
> > > > > > >> ID
> > > > > > >> > in
> > > > > > >> > > > >>>>>>>>>> response to
> > > > > > >> > > > >>>>>>>>>>>> the
> > > > > > >> > > > >>>>>>>>>>>>>>> first
> > > > > > >> > > > >>>>>>>>>>>>>>> heartbeat, and put the member in a
> special
> > > > PENDING
> > > > > > >> > state
> > > > > > >> > > > >>> with
> > > > > > >> > > > >>>>> no
> > > > > > >> > > > >>>>>>>>>>>>>>> assignments until the client sends the
> > next
> > > > > > >> heartbeat,
> > > > > > >> > > > >> thus
> > > > > > >> > > > >>>>>>>>>>> confirming
> > > > > > >> > > > >>>>>>>>>>>>> it
> > > > > > >> > > > >>>>>>>>>>>>>>> has received the member ID. This would
> not
> > > be
> > > > a
> > > > > > >> > protocol
> > > > > > >> > > > >>>> change
> > > > > > >> > > > >>>>>>>>>> at
> > > > > > >> > > > >>>>>>>>>>>> all,
> > > > > > >> > > > >>>>>>>>>>>>>>> just
> > > > > > >> > > > >>>>>>>>>>>>>>> a change to the GC to keep a new member
> in
> > > the
> > > > > > lobby
> > > > > > >> > > > >> until
> > > > > > >> > > > >>>> it’s
> > > > > > >> > > > >>>>>>>>>>>>> comfirmed
> > > > > > >> > > > >>>>>>>>>>>>>>> it knows its member ID.
> > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>>> Thanks,
> > > > > > >> > > > >>>>>>>>>>>>>>> Andrew
> > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>>>> On 13 Aug 2024, at 15:59, TengYao Chi <
> > > > > > >> > > > >>> kitingiao@gmail.com>
> > > > > > >> > > > >>>>>>>>>> wrote:
> > > > > > >> > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>>>> Hi Chia-Ping,
> > > > > > >> > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>>>> Thanks for review and suggestions.
> > > > > > >> > > > >>>>>>>>>>>>>>>> I have updated the content of KIP
> > > > accordingly.
> > > > > > >> > > > >>>>>>>>>>>>>>>> Please take a look.
> > > > > > >> > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>>>> Best regards,
> > > > > > >> > > > >>>>>>>>>>>>>>>> TengYao
> > > > > > >> > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > > > > > 2024年8月13日
> > > > > > >> 週二
> > > > > > >> > > > >>>>> 下午9:45寫道：
> > > > > > >> > > > >>>>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>>>>> hi TengYao
> > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>>>>> thanks for this KIP.
> > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>>>>> 1) could you please describe the
> > > > before/after
> > > > > > >> > behavior
> > > > > > >> > > > >> in
> > > > > > >> > > > >>>> the
> > > > > > >> > > > >>>>>>>>>>>>> "Proposed
> > > > > > >> > > > >>>>>>>>>>>>>>>>> Changes" section? IIRC, current RPC
> > allows
> > > > HB
> > > > > > >> having
> > > > > > >> > > > >>> member
> > > > > > >> > > > >>>>> id
> > > > > > >> > > > >>>>>>>>>>>>>>> generated by
> > > > > > >> > > > >>>>>>>>>>>>>>>>> client, right? If HB has no member ID,
> > > > server
> > > > > > will
> > > > > > >> > > > >>> generate
> > > > > > >> > > > >>>>> one
> > > > > > >> > > > >>>>>>>>>>> and
> > > > > > >> > > > >>>>>>>>>>>>> then
> > > > > > >> > > > >>>>>>>>>>>>>>>>> return. The new behavior will enforce
> HB
> > > > "must"
> > > > > > >> have
> > > > > > >> > > > >>> member
> > > > > > >> > > > >>>>> ID.
> > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>>>>> 2) could you please write the version
> > > number
> > > > > > >> > explicitly
> > > > > > >> > > > >>> in
> > > > > > >> > > > >>>>> the
> > > > > > >> > > > >>>>>>>>>> KIP
> > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>>>>> 3) how new client code handle the old
> > HB?
> > > > Does
> > > > > > it
> > > > > > >> > > > >> always
> > > > > > >> > > > >>>>>>>>>> generate
> > > > > > >> > > > >>>>>>>>>>>>> member
> > > > > > >> > > > >>>>>>>>>>>>>>>>> ID on client-side even though that is
> > not
> > > > > > >> restricted?
> > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>>>>> Best,
> > > > > > >> > > > >>>>>>>>>>>>>>>>> Chia-Ping
> > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>>>>> On 2024/08/13 06:20:42 TengYao Chi
> > wrote:
> > > > > > >> > > > >>>>>>>>>>>>>>>>>> Hello everyone,
> > > > > > >> > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>>>>>> I would like to start a discussion
> > thread
> > > > on
> > > > > > >> > KIP-1082,
> > > > > > >> > > > >>>> which
> > > > > > >> > > > >>>>>>>>>>>> proposes
> > > > > > >> > > > >>>>>>>>>>>>>>>>>> enabling id generation for clients
> over
> > > the
> > > > > > >> > > > >>>>>>>>>>> ConsumerGroupHeartbeat
> > > > > > >> > > > >>>>>>>>>>>>> RPC.
> > > > > > >> > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>>>>>> Here is the KIP Link: KIP-1082
> > > > > > >> > > > >>>>>>>>>>>>>>>>>> <
> > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>
> > > > > > >> > > > >>>>>>>
> > > > > > >> > > > >>>>>
> > > > > > >> > > > >>>>
> > > > > > >> > > > >>>
> > > > > > >> > > > >>
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
> > > > > > >> > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>>>>>> Please take a look and let me know
> what
> > > you
> > > > > > >> think,
> > > > > > >> > > > >> and I
> > > > > > >> > > > >>>>> would
> > > > > > >> > > > >>>>>>>>>>>>>>> appreciate
> > > > > > >> > > > >>>>>>>>>>>>>>>>>> any suggestions and feedback.
> > > > > > >> > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>>>>>> Best regards,
> > > > > > >> > > > >>>>>>>>>>>>>>>>>> TengYao
> > > > > > >> > > > >>>>>>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>>
> > > > > > >> > > > >>>>>>>>>
> > > > > > >> > > > >>>>>>>
> > > > > > >> > > > >>>>>>>
> > > > > > >> > > > >>>>>
> > > > > > >> > > > >>>>
> > > > > > >> > > > >>>
> > > > > > >> > > > >>
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[David Jacot <dj...@confluent.io.INVALID>]

Hi Chia-Ping,

> Checks whether the member epoch matches the member epoch in its current
assignment. FENCED_MEMBER_EPOCH is returned otherwise. The member is also
removed from the group.

We did not implement it like this in the end. We keep the member and only
request him to re-synchronise himself by returning the FENCED_MEMBER_EPOCH
error. We can remove it from the KIP.

> Given that member id is the incarnation of consumer, it is hard to detect
the scenario of "two consumers have the same member id" (it may be caused
by the lost response). I guess the best approach is the later -created
consumer should be idle with fenced error, and we should add this to the
KIP (maybe we should revise the error message carried by
`FencedMemberEpochException`). BUT, we need to rethink the approach if
above description "The member is also removed from the group" is true

I don't think that we have a reliable way to detect whether two members end
up using the same member id. I don't think that it is possible if we
generate the member id on the client side. By the way, we don't handle this
case either on the server side today. We blindly generate a member id and
use it. However, we could do better on the server side. For instance, we
could check whether the member id is already used and we could regenerate
another one if it is. However, I tend to believe that the risk of collision
within a group is neglectable.

Best,
David

On Fri, Sep 20, 2024 at 8:11 PM Chia-Ping Tsai <ch...@gmail.com> wrote:

> wait a minute. There is a description about fenced error in the KIP-848
>
> > Checks whether the member epoch matches the member epoch in its current
> assignment. FENCED_MEMBER_EPOCH is returned otherwise. The member is also
> removed from the group.
>
> I failed to find the code of implementing "The member is also removed from
> the group.". I will appreciate it if someone can share the code link to me.
>
> Let's assume it will be implemented eventually, and I feel that will cause
> a serious issue as the consumers having the same member id will remove the
> existing  member and then create itself member?
>
> > Once the client receives this exception, it should consider the exception
> as a serious error and stop the process, and the user will need to assign a
> new memberId.
>
> Given that member id is the incarnation of consumer, it is hard to detect
> the scenario of "two consumers have the same member id" (it may be caused
> by the lost response). I guess the best approach is the later -created
> consumer should be idle with fenced error, and we should add this to the
> KIP (maybe we should revise the error message carried by
> `FencedMemberEpochException`). BUT, we need to rethink the approach if
> above description "The member is also removed from the group" is true
>
> Best,
> Chia-Ping
>
>
>
> TengYao Chi <ki...@gmail.com> 於 2024年9月20日 週五 下午8:17寫道：
>
> > Hi Chia-Ping,
> >
> > Thanks for pointing out this issue.
> >
> > I’m thinking that maybe we might need to define a new Exception to handle
> > this scenario.
> > Once the client receives this exception, it should consider the exception
> > as a serious error and stop the process, and the user will need to
> assign a
> > new memberId.
> > What do you think?
> >
> > Best Regards,
> > TengYao
> >
> > Chia-Ping Tsai <ch...@apache.org> 於 2024年9月20日 週五 下午7:00寫道：
> >
> > > > This part is not clear either. It basically says that if a member
> joins
> > > with an existing member id but a different epoch, it will be fenced.
> Then
> > > it must rejoin with the same member id and epoch zero. This is already
> > the
> > > current behavior and it does not help with detecting duplicates, right?
> > >
> > > The duplicates issue is interesting. The member id is generated on
> > > server-side by UUID before, so it seems to me the fenced member epoch
> > > happens only if the client miss the response with bumped epoch, and
> this
> > > case is recoverable.
> > >
> > > however, v1 brings a different fenced epoch scenario. Users, now, have
> > > responsibility to generate unique member id, hence clients may
> encounter
> > > infinite fenced error if there are >1 clients having same member id due
> > to
> > > config error (or other engineer error)
> > >
> > > Maybe we can highlight this scenario if we start to requires clients to
> > > generate unique member id.
> > >
> > > Best,
> > > Chia-Ping
> > >
> > > On 2024/09/19 18:36:48 David Jacot wrote:
> > > > Hi,
> > > >
> > > > Thanks for the update. I have a few nits:
> > > >
> > > > > If the member ID is null or empty, the server will reject the
> request
> > > > with an InvalidRequestException.
> > > > We should clarify that this should only apply to version >= 1.
> > > >
> > > > > The consumer instance must generate a member ID, and this ID should
> > > > remain consistent for the duration of the consumer's session. Here, a
> > > > "session" is defined as the period from the consumer's first
> heartbeat
> > > > until it leaves the group, either through a graceful shutdown, a
> > > heartbeat
> > > > timeout, or the process stopping or dying. The consumer instance
> should
> > > > reuse the same member ID for all heartbeats and rejoin attempts to
> > > maintain
> > > > continuity within the group.
> > > >
> > > > This part is not clear to me. When the member leaves the group, it
> > should
> > > > not reset the member id. I would rather say that the member must
> > generate
> > > > its member id when it starts and it must keep it until the process
> > stops.
> > > > It is basically an incarnation of the process.
> > > >
> > > > > If a conflict arises where the member ID generated by the client is
> > > > detected to be a duplicate within the same group (for example, the
> same
> > > > member ID is associated with another active member in the group), the
> > > > server will handle this by comparing the memberEpoch values of the
> > > > conflicting members. The member with the lower memberEpoch is
> > considered
> > > > outdated and will be fenced off by the server. When this occurs, the
> > > server
> > > > responds with a FENCED_MEMBER_EPOCH error to the client, signaling it
> > to
> > > > rejoin the group with the same member ID while resetting the
> > memberEpoch
> > > to
> > > > zero. This ensures that the client properly resynchronizes and
> > maintains
> > > > the continuity and consistency of the group membership.
> > > >
> > > > This part is not clear either. It basically says that if a member
> joins
> > > > with an existing member id but a different epoch, it will be fenced.
> > Then
> > > > it must rejoin with the same member id and epoch zero. This is
> already
> > > the
> > > > current behavior and it does not help with detecting duplicates,
> right?
> > > > Should we just remove the paragraph?
> > > >
> > > > > A member ID mismatch occurs within a session: If the server
> detects a
> > > > mismatch between the provided member ID and the expected member ID
> for
> > an
> > > > ongoing session, it should return a UNKNOWN_MEMBER_ID  error.
> > > >
> > > > How could we detect a mismatch between the provided and the expected
> > > member
> > > > id? My understanding is that we can only know whether the provided
> > member
> > > > id exists or not. This is already implemented.
> > > >
> > > > Thanks,
> > > > David
> > > >
> > > > On Sat, Sep 14, 2024 at 9:31 AM TengYao Chi <ki...@gmail.com>
> > wrote:
> > > >
> > > > > Hello everyone,
> > > > >
> > > > > Since this KIP has been fully discussed, I will initiate a vote for
> > it
> > > next
> > > > > Monday.
> > > > > Thank you and have a nice weekend.
> > > > >
> > > > > Best regards,
> > > > > TengYao
> > > > >
> > > > > TengYao Chi <ki...@gmail.com> 於 2024年9月5日 週四 下午2:19寫道：
> > > > >
> > > > > > Hello everyone,
> > > > > >
> > > > > > KT2: It looks like everyone who has expressed an opinion supports
> > the
> > > > > > second option: “Document a recommendation for clients to use
> UUIDs
> > as
> > > > > > member IDs, without strictly enforcing it.”
> > > > > > I have updated the KIP accordingly.
> > > > > > Please take a look, and let me know if you have any thoughts or
> > > feedback.
> > > > > >
> > > > > > Thank you!
> > > > > >
> > > > > > Best regards,
> > > > > > TengYao
> > > > > >
> > > > > > Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月30日 週五 下午9:56寫道：
> > > > > >
> > > > > >> hi TengYao
> > > > > >>
> > > > > >> KT2: +1 to second approach
> > > > > >>
> > > > > >> Best,
> > > > > >> Chia-Ping
> > > > > >>
> > > > > >>
> > > > > >> David Jacot <dj...@confluent.io.invalid> 於 2024年8月30日 週五
> > 下午9:15寫道：
> > > > > >>
> > > > > >> > Hi TengYao,
> > > > > >> >
> > > > > >> > KT2: I don't think that we can realistically validate the uuid
> > on
> > > the
> > > > > >> > server. It is basically a string of chars. So I lean towards
> > > having a
> > > > > >> good
> > > > > >> > recommendation in the KIP and in the document of the field in
> > the
> > > > > RPC's
> > > > > >> > definition.
> > > > > >> >
> > > > > >> > Best,
> > > > > >> > David
> > > > > >> >
> > > > > >> > On Fri, Aug 30, 2024 at 3:02 PM TengYao Chi <
> > kitingiao@gmail.com>
> > > > > >> wrote:
> > > > > >> >
> > > > > >> > > Hello Kirk !
> > > > > >> > >
> > > > > >> > > Thank you for your comments !
> > > > > >> > >
> > > > > >> > > KT1: Yes, you are correct. The issue is not unique to the
> > > initial
> > > > > >> > > heartbeat; there can always be cases where the broker might
> > lose
> > > > > >> > connection
> > > > > >> > > with a member.
> > > > > >> > >
> > > > > >> > > KT2: Currently, if the client doesn't have a member ID and
> the
> > > > > >> > memberEpoch
> > > > > >> > > equals 0, the coordinator will generate a UUID as the member
> > ID
> > > for
> > > > > >> the
> > > > > >> > > client. However, at the RPC level, the member ID is sent as
> a
> > > > > literal
> > > > > >> > > string, meaning there are no restrictions on the format at
> > this
> > > > > level.
> > > > > >> > > This also reminds me that we haven't reached a final
> > conclusion
> > > on
> > > > > >> how to
> > > > > >> > > enforce the use of UUIDs.
> > > > > >> > > From our previous discussions, I recall two possible
> > approaches:
> > > > > >> > > The first is to validate the UUID on the server side, and if
> > > it's
> > > > > not
> > > > > >> > > valid, throw an exception to the client.
> > > > > >> > > The second is to document a recommendation for clients to
> use
> > > UUIDs
> > > > > as
> > > > > >> > > member IDs, without strictly enforcing it.
> > > > > >> > > I think it's time to decide on the approach we want to take.
> > > > > >> > >
> > > > > >> > > KT3: Yes, "session" can be considered synonymous with
> > > "membership"
> > > > > in
> > > > > >> > this
> > > > > >> > > context.
> > > > > >> > >
> > > > > >> > > KT4: Thank you for pointing that out. I will update the
> > wording
> > > to
> > > > > >> > > specifically say this behavior is for consumers.
> > > > > >> > >
> > > > > >> > > Thanks again for your comments.
> > > > > >> > >
> > > > > >> > > Best regards,
> > > > > >> > > TengYao
> > > > > >> > >
> > > > > >> > > Kirk True <ki...@kirktrue.pro> 於 2024年8月30日 週五 上午12:39寫道：
> > > > > >> > >
> > > > > >> > > > Hi TengYao!
> > > > > >> > > >
> > > > > >> > > > Sorry for being late to the discussion...
> > > > > >> > > >
> > > > > >> > > > After reading the thread and then the KIP, I had a few
> > > > > >> > > questions/comments:
> > > > > >> > > >
> > > > > >> > > > KT1: In Motivation, it states: "This scenario can result
> in
> > > the
> > > > > >> broker
> > > > > >> > > > registering a new member for which it will never receive a
> > > proper
> > > > > >> leave
> > > > > >> > > > request.” Just to be clear, the broker will always have
> > cases
> > > > > where
> > > > > >> it
> > > > > >> > > > might lose connection with a member. That’s not unique to
> > the
> > > > > >> initial
> > > > > >> > > > heartbeat, right?
> > > > > >> > > >
> > > > > >> > > > KT2: There was a bit of back and forth about format of the
> > > member
> > > > > >> ID.
> > > > > >> > > From
> > > > > >> > > > what I gathered in the thread, the member ID is still
> > defined
> > > in
> > > > > the
> > > > > >> > RPC
> > > > > >> > > as
> > > > > >> > > > a string and not a UUID, right? The KIP states that the
> > > “client
> > > > > must
> > > > > >> > > > generate a UUID as the member ID” and that the “server
> will
> > > > > validate
> > > > > >> > > that a
> > > > > >> > > > valid UUID is provided.” Is that a change for the server,
> or
> > > is it
> > > > > >> > > already
> > > > > >> > > > enforced as a UUID?
> > > > > >> > > >
> > > > > >> > > > KT3: Lianet mentioned some confusion over the use of the
> > word
> > > > > >> > “session.”
> > > > > >> > > > Isn’t “session” synonymous with “membership?”
> > > > > >> > > >
> > > > > >> > > > KT4: Under “Member ID Lifecycle,” it states: "The client
> > > should
> > > > > >> reuse
> > > > > >> > the
> > > > > >> > > > same UUID as the member ID for all heartbeats and rejoin
> > > attempts
> > > > > to
> > > > > >> > > > maintain continuity within the group.” Could we change the
> > > first
> > > > > >> part
> > > > > >> > of
> > > > > >> > > > that to “The Consumer instance should…” We do have
> lifetimes
> > > that
> > > > > >> > extend
> > > > > >> > > > past the lifetime of a client instance (such as the
> > > transaction
> > > > > ID).
> > > > > >> > > >
> > > > > >> > > > Thanks,
> > > > > >> > > > Kirk
> > > > > >> > > >
> > > > > >> > > > > On Aug 29, 2024, at 1:28 AM, TengYao Chi <
> > > kitingiao@gmail.com>
> > > > > >> > wrote:
> > > > > >> > > > >
> > > > > >> > > > > Hi David,
> > > > > >> > > > >
> > > > > >> > > > > Thank you for pointing that out.
> > > > > >> > > > > I have updated the content of the KIP based on Lianet's
> > and
> > > your
> > > > > >> > > > feedback.
> > > > > >> > > > > Please take a look and let me know your thoughts.
> > > > > >> > > > >
> > > > > >> > > > > Best regards,
> > > > > >> > > > > TengYao
> > > > > >> > > > >
> > > > > >> > > > > David Jacot <dj...@confluent.io.invalid> 於 2024年8月29日
> 週四
> > > > > >> 下午3:20寫道：
> > > > > >> > > > >
> > > > > >> > > > >> Hi TengYao,
> > > > > >> > > > >>
> > > > > >> > > > >> Thanks for the update. I haven't fully read it yet but
> I
> > > will
> > > > > >> soon.
> > > > > >> > > > >>
> > > > > >> > > > >> LM4: This is incorrect. The consumer must keep its
> member
> > > id
> > > > > >> during
> > > > > >> > > its
> > > > > >> > > > >> entire lifetime (until the process stops or dies). The
> > > protocol
> > > > > >> > > > stipulates
> > > > > >> > > > >> that a member must rejoin with the same member id and
> the
> > > > > member
> > > > > >> > epoch
> > > > > >> > > > set
> > > > > >> > > > >> to zero when an FENCED_MEMBER_EPOCH occurs. This allows
> > the
> > > > > >> member
> > > > > >> > to
> > > > > >> > > > >> resynchronize itself. We should not change this
> > behavior. I
> > > > > think
> > > > > >> > that
> > > > > >> > > > we
> > > > > >> > > > >> should see the client side generation id as an
> > incarnation
> > > id
> > > > > of
> > > > > >> the
> > > > > >> > > > >> application. It is generated once and kept until it
> stops
> > > or
> > > > > >> dies.
> > > > > >> > > > >>
> > > > > >> > > > >> Best,
> > > > > >> > > > >> David
> > > > > >> > > > >>
> > > > > >> > > > >> On Thu, Aug 29, 2024 at 6:21 AM TengYao Chi <
> > > > > kitingiao@gmail.com
> > > > > >> >
> > > > > >> > > > wrote:
> > > > > >> > > > >>
> > > > > >> > > > >>> Hello Lianet !
> > > > > >> > > > >>>
> > > > > >> > > > >>> Thanks for the reviews and suggestions!
> > > > > >> > > > >>>
> > > > > >> > > > >>> LM1: Indeed, we plan to enforce client-side ID
> > generation
> > > in
> > > > > the
> > > > > >> > > > future,
> > > > > >> > > > >>> and it is not an alternative. I will change the title
> > > > > >> accordingly.
> > > > > >> > > > >>>
> > > > > >> > > > >>> LM2: Yes, that's the expectation. I will add that
> > > statement to
> > > > > >> the
> > > > > >> > > > public
> > > > > >> > > > >>> interface section.
> > > > > >> > > > >>>
> > > > > >> > > > >>> LM3: Thank you for the high-level perspective review.
> I
> > > think
> > > > > >> > you're
> > > > > >> > > > >> right;
> > > > > >> > > > >>> our intention isn't very clear since it was placed at
> > the
> > > end
> > > > > of
> > > > > >> > the
> > > > > >> > > > >>> section. I will try to rephrase that section to make
> it
> > > more
> > > > > >> > obvious.
> > > > > >> > > > >>>
> > > > > >> > > > >>> LM4: Regarding the definition of "session" in this
> KIP,
> > I
> > > > > >> believe
> > > > > >> > it
> > > > > >> > > > >> refers
> > > > > >> > > > >>> to the period between the *first-time heartbeat* and
> > when
> > > the
> > > > > >> > > *consumer
> > > > > >> > > > >>> leaves the group* (whether through a graceful shutdown
> > or
> > > a
> > > > > >> > heartbeat
> > > > > >> > > > >>> timeout). The consumer should reuse its UUID if it has
> > > been
> > > > > >> > generated
> > > > > >> > > > >>> before. The only situation in which it will regenerate
> > the
> > > > > UUID
> > > > > >> is
> > > > > >> > if
> > > > > >> > > > the
> > > > > >> > > > >>> coordinator finds that there is already a consumer
> with
> > > the
> > > > > same
> > > > > >> > > UUID.
> > > > > >> > > > >>> IIRC, the coordinator should compare the member
> epochs,
> > > and
> > > > > the
> > > > > >> > > > >>> later-joined consumer should be fenced off by the
> > > coordinator
> > > > > >> due
> > > > > >> > to
> > > > > >> > > > >> having
> > > > > >> > > > >>> a lower member epoch. Once the consumer receives a
> > > > > >> > > > `FENCED_MEMBER_EPOCH`
> > > > > >> > > > >>> error, it will generate a new UUID and attempt to
> > rejoin.
> > > I
> > > > > will
> > > > > >> > > > clarify
> > > > > >> > > > >>> this in the KIP.
> > > > > >> > > > >>>
> > > > > >> > > > >>> Thanks again for your reviews, I really appreciate it.
> > > > > >> > > > >>>
> > > > > >> > > > >>> Best regards,
> > > > > >> > > > >>> TengYao
> > > > > >> > > > >>>
> > > > > >> > > > >>> Lianet M. <li...@gmail.com> 於 2024年8月28日 週三
> > 下午7:12寫道：
> > > > > >> > > > >>>
> > > > > >> > > > >>>> Hello TengYao! Thanks for taking on this issue, we've
> > > been
> > > > > >> going
> > > > > >> > > > around
> > > > > >> > > > >>> it
> > > > > >> > > > >>>> for a while.
> > > > > >> > > > >>>>
> > > > > >> > > > >>>> LM1: About the title of the KIP: "Enable ID
> Generation
> > > for
> > > > > >> Clients
> > > > > >> > > > over
> > > > > >> > > > >>> the
> > > > > >> > > > >>>> ConsumerGroupHeartbeat RPC". I find it confusing
> > because
> > > it
> > > > > >> hints
> > > > > >> > > that
> > > > > >> > > > >>>> we're adding it as an alternative (which was
> discussed
> > > and
> > > > > >> > > discarded,
> > > > > >> > > > >> in
> > > > > >> > > > >>>> favour of really enforcing it). It's also missing the
> > > core
> > > > > >> change
> > > > > >> > > imo,
> > > > > >> > > > >>>> which is "where" the generation happens. So, maybe
> more
> > > to
> > > > > the
> > > > > >> > point
> > > > > >> > > > >> with
> > > > > >> > > > >>>> something along the lines of "Client-side generated
> ID
> > > for
> > > > > >> clients
> > > > > >> > > > over
> > > > > >> > > > >>>> ConsumerGroupHeartbeat RPC"?
> > > > > >> > > > >>>>
> > > > > >> > > > >>>> LM2: On the public interfaces section, the KIP states
> > > that
> > > > > "the
> > > > > >> > > server
> > > > > >> > > > >>> will
> > > > > >> > > > >>>> reject the request", but we should agree on the
> > specific
> > > > > error
> > > > > >> > > type. I
> > > > > >> > > > >>>> expect it should fail with an
> InvalidRequestException,
> > is
> > > > > that
> > > > > >> the
> > > > > >> > > > >>>> intention? (This was also suggested in the discussion
> > > thread
> > > > > >> > before
> > > > > >> > > > but
> > > > > >> > > > >>> is
> > > > > >> > > > >>>> not in the KIP).
> > > > > >> > > > >>>>
> > > > > >> > > > >>>> LM3. Related to my previous point, I find that to be
> > the
> > > true
> > > > > >> > > > >>> public-facing
> > > > > >> > > > >>>> change (member ID mandatory at the protocol level),
> but
> > > it's
> > > > > >> only
> > > > > >> > at
> > > > > >> > > > >> the
> > > > > >> > > > >>>> end of the Public interfaces changes, kind of lost
> > among
> > > > > >> details
> > > > > >> > of
> > > > > >> > > > how
> > > > > >> > > > >>>> we're going to do it. Should we rephrase that section
> > > with
> > > > > the
> > > > > >> > > actual
> > > > > >> > > > >>>> change first, and the hows after (ex. Bumping the
> > > version is
> > > > > >> not
> > > > > >> > the
> > > > > >> > > > >>>> public-facing change in this case, it's just the
> > > mechanism to
> > > > > >> > > properly
> > > > > >> > > > >>>> introduce our change)
> > > > > >> > > > >>>>
> > > > > >> > > > >>>> LM4. Regarding the lifetime of the UUID: the KIP
> states
> > > we
> > > > > will
> > > > > >> > > > "Verify
> > > > > >> > > > >>>> that the UUID remains consistent across all
> subsequent
> > > > > >> heartbeats
> > > > > >> > > > >> during
> > > > > >> > > > >>>> the session". What is this "session" referring to
> > here? I
> > > > > would
> > > > > >> > > expect
> > > > > >> > > > >>> that
> > > > > >> > > > >>>> the UUID is associated to a consumer instance
> > (generated
> > > for
> > > > > >> the
> > > > > >> > > > >> consumer
> > > > > >> > > > >>>> the first time it needs to send a HB if it doesn't
> have
> > > the
> > > > > >> UUID
> > > > > >> > > yet.
> > > > > >> > > > >>> From
> > > > > >> > > > >>>> there on, every time it needs to send a "first HB"
> > > again, it
> > > > > >> will
> > > > > >> > > > reuse
> > > > > >> > > > >>> its
> > > > > >> > > > >>>> UUID, is that the intention? Note that we should
> > consider
> > > > > that
> > > > > >> the
> > > > > >> > > > same
> > > > > >> > > > >>>> consumer instance may have many "first heartbeats",
> > > meaning
> > > > > >> > > heartbeats
> > > > > >> > > > >> to
> > > > > >> > > > >>>> join the group when it's not part of it (ex. consumer
> > > > > >> unsubscribe
> > > > > >> > +
> > > > > >> > > > >>>> subscribe, fenced, stale). Is this the intention or
> are
> > > you
> > > > > >> > > > considering
> > > > > >> > > > >>> the
> > > > > >> > > > >>>> lifetime differently? We should clarify it in the
> KIP.
> > > > > >> > > > >>>>
> > > > > >> > > > >>>> Thanks!
> > > > > >> > > > >>>>
> > > > > >> > > > >>>> Lianet
> > > > > >> > > > >>>>
> > > > > >> > > > >>>> On Tue, Aug 27, 2024 at 2:27 AM TengYao Chi <
> > > > > >> kitingiao@gmail.com>
> > > > > >> > > > >> wrote:
> > > > > >> > > > >>>>
> > > > > >> > > > >>>>> Hi everyone,
> > > > > >> > > > >>>>>
> > > > > >> > > > >>>>> I have revised this KIP multiple times based on the
> > > feedback
> > > > > >> from
> > > > > >> > > our
> > > > > >> > > > >>>>> discussions.
> > > > > >> > > > >>>>> I would greatly appreciate it if you could review it
> > > when
> > > > > you
> > > > > >> > have
> > > > > >> > > > >> the
> > > > > >> > > > >>>>> time.
> > > > > >> > > > >>>>> If there are no further comments or suggestions, I
> > plan
> > > to
> > > > > >> > proceed
> > > > > >> > > > >> with
> > > > > >> > > > >>>>> initiating a vote soon.
> > > > > >> > > > >>>>>
> > > > > >> > > > >>>>> Best regards,
> > > > > >> > > > >>>>> TengYao
> > > > > >> > > > >>>>>
> > > > > >> > > > >>>>> TengYao Chi <ki...@gmail.com> 於 2024年8月23日 週五
> > > 下午2:43寫道：
> > > > > >> > > > >>>>>
> > > > > >> > > > >>>>>> Hi Andrew,
> > > > > >> > > > >>>>>> Thank you for your previous feedback and insights.
> > > > > >> > > > >>>>>> Your contributions have added valuable perspectives
> > to
> > > the
> > > > > >> > > > >>> discussions.
> > > > > >> > > > >>>>>> And we also benefit from the comparison of
> different
> > > > > >> solutions.
> > > > > >> > > > >>>>>> I’m also looking forward to seeing an initial
> version
> > > in
> > > > > >> > KIP-932,
> > > > > >> > > > >> as
> > > > > >> > > > >>> it
> > > > > >> > > > >>>>>> will provide a good reference for future
> > > implementations.
> > > > > >> > > > >>>>>>
> > > > > >> > > > >>>>>> Regarding your comment on AS2, I wanted to clarify
> > > that my
> > > > > >> > > > >>>> specification
> > > > > >> > > > >>>>>> references org.apache.kafka.common.Uuid.
> > > > > >> > > > >>>>>> I believe we’re referring to the same class, and it
> > > might
> > > > > >> just
> > > > > >> > be
> > > > > >> > > a
> > > > > >> > > > >>>> small
> > > > > >> > > > >>>>>> oversight due to the busy schedule.
> > > > > >> > > > >>>>>>
> > > > > >> > > > >>>>>> I want to express my gratitude once again for your
> > many
> > > > > >> > insightful
> > > > > >> > > > >>>>>> comments, which have helped the discussion progress
> > > > > smoothly.
> > > > > >> > > > >>>>>>
> > > > > >> > > > >>>>>> Best regards,
> > > > > >> > > > >>>>>> TengYao
> > > > > >> > > > >>>>>>
> > > > > >> > > > >>>>>>
> > > > > >> > > > >>>>>> Andrew Schofield <an...@live.com> 於
> > > 2024年8月22日
> > > > > 週四
> > > > > >> > > > >>>> 下午11:28寫道：
> > > > > >> > > > >>>>>>
> > > > > >> > > > >>>>>>> Hi TengYao,
> > > > > >> > > > >>>>>>> I’ve been reading through the comments and I’m
> happy
> > > that
> > > > > >> the
> > > > > >> > > > >> lobby
> > > > > >> > > > >>>>>>> approach has not gained support.
> > > > > >> > > > >>>>>>>
> > > > > >> > > > >>>>>>> Assuming that this KIP is voted, I will be happy
> to
> > > change
> > > > > >> > > KIP-932
> > > > > >> > > > >>> so
> > > > > >> > > > >>>>>>> that it only supports client-side member ID
> > > generation.
> > > > > >> Because
> > > > > >> > > > >> that
> > > > > >> > > > >>>> KIP
> > > > > >> > > > >>>>>>> is still
> > > > > >> > > > >>>>>>> under development, I can do this in the first
> > version
> > > of
> > > > > >> > > > >>>>>>> ShareGroupHeartbeat.
> > > > > >> > > > >>>>>>>
> > > > > >> > > > >>>>>>> AS2: For the encoding section, I suppose the
> > specific
> > > > > >> encoding
> > > > > >> > > > >> which
> > > > > >> > > > >>>>>>> is used is what org.apache.kafka.utils.Uuid uses.
> > > > > >> > > > >>>>>>>
> > > > > >> > > > >>>>>>> Thanks,
> > > > > >> > > > >>>>>>> Andrew
> > > > > >> > > > >>>>>>>
> > > > > >> > > > >>>>>>>> On 14 Aug 2024, at 17:00, TengYao Chi <
> > > > > kitingiao@gmail.com
> > > > > >> >
> > > > > >> > > > >>> wrote:
> > > > > >> > > > >>>>>>>>
> > > > > >> > > > >>>>>>>> Hello Apoorv,
> > > > > >> > > > >>>>>>>> Thank you for your feedback.
> > > > > >> > > > >>>>>>>> Regarding the questions you raised,
> unfortunately,
> > > this
> > > > > KIP
> > > > > >> > > > >> cannot
> > > > > >> > > > >>>>>>>> guarantee the order of heartbeats. As with many
> > > classic
> > > > > >> > > > >>> distributed
> > > > > >> > > > >>>>>>> system
> > > > > >> > > > >>>>>>>> challenges, what we can do is make our best
> effort
> > to
> > > > > >> ensure
> > > > > >> > > > >> that
> > > > > >> > > > >>>>> there
> > > > > >> > > > >>>>>>> are
> > > > > >> > > > >>>>>>>> no idle members or stale assignments under normal
> > > > > >> > circumstances.
> > > > > >> > > > >>>>>>>>
> > > > > >> > > > >>>>>>>> As for the lobby approach, I’m not a fan of it
> > > because it
> > > > > >> > > > >> requires
> > > > > >> > > > >>>>>>> adding a
> > > > > >> > > > >>>>>>>> mechanism to maintain client state within the
> > > > > >> ConsumerGroup,
> > > > > >> > > > >>> which,
> > > > > >> > > > >>>> in
> > > > > >> > > > >>>>>>> my
> > > > > >> > > > >>>>>>>> view, resembles something like a two-phase
> commit.
> > > This
> > > > > >> would
> > > > > >> > > > >>>>> introduce
> > > > > >> > > > >>>>>>>> more complexity than the proposal in this KIP,
> > which
> > > is
> > > > > >> > > > >> something
> > > > > >> > > > >>> we
> > > > > >> > > > >>>>>>> want
> > > > > >> > > > >>>>>>>> to avoid. KIP-848 aims to simplify the existing
> > > protocol,
> > > > > >> and
> > > > > >> > > > >>> while
> > > > > >> > > > >>>>> the
> > > > > >> > > > >>>>>>>> lobby approach is a good one, I believe it is not
> > the
> > > > > right
> > > > > >> > fit
> > > > > >> > > > >>> for
> > > > > >> > > > >>>>> this
> > > > > >> > > > >>>>>>>> particular situation.
> > > > > >> > > > >>>>>>>>
> > > > > >> > > > >>>>>>>> Best regards,
> > > > > >> > > > >>>>>>>> TengYao
> > > > > >> > > > >>>>>>>>
> > > > > >> > > > >>>>>>>> TengYao Chi <ki...@gmail.com> 於 2024年8月14日
> 週三
> > > > > >> 下午11:45寫道：
> > > > > >> > > > >>>>>>>>
> > > > > >> > > > >>>>>>>>> Hi David,
> > > > > >> > > > >>>>>>>>>
> > > > > >> > > > >>>>>>>>> I really appreciate your review and suggestions.
> > As
> > > I am
> > > > > >> > still
> > > > > >> > > > >>>>> gaining
> > > > > >> > > > >>>>>>>>> experience in writing KIPs, your input has been
> > > > > incredibly
> > > > > >> > > > >>>> helpful. I
> > > > > >> > > > >>>>>>> am
> > > > > >> > > > >>>>>>>>> currently applying your suggestions to the KIP
> and
> > > will
> > > > > >> > > > >> complete
> > > > > >> > > > >>> it
> > > > > >> > > > >>>>> as
> > > > > >> > > > >>>>>>> soon
> > > > > >> > > > >>>>>>>>> as possible.
> > > > > >> > > > >>>>>>>>> Regarding the UUID part, I think we haven’t
> > reached
> > > a
> > > > > >> > > > >> conclusion
> > > > > >> > > > >>>>>>> yet.(So
> > > > > >> > > > >>>>>>>>> far according to this thread)
> > > > > >> > > > >>>>>>>>> However, I will review the current
> implementation
> > > in the
> > > > > >> > Kafka
> > > > > >> > > > >>>> `Uuid`
> > > > > >> > > > >>>>>>>>> class and include a brief specification in the
> > KIP.
> > > > > >> > > > >>>>>>>>>
> > > > > >> > > > >>>>>>>>> Once again, thank you so much for your help.
> > > > > >> > > > >>>>>>>>>
> > > > > >> > > > >>>>>>>>> Best regards,
> > > > > >> > > > >>>>>>>>> TengYao
> > > > > >> > > > >>>>>>>>>
> > > > > >> > > > >>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> 2024年8月14日
> > 週三
> > > > > >> > 下午11:14寫道：
> > > > > >> > > > >>>>>>>>>
> > > > > >> > > > >>>>>>>>>> hi Apoorv
> > > > > >> > > > >>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>> As the memberId is now known to the client,
> and
> > > client
> > > > > >> > might
> > > > > >> > > > >>> send
> > > > > >> > > > >>>>> the
> > > > > >> > > > >>>>>>>>>> leave
> > > > > >> > > > >>>>>>>>>> group heartbeat on shutdown prior to receiving
> > the
> > > > > >> initial
> > > > > >> > > > >>>> heartbeat
> > > > > >> > > > >>>>>>>>>> response. If that's true then how do we
> guarantee
> > > that
> > > > > >> the 2
> > > > > >> > > > >>>>> requests
> > > > > >> > > > >>>>>>> to
> > > > > >> > > > >>>>>>>>>> join and leave will be processed in order,
> which
> > > could
> > > > > >> still
> > > > > >> > > > >>> leave
> > > > > >> > > > >>>>>>> stale
> > > > > >> > > > >>>>>>>>>> members or throw unknown member id exceptions?
> > > > > >> > > > >>>>>>>>>>
> > > > > >> > > > >>>>>>>>>> This is definitely a good question. the short
> > > answer:
> > > > > no
> > > > > >> > > > >>> guarantee
> > > > > >> > > > >>>>> but
> > > > > >> > > > >>>>>>>>>> best
> > > > > >> > > > >>>>>>>>>> efforts
> > > > > >> > > > >>>>>>>>>>
> > > > > >> > > > >>>>>>>>>> Please notice the root cause is "we have no
> > enough
> > > time
> > > > > >> to
> > > > > >> > > > >> wait
> > > > > >> > > > >>>>>>> member id
> > > > > >> > > > >>>>>>>>>> (response) when closing consumer". Sadly, we
> can'
> > > > > >> guarantee
> > > > > >> > > > >> the
> > > > > >> > > > >>>>>>> request
> > > > > >> > > > >>>>>>>>>> order due to the same reason.
> > > > > >> > > > >>>>>>>>>>
> > > > > >> > > > >>>>>>>>>> However, in contrast to previous behavior,
> there
> > > is one
> > > > > >> big
> > > > > >> > > > >>>> benefit
> > > > > >> > > > >>>>>>> of new
> > > > > >> > > > >>>>>>>>>> approach - we can try STONITH because we know
> the
> > > > > member
> > > > > >> id
> > > > > >> > > > >>>>>>>>>>
> > > > > >> > > > >>>>>>>>>> Best,
> > > > > >> > > > >>>>>>>>>> Chia-Ping
> > > > > >> > > > >>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>
> > > > > >> > > > >>>>>>>>>> Apoorv Mittal <ap...@gmail.com> 於
> > > 2024年8月14日
> > > > > 週三
> > > > > >> > > > >>>> 下午8:55寫道：
> > > > > >> > > > >>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>> Hi TengYao,
> > > > > >> > > > >>>>>>>>>>> Thanks for the KIP. Continuing on the point
> > which
> > > > > Andrew
> > > > > >> > > > >>>> mentioned
> > > > > >> > > > >>>>> as
> > > > > >> > > > >>>>>>>>>> AS1.
> > > > > >> > > > >>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>> As the memberId is now known to the client,
> and
> > > client
> > > > > >> > might
> > > > > >> > > > >>> send
> > > > > >> > > > >>>>> the
> > > > > >> > > > >>>>>>>>>> leave
> > > > > >> > > > >>>>>>>>>>> group heartbeat on shutdown prior to receiving
> > the
> > > > > >> initial
> > > > > >> > > > >>>>> heartbeat
> > > > > >> > > > >>>>>>>>>>> response. If that's true then how do we
> > guarantee
> > > that
> > > > > >> the
> > > > > >> > 2
> > > > > >> > > > >>>>>>> requests to
> > > > > >> > > > >>>>>>>>>>> join and leave will be processed in order,
> which
> > > could
> > > > > >> > still
> > > > > >> > > > >>>> leave
> > > > > >> > > > >>>>>>> stale
> > > > > >> > > > >>>>>>>>>>> members or throw unknown member id exceptions?
> > > > > >> > > > >>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>> Though the client side member id generation is
> > > helpful
> > > > > >> > which
> > > > > >> > > > >>> will
> > > > > >> > > > >>>>>>>>>> represent
> > > > > >> > > > >>>>>>>>>>> the same group perspective as from client and
> > > broker's
> > > > > >> end.
> > > > > >> > > > >>> But I
> > > > > >> > > > >>>>>>> think
> > > > > >> > > > >>>>>>>>>> the
> > > > > >> > > > >>>>>>>>>>> major concern we want to solve here is Stale
> > > Partition
> > > > > >> > > > >>>> Assignments
> > > > > >> > > > >>>>>>> which
> > > > > >> > > > >>>>>>>>>>> might still exist with the new approach. I am
> > > leaning
> > > > > >> > towards
> > > > > >> > > > >>> the
> > > > > >> > > > >>>>>>>>>>> suggestion mentioned by Andrew where partition
> > > > > >> assignment
> > > > > >> > > > >>>> triggers
> > > > > >> > > > >>>>> on
> > > > > >> > > > >>>>>>>>>>> subsequent heartbeat when client acknowledges
> > the
> > > > > >> initial
> > > > > >> > > > >>>>> heartbeat,
> > > > > >> > > > >>>>>>>>>>> delayed partition assignment.
> > > > > >> > > > >>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>> Though on a separate note, I have a different
> > > > > question.
> > > > > >> > What
> > > > > >> > > > >>>>> happens
> > > > > >> > > > >>>>>>>>>> when
> > > > > >> > > > >>>>>>>>>>> there is an issue with the client which sends
> > the
> > > > > >> initial
> > > > > >> > > > >>>> heartbeat
> > > > > >> > > > >>>>>>>>>> without
> > > > > >> > > > >>>>>>>>>>> memberId, then crashes and restarts? I think
> we
> > > must
> > > > > be
> > > > > >> > > > >>>>> re-triggering
> > > > > >> > > > >>>>>>>>>>> assignments and expiring members only after
> the
> > > > > >> heartbeat
> > > > > >> > > > >>> session
> > > > > >> > > > >>>>>>>>>> timeout?
> > > > > >> > > > >>>>>>>>>>> If that's true then shall delayed partition
> > > assignment
> > > > > >> can
> > > > > >> > > > >> help
> > > > > >> > > > >>>>>>> benefit
> > > > > >> > > > >>>>>>>>>> us
> > > > > >> > > > >>>>>>>>>>> from this situation as well?
> > > > > >> > > > >>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>> Regards,
> > > > > >> > > > >>>>>>>>>>> Apoorv Mittal
> > > > > >> > > > >>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>> On Wed, Aug 14, 2024 at 12:51 PM David Jacot
> > > > > >> > > > >>>>>>>>>> <dj...@confluent.io.invalid>
> > > > > >> > > > >>>>>>>>>>> wrote:
> > > > > >> > > > >>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>> Hi Andrew,
> > > > > >> > > > >>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>> Personally, I don't like the lobby approach.
> It
> > > makes
> > > > > >> > things
> > > > > >> > > > >>>> more
> > > > > >> > > > >>>>>>>>>>>> complicated and it would require changing the
> > > records
> > > > > >> on
> > > > > >> > the
> > > > > >> > > > >>>>> server
> > > > > >> > > > >>>>>>>>>> too.
> > > > > >> > > > >>>>>>>>>>>> This is why I initially suggested the
> rejected
> > > > > >> alternative
> > > > > >> > > > >> #2
> > > > > >> > > > >>>>> which
> > > > > >> > > > >>>>>>> is
> > > > > >> > > > >>>>>>>>>>>> pretty close but also not perfect.
> > > > > >> > > > >>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>> I'd like to clarify one thing. The
> > > > > >> ConsumerGroupHeartbeat
> > > > > >> > > > >> API
> > > > > >> > > > >>>>>>> already
> > > > > >> > > > >>>>>>>>>>>> supports generating the member id on the
> client
> > > so we
> > > > > >> > don't
> > > > > >> > > > >>> need
> > > > > >> > > > >>>>> any
> > > > > >> > > > >>>>>>>>>>>> conditional logic on the client side. This is
> > > > > actually
> > > > > >> > what
> > > > > >> > > > >> we
> > > > > >> > > > >>>>>>> wanted
> > > > > >> > > > >>>>>>>>>> to
> > > > > >> > > > >>>>>>>>>>> do
> > > > > >> > > > >>>>>>>>>>>> in the first place but the idea got pushed
> back
> > > by
> > > > > >> Magnus
> > > > > >> > > > >> back
> > > > > >> > > > >>>>> then
> > > > > >> > > > >>>>>>>>>>> because
> > > > > >> > > > >>>>>>>>>>>> generating uuid from librdkafka required a
> new
> > > > > >> dependency.
> > > > > >> > > > >> It
> > > > > >> > > > >>>>> turns
> > > > > >> > > > >>>>>>>>>> out
> > > > > >> > > > >>>>>>>>>>>> that librdkafka has that dependency today. In
> > > > > >> retrospect,
> > > > > >> > we
> > > > > >> > > > >>>>> should
> > > > > >> > > > >>>>>>>>>> have
> > > > > >> > > > >>>>>>>>>>>> pushed back on this. Long story short, we can
> > > just do
> > > > > >> it.
> > > > > >> > > > >> The
> > > > > >> > > > >>>>>>>>>> proposal in
> > > > > >> > > > >>>>>>>>>>>> this KIP is to make the member id required in
> > > future
> > > > > >> > > > >> versions.
> > > > > >> > > > >>>> We
> > > > > >> > > > >>>>>>>>>> could
> > > > > >> > > > >>>>>>>>>>>> also decide not to do it and to keep
> supporting
> > > both
> > > > > >> > > > >>>> approaches. I
> > > > > >> > > > >>>>>>>>>> would
> > > > > >> > > > >>>>>>>>>>>> also be fine with this.
> > > > > >> > > > >>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>> Best,
> > > > > >> > > > >>>>>>>>>>>> David
> > > > > >> > > > >>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>> On Wed, Aug 14, 2024 at 12:30 PM Andrew
> > > Schofield <
> > > > > >> > > > >>>>>>>>>>>> andrew_schofield@live.com>
> > > > > >> > > > >>>>>>>>>>>> wrote:
> > > > > >> > > > >>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>> Hi TengYao,
> > > > > >> > > > >>>>>>>>>>>>> Thanks for your response. I’ll have just one
> > > more
> > > > > try
> > > > > >> to
> > > > > >> > > > >>>>> persuade.
> > > > > >> > > > >>>>>>>>>>>>> I feel that I will need to follow the
> approach
> > > with
> > > > > >> > KIP-932
> > > > > >> > > > >>>> when
> > > > > >> > > > >>>>>>>>>> we’ve
> > > > > >> > > > >>>>>>>>>>>>> made a decision, so I do have more than a
> > > passing
> > > > > >> > interest
> > > > > >> > > > >> in
> > > > > >> > > > >>>>> this.
> > > > > >> > > > >>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>> A group member in the lobby is in the group,
> > > but it
> > > > > >> does
> > > > > >> > > > >> not
> > > > > >> > > > >>>> have
> > > > > >> > > > >>>>>>>>>> any
> > > > > >> > > > >>>>>>>>>>>>> assignments. A member of a consumer group
> can
> > > have
> > > > > no
> > > > > >> > > > >>> assigned
> > > > > >> > > > >>>>>>>>>>>>> partitions (such as 5 CG members subscribed
> > to a
> > > > > topic
> > > > > >> > > > >> with 4
> > > > > >> > > > >>>>>>>>>>>> partitions),
> > > > > >> > > > >>>>>>>>>>>>> so it’s a situation that consumer group
> > members
> > > > > >> already
> > > > > >> > > > >>> expect.
> > > > > >> > > > >>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>> One of Kafka’s strengths is the way that we
> > > handle
> > > > > API
> > > > > >> > > > >>>>> versioning.
> > > > > >> > > > >>>>>>>>>>>>> But, there is a cost - the behaviour is
> > > different
> > > > > >> > depending
> > > > > >> > > > >>> on
> > > > > >> > > > >>>>> the
> > > > > >> > > > >>>>>>>>>> RPC
> > > > > >> > > > >>>>>>>>>>>>> version. KIP-848 is on the cusp of
> completion,
> > > but
> > > > > >> we’re
> > > > > >> > > > >>>> already
> > > > > >> > > > >>>>>>>>>> adding
> > > > > >> > > > >>>>>>>>>>>>> conditional logic for v0/v1 for
> > > > > >> ConsumerGroupHeartbeat.
> > > > > >> > > > >>> That’s
> > > > > >> > > > >>>> a
> > > > > >> > > > >>>>>>>>>> pity.
> > > > > >> > > > >>>>>>>>>>>>> Only a minor issue, but it’s unfortunate.
> > > > > >> > > > >>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>> Thanks,
> > > > > >> > > > >>>>>>>>>>>>> Andrew
> > > > > >> > > > >>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>> On 14 Aug 2024, at 08:47, TengYao Chi <
> > > > > >> > > > >> kitingiao@gmail.com>
> > > > > >> > > > >>>>>>>>>> wrote:
> > > > > >> > > > >>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>> Hello Andrew
> > > > > >> > > > >>>>>>>>>>>>>> Thank you for your thoughtful suggestions
> and
> > > > > getting
> > > > > >> > the
> > > > > >> > > > >>>>>>>>>> discussion
> > > > > >> > > > >>>>>>>>>>>>> going.
> > > > > >> > > > >>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>> To AS1:
> > > > > >> > > > >>>>>>>>>>>>>> In the current scenario where the server
> > > generates
> > > > > >> the
> > > > > >> > > > >> UUID,
> > > > > >> > > > >>>> if
> > > > > >> > > > >>>>>>>>>> the
> > > > > >> > > > >>>>>>>>>>>>> client
> > > > > >> > > > >>>>>>>>>>>>>> shuts down before receiving the memberId
> > > generated
> > > > > by
> > > > > >> > the
> > > > > >> > > > >> GC
> > > > > >> > > > >>>>>>>>>>>> (regardless
> > > > > >> > > > >>>>>>>>>>>>> of
> > > > > >> > > > >>>>>>>>>>>>>> whether it’s a graceful shutdown or not),
> the
> > > GC
> > > > > will
> > > > > >> > > > >> still
> > > > > >> > > > >>>> have
> > > > > >> > > > >>>>>>>>>> to
> > > > > >> > > > >>>>>>>>>>>> wait
> > > > > >> > > > >>>>>>>>>>>>>> for the heartbeat timeout because the
> client
> > > > > doesn’t
> > > > > >> > know
> > > > > >> > > > >>> its
> > > > > >> > > > >>>>>>>>>>> memberId.
> > > > > >> > > > >>>>>>>>>>>>>> This KIP indeed cannot completely resolve
> the
> > > > > >> > idempotency
> > > > > >> > > > >>>> issue,
> > > > > >> > > > >>>>>>>>>> but
> > > > > >> > > > >>>>>>>>>>> it
> > > > > >> > > > >>>>>>>>>>>>> can
> > > > > >> > > > >>>>>>>>>>>>>> better handle shutdown scenarios under
> normal
> > > > > >> > > > >> circumstances
> > > > > >> > > > >>>>>>>>>> because
> > > > > >> > > > >>>>>>>>>>> the
> > > > > >> > > > >>>>>>>>>>>>>> client always knows its memberId. Even if
> the
> > > > > client
> > > > > >> > shuts
> > > > > >> > > > >>>> down
> > > > > >> > > > >>>>>>>>>>>>> immediately
> > > > > >> > > > >>>>>>>>>>>>>> after the initial heartbeat, as long as it
> > > > > performs a
> > > > > >> > > > >>> graceful
> > > > > >> > > > >>>>>>>>>>> shutdown
> > > > > >> > > > >>>>>>>>>>>>> and
> > > > > >> > > > >>>>>>>>>>>>>> sends a leave heartbeat, the GC can manage
> > the
> > > > > >> situation
> > > > > >> > > > >> and
> > > > > >> > > > >>>>>>>>>> remove
> > > > > >> > > > >>>>>>>>>>> the
> > > > > >> > > > >>>>>>>>>>>>>> member. Therefore, the goal of this KIP is
> to
> > > > > address
> > > > > >> > the
> > > > > >> > > > >>>> issue
> > > > > >> > > > >>>>>>>>>> where
> > > > > >> > > > >>>>>>>>>>>> the
> > > > > >> > > > >>>>>>>>>>>>>> GC has to wait for the heartbeat timeout
> due
> > > to the
> > > > > >> > client
> > > > > >> > > > >>>>> leaving
> > > > > >> > > > >>>>>>>>>>>>> without
> > > > > >> > > > >>>>>>>>>>>>>> knowing its memberId, which leads to
> reduced
> > > > > >> throughput
> > > > > >> > > > >> and
> > > > > >> > > > >>>>>>>>>> limited
> > > > > >> > > > >>>>>>>>>>>>>> scalability.
> > > > > >> > > > >>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>> The solution you suggest has also been
> > > proposed by
> > > > > >> > David.
> > > > > >> > > > >>> The
> > > > > >> > > > >>>>>>>>>> concern
> > > > > >> > > > >>>>>>>>>>>>> with
> > > > > >> > > > >>>>>>>>>>>>>> this approach is that it introduces
> > additional
> > > > > >> > complexity
> > > > > >> > > > >>> for
> > > > > >> > > > >>>>>>>>>>>>>> compatibility, as the new server would not
> > > > > >> immediately
> > > > > >> > add
> > > > > >> > > > >>> the
> > > > > >> > > > >>>>>>>>>> member
> > > > > >> > > > >>>>>>>>>>>> to
> > > > > >> > > > >>>>>>>>>>>>>> the group, while the old server would. This
> > > > > requires
> > > > > >> > > > >> clients
> > > > > >> > > > >>>> to
> > > > > >> > > > >>>>>>>>>>>>>> differentiate whether their memberId has
> been
> > > added
> > > > > >> to
> > > > > >> > the
> > > > > >> > > > >>>> group
> > > > > >> > > > >>>>>>>>>> or
> > > > > >> > > > >>>>>>>>>>>> not,
> > > > > >> > > > >>>>>>>>>>>>>> which could result in unexpected logs.
> > > > > >> > > > >>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>> Best Regards,
> > > > > >> > > > >>>>>>>>>>>>>> TengYao
> > > > > >> > > > >>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>> Andrew Schofield <
> andrew_schofield@live.com>
> > 於
> > > > > >> > 2024年8月14日
> > > > > >> > > > >>> 週三
> > > > > >> > > > >>>>>>>>>>>> 上午12:29寫道：
> > > > > >> > > > >>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>>> Hi TengYao,
> > > > > >> > > > >>>>>>>>>>>>>>> Thanks for the KIP. I wonder if there’s a
> > > > > different
> > > > > >> way
> > > > > >> > > > >> to
> > > > > >> > > > >>>>> close
> > > > > >> > > > >>>>>>>>>>> what
> > > > > >> > > > >>>>>>>>>>>>>>> is quite a small window.
> > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>>> AS1: It is true that the initial heartbeat
> > is
> > > not
> > > > > >> > > > >>> idempotent,
> > > > > >> > > > >>>>> but
> > > > > >> > > > >>>>>>>>>>> this
> > > > > >> > > > >>>>>>>>>>>>>>> remains
> > > > > >> > > > >>>>>>>>>>>>>>> true with this KIP. It’s just differently
> > not
> > > > > >> > idempotent.
> > > > > >> > > > >>> If
> > > > > >> > > > >>>>> the
> > > > > >> > > > >>>>>>>>>>>> client
> > > > > >> > > > >>>>>>>>>>>>>>> makes its
> > > > > >> > > > >>>>>>>>>>>>>>> own member ID, sends a request and dies,
> the
> > > GC
> > > > > will
> > > > > >> > > > >> still
> > > > > >> > > > >>>> have
> > > > > >> > > > >>>>>>>>>>> added
> > > > > >> > > > >>>>>>>>>>>>>>> the member to the group and it will hang
> > > around
> > > > > >> until
> > > > > >> > the
> > > > > >> > > > >>>>> session
> > > > > >> > > > >>>>>>>>>>>>> expires.
> > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>>> I wonder if the GC could still generate
> the
> > > member
> > > > > >> ID
> > > > > >> > in
> > > > > >> > > > >>>>>>>>>> response to
> > > > > >> > > > >>>>>>>>>>>> the
> > > > > >> > > > >>>>>>>>>>>>>>> first
> > > > > >> > > > >>>>>>>>>>>>>>> heartbeat, and put the member in a special
> > > PENDING
> > > > > >> > state
> > > > > >> > > > >>> with
> > > > > >> > > > >>>>> no
> > > > > >> > > > >>>>>>>>>>>>>>> assignments until the client sends the
> next
> > > > > >> heartbeat,
> > > > > >> > > > >> thus
> > > > > >> > > > >>>>>>>>>>> confirming
> > > > > >> > > > >>>>>>>>>>>>> it
> > > > > >> > > > >>>>>>>>>>>>>>> has received the member ID. This would not
> > be
> > > a
> > > > > >> > protocol
> > > > > >> > > > >>>> change
> > > > > >> > > > >>>>>>>>>> at
> > > > > >> > > > >>>>>>>>>>>> all,
> > > > > >> > > > >>>>>>>>>>>>>>> just
> > > > > >> > > > >>>>>>>>>>>>>>> a change to the GC to keep a new member in
> > the
> > > > > lobby
> > > > > >> > > > >> until
> > > > > >> > > > >>>> it’s
> > > > > >> > > > >>>>>>>>>>>>> comfirmed
> > > > > >> > > > >>>>>>>>>>>>>>> it knows its member ID.
> > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>>> Thanks,
> > > > > >> > > > >>>>>>>>>>>>>>> Andrew
> > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>>>> On 13 Aug 2024, at 15:59, TengYao Chi <
> > > > > >> > > > >>> kitingiao@gmail.com>
> > > > > >> > > > >>>>>>>>>> wrote:
> > > > > >> > > > >>>>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>>>> Hi Chia-Ping,
> > > > > >> > > > >>>>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>>>> Thanks for review and suggestions.
> > > > > >> > > > >>>>>>>>>>>>>>>> I have updated the content of KIP
> > > accordingly.
> > > > > >> > > > >>>>>>>>>>>>>>>> Please take a look.
> > > > > >> > > > >>>>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>>>> Best regards,
> > > > > >> > > > >>>>>>>>>>>>>>>> TengYao
> > > > > >> > > > >>>>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > > > > 2024年8月13日
> > > > > >> 週二
> > > > > >> > > > >>>>> 下午9:45寫道：
> > > > > >> > > > >>>>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>>>>> hi TengYao
> > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>>>>> thanks for this KIP.
> > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>>>>> 1) could you please describe the
> > > before/after
> > > > > >> > behavior
> > > > > >> > > > >> in
> > > > > >> > > > >>>> the
> > > > > >> > > > >>>>>>>>>>>>> "Proposed
> > > > > >> > > > >>>>>>>>>>>>>>>>> Changes" section? IIRC, current RPC
> allows
> > > HB
> > > > > >> having
> > > > > >> > > > >>> member
> > > > > >> > > > >>>>> id
> > > > > >> > > > >>>>>>>>>>>>>>> generated by
> > > > > >> > > > >>>>>>>>>>>>>>>>> client, right? If HB has no member ID,
> > > server
> > > > > will
> > > > > >> > > > >>> generate
> > > > > >> > > > >>>>> one
> > > > > >> > > > >>>>>>>>>>> and
> > > > > >> > > > >>>>>>>>>>>>> then
> > > > > >> > > > >>>>>>>>>>>>>>>>> return. The new behavior will enforce HB
> > > "must"
> > > > > >> have
> > > > > >> > > > >>> member
> > > > > >> > > > >>>>> ID.
> > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>>>>> 2) could you please write the version
> > number
> > > > > >> > explicitly
> > > > > >> > > > >>> in
> > > > > >> > > > >>>>> the
> > > > > >> > > > >>>>>>>>>> KIP
> > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>>>>> 3) how new client code handle the old
> HB?
> > > Does
> > > > > it
> > > > > >> > > > >> always
> > > > > >> > > > >>>>>>>>>> generate
> > > > > >> > > > >>>>>>>>>>>>> member
> > > > > >> > > > >>>>>>>>>>>>>>>>> ID on client-side even though that is
> not
> > > > > >> restricted?
> > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>>>>> Best,
> > > > > >> > > > >>>>>>>>>>>>>>>>> Chia-Ping
> > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>>>>> On 2024/08/13 06:20:42 TengYao Chi
> wrote:
> > > > > >> > > > >>>>>>>>>>>>>>>>>> Hello everyone,
> > > > > >> > > > >>>>>>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>>>>>> I would like to start a discussion
> thread
> > > on
> > > > > >> > KIP-1082,
> > > > > >> > > > >>>> which
> > > > > >> > > > >>>>>>>>>>>> proposes
> > > > > >> > > > >>>>>>>>>>>>>>>>>> enabling id generation for clients over
> > the
> > > > > >> > > > >>>>>>>>>>> ConsumerGroupHeartbeat
> > > > > >> > > > >>>>>>>>>>>>> RPC.
> > > > > >> > > > >>>>>>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>>>>>> Here is the KIP Link: KIP-1082
> > > > > >> > > > >>>>>>>>>>>>>>>>>> <
> > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>
> > > > > >> > > > >>>>>>>
> > > > > >> > > > >>>>>
> > > > > >> > > > >>>>
> > > > > >> > > > >>>
> > > > > >> > > > >>
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
> > > > > >> > > > >>>>>>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>>>>>> Please take a look and let me know what
> > you
> > > > > >> think,
> > > > > >> > > > >> and I
> > > > > >> > > > >>>>> would
> > > > > >> > > > >>>>>>>>>>>>>>> appreciate
> > > > > >> > > > >>>>>>>>>>>>>>>>>> any suggestions and feedback.
> > > > > >> > > > >>>>>>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>>>>>> Best regards,
> > > > > >> > > > >>>>>>>>>>>>>>>>>> TengYao
> > > > > >> > > > >>>>>>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>>
> > > > > >> > > > >>>>>>>>>>
> > > > > >> > > > >>>>>>>>>
> > > > > >> > > > >>>>>>>
> > > > > >> > > > >>>>>>>
> > > > > >> > > > >>>>>
> > > > > >> > > > >>>>
> > > > > >> > > > >>>
> > > > > >> > > > >>
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[Chia-Ping Tsai <ch...@gmail.com>]

wait a minute. There is a description about fenced error in the KIP-848

> Checks whether the member epoch matches the member epoch in its current
assignment. FENCED_MEMBER_EPOCH is returned otherwise. The member is also
removed from the group.

I failed to find the code of implementing "The member is also removed from
the group.". I will appreciate it if someone can share the code link to me.

Let's assume it will be implemented eventually, and I feel that will cause
a serious issue as the consumers having the same member id will remove the
existing  member and then create itself member?

> Once the client receives this exception, it should consider the exception
as a serious error and stop the process, and the user will need to assign a
new memberId.

Given that member id is the incarnation of consumer, it is hard to detect
the scenario of "two consumers have the same member id" (it may be caused
by the lost response). I guess the best approach is the later -created
consumer should be idle with fenced error, and we should add this to the
KIP (maybe we should revise the error message carried by
`FencedMemberEpochException`). BUT, we need to rethink the approach if
above description "The member is also removed from the group" is true

Best,
Chia-Ping



TengYao Chi <ki...@gmail.com> 於 2024年9月20日 週五 下午8:17寫道：

> Hi Chia-Ping,
>
> Thanks for pointing out this issue.
>
> I’m thinking that maybe we might need to define a new Exception to handle
> this scenario.
> Once the client receives this exception, it should consider the exception
> as a serious error and stop the process, and the user will need to assign a
> new memberId.
> What do you think?
>
> Best Regards,
> TengYao
>
> Chia-Ping Tsai <ch...@apache.org> 於 2024年9月20日 週五 下午7:00寫道：
>
> > > This part is not clear either. It basically says that if a member joins
> > with an existing member id but a different epoch, it will be fenced. Then
> > it must rejoin with the same member id and epoch zero. This is already
> the
> > current behavior and it does not help with detecting duplicates, right?
> >
> > The duplicates issue is interesting. The member id is generated on
> > server-side by UUID before, so it seems to me the fenced member epoch
> > happens only if the client miss the response with bumped epoch, and this
> > case is recoverable.
> >
> > however, v1 brings a different fenced epoch scenario. Users, now, have
> > responsibility to generate unique member id, hence clients may encounter
> > infinite fenced error if there are >1 clients having same member id due
> to
> > config error (or other engineer error)
> >
> > Maybe we can highlight this scenario if we start to requires clients to
> > generate unique member id.
> >
> > Best,
> > Chia-Ping
> >
> > On 2024/09/19 18:36:48 David Jacot wrote:
> > > Hi,
> > >
> > > Thanks for the update. I have a few nits:
> > >
> > > > If the member ID is null or empty, the server will reject the request
> > > with an InvalidRequestException.
> > > We should clarify that this should only apply to version >= 1.
> > >
> > > > The consumer instance must generate a member ID, and this ID should
> > > remain consistent for the duration of the consumer's session. Here, a
> > > "session" is defined as the period from the consumer's first heartbeat
> > > until it leaves the group, either through a graceful shutdown, a
> > heartbeat
> > > timeout, or the process stopping or dying. The consumer instance should
> > > reuse the same member ID for all heartbeats and rejoin attempts to
> > maintain
> > > continuity within the group.
> > >
> > > This part is not clear to me. When the member leaves the group, it
> should
> > > not reset the member id. I would rather say that the member must
> generate
> > > its member id when it starts and it must keep it until the process
> stops.
> > > It is basically an incarnation of the process.
> > >
> > > > If a conflict arises where the member ID generated by the client is
> > > detected to be a duplicate within the same group (for example, the same
> > > member ID is associated with another active member in the group), the
> > > server will handle this by comparing the memberEpoch values of the
> > > conflicting members. The member with the lower memberEpoch is
> considered
> > > outdated and will be fenced off by the server. When this occurs, the
> > server
> > > responds with a FENCED_MEMBER_EPOCH error to the client, signaling it
> to
> > > rejoin the group with the same member ID while resetting the
> memberEpoch
> > to
> > > zero. This ensures that the client properly resynchronizes and
> maintains
> > > the continuity and consistency of the group membership.
> > >
> > > This part is not clear either. It basically says that if a member joins
> > > with an existing member id but a different epoch, it will be fenced.
> Then
> > > it must rejoin with the same member id and epoch zero. This is already
> > the
> > > current behavior and it does not help with detecting duplicates, right?
> > > Should we just remove the paragraph?
> > >
> > > > A member ID mismatch occurs within a session: If the server detects a
> > > mismatch between the provided member ID and the expected member ID for
> an
> > > ongoing session, it should return a UNKNOWN_MEMBER_ID  error.
> > >
> > > How could we detect a mismatch between the provided and the expected
> > member
> > > id? My understanding is that we can only know whether the provided
> member
> > > id exists or not. This is already implemented.
> > >
> > > Thanks,
> > > David
> > >
> > > On Sat, Sep 14, 2024 at 9:31 AM TengYao Chi <ki...@gmail.com>
> wrote:
> > >
> > > > Hello everyone,
> > > >
> > > > Since this KIP has been fully discussed, I will initiate a vote for
> it
> > next
> > > > Monday.
> > > > Thank you and have a nice weekend.
> > > >
> > > > Best regards,
> > > > TengYao
> > > >
> > > > TengYao Chi <ki...@gmail.com> 於 2024年9月5日 週四 下午2:19寫道：
> > > >
> > > > > Hello everyone,
> > > > >
> > > > > KT2: It looks like everyone who has expressed an opinion supports
> the
> > > > > second option: “Document a recommendation for clients to use UUIDs
> as
> > > > > member IDs, without strictly enforcing it.”
> > > > > I have updated the KIP accordingly.
> > > > > Please take a look, and let me know if you have any thoughts or
> > feedback.
> > > > >
> > > > > Thank you!
> > > > >
> > > > > Best regards,
> > > > > TengYao
> > > > >
> > > > > Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月30日 週五 下午9:56寫道：
> > > > >
> > > > >> hi TengYao
> > > > >>
> > > > >> KT2: +1 to second approach
> > > > >>
> > > > >> Best,
> > > > >> Chia-Ping
> > > > >>
> > > > >>
> > > > >> David Jacot <dj...@confluent.io.invalid> 於 2024年8月30日 週五
> 下午9:15寫道：
> > > > >>
> > > > >> > Hi TengYao,
> > > > >> >
> > > > >> > KT2: I don't think that we can realistically validate the uuid
> on
> > the
> > > > >> > server. It is basically a string of chars. So I lean towards
> > having a
> > > > >> good
> > > > >> > recommendation in the KIP and in the document of the field in
> the
> > > > RPC's
> > > > >> > definition.
> > > > >> >
> > > > >> > Best,
> > > > >> > David
> > > > >> >
> > > > >> > On Fri, Aug 30, 2024 at 3:02 PM TengYao Chi <
> kitingiao@gmail.com>
> > > > >> wrote:
> > > > >> >
> > > > >> > > Hello Kirk !
> > > > >> > >
> > > > >> > > Thank you for your comments !
> > > > >> > >
> > > > >> > > KT1: Yes, you are correct. The issue is not unique to the
> > initial
> > > > >> > > heartbeat; there can always be cases where the broker might
> lose
> > > > >> > connection
> > > > >> > > with a member.
> > > > >> > >
> > > > >> > > KT2: Currently, if the client doesn't have a member ID and the
> > > > >> > memberEpoch
> > > > >> > > equals 0, the coordinator will generate a UUID as the member
> ID
> > for
> > > > >> the
> > > > >> > > client. However, at the RPC level, the member ID is sent as a
> > > > literal
> > > > >> > > string, meaning there are no restrictions on the format at
> this
> > > > level.
> > > > >> > > This also reminds me that we haven't reached a final
> conclusion
> > on
> > > > >> how to
> > > > >> > > enforce the use of UUIDs.
> > > > >> > > From our previous discussions, I recall two possible
> approaches:
> > > > >> > > The first is to validate the UUID on the server side, and if
> > it's
> > > > not
> > > > >> > > valid, throw an exception to the client.
> > > > >> > > The second is to document a recommendation for clients to use
> > UUIDs
> > > > as
> > > > >> > > member IDs, without strictly enforcing it.
> > > > >> > > I think it's time to decide on the approach we want to take.
> > > > >> > >
> > > > >> > > KT3: Yes, "session" can be considered synonymous with
> > "membership"
> > > > in
> > > > >> > this
> > > > >> > > context.
> > > > >> > >
> > > > >> > > KT4: Thank you for pointing that out. I will update the
> wording
> > to
> > > > >> > > specifically say this behavior is for consumers.
> > > > >> > >
> > > > >> > > Thanks again for your comments.
> > > > >> > >
> > > > >> > > Best regards,
> > > > >> > > TengYao
> > > > >> > >
> > > > >> > > Kirk True <ki...@kirktrue.pro> 於 2024年8月30日 週五 上午12:39寫道：
> > > > >> > >
> > > > >> > > > Hi TengYao!
> > > > >> > > >
> > > > >> > > > Sorry for being late to the discussion...
> > > > >> > > >
> > > > >> > > > After reading the thread and then the KIP, I had a few
> > > > >> > > questions/comments:
> > > > >> > > >
> > > > >> > > > KT1: In Motivation, it states: "This scenario can result in
> > the
> > > > >> broker
> > > > >> > > > registering a new member for which it will never receive a
> > proper
> > > > >> leave
> > > > >> > > > request.” Just to be clear, the broker will always have
> cases
> > > > where
> > > > >> it
> > > > >> > > > might lose connection with a member. That’s not unique to
> the
> > > > >> initial
> > > > >> > > > heartbeat, right?
> > > > >> > > >
> > > > >> > > > KT2: There was a bit of back and forth about format of the
> > member
> > > > >> ID.
> > > > >> > > From
> > > > >> > > > what I gathered in the thread, the member ID is still
> defined
> > in
> > > > the
> > > > >> > RPC
> > > > >> > > as
> > > > >> > > > a string and not a UUID, right? The KIP states that the
> > “client
> > > > must
> > > > >> > > > generate a UUID as the member ID” and that the “server will
> > > > validate
> > > > >> > > that a
> > > > >> > > > valid UUID is provided.” Is that a change for the server, or
> > is it
> > > > >> > > already
> > > > >> > > > enforced as a UUID?
> > > > >> > > >
> > > > >> > > > KT3: Lianet mentioned some confusion over the use of the
> word
> > > > >> > “session.”
> > > > >> > > > Isn’t “session” synonymous with “membership?”
> > > > >> > > >
> > > > >> > > > KT4: Under “Member ID Lifecycle,” it states: "The client
> > should
> > > > >> reuse
> > > > >> > the
> > > > >> > > > same UUID as the member ID for all heartbeats and rejoin
> > attempts
> > > > to
> > > > >> > > > maintain continuity within the group.” Could we change the
> > first
> > > > >> part
> > > > >> > of
> > > > >> > > > that to “The Consumer instance should…” We do have lifetimes
> > that
> > > > >> > extend
> > > > >> > > > past the lifetime of a client instance (such as the
> > transaction
> > > > ID).
> > > > >> > > >
> > > > >> > > > Thanks,
> > > > >> > > > Kirk
> > > > >> > > >
> > > > >> > > > > On Aug 29, 2024, at 1:28 AM, TengYao Chi <
> > kitingiao@gmail.com>
> > > > >> > wrote:
> > > > >> > > > >
> > > > >> > > > > Hi David,
> > > > >> > > > >
> > > > >> > > > > Thank you for pointing that out.
> > > > >> > > > > I have updated the content of the KIP based on Lianet's
> and
> > your
> > > > >> > > > feedback.
> > > > >> > > > > Please take a look and let me know your thoughts.
> > > > >> > > > >
> > > > >> > > > > Best regards,
> > > > >> > > > > TengYao
> > > > >> > > > >
> > > > >> > > > > David Jacot <dj...@confluent.io.invalid> 於 2024年8月29日 週四
> > > > >> 下午3:20寫道：
> > > > >> > > > >
> > > > >> > > > >> Hi TengYao,
> > > > >> > > > >>
> > > > >> > > > >> Thanks for the update. I haven't fully read it yet but I
> > will
> > > > >> soon.
> > > > >> > > > >>
> > > > >> > > > >> LM4: This is incorrect. The consumer must keep its member
> > id
> > > > >> during
> > > > >> > > its
> > > > >> > > > >> entire lifetime (until the process stops or dies). The
> > protocol
> > > > >> > > > stipulates
> > > > >> > > > >> that a member must rejoin with the same member id and the
> > > > member
> > > > >> > epoch
> > > > >> > > > set
> > > > >> > > > >> to zero when an FENCED_MEMBER_EPOCH occurs. This allows
> the
> > > > >> member
> > > > >> > to
> > > > >> > > > >> resynchronize itself. We should not change this
> behavior. I
> > > > think
> > > > >> > that
> > > > >> > > > we
> > > > >> > > > >> should see the client side generation id as an
> incarnation
> > id
> > > > of
> > > > >> the
> > > > >> > > > >> application. It is generated once and kept until it stops
> > or
> > > > >> dies.
> > > > >> > > > >>
> > > > >> > > > >> Best,
> > > > >> > > > >> David
> > > > >> > > > >>
> > > > >> > > > >> On Thu, Aug 29, 2024 at 6:21 AM TengYao Chi <
> > > > kitingiao@gmail.com
> > > > >> >
> > > > >> > > > wrote:
> > > > >> > > > >>
> > > > >> > > > >>> Hello Lianet !
> > > > >> > > > >>>
> > > > >> > > > >>> Thanks for the reviews and suggestions!
> > > > >> > > > >>>
> > > > >> > > > >>> LM1: Indeed, we plan to enforce client-side ID
> generation
> > in
> > > > the
> > > > >> > > > future,
> > > > >> > > > >>> and it is not an alternative. I will change the title
> > > > >> accordingly.
> > > > >> > > > >>>
> > > > >> > > > >>> LM2: Yes, that's the expectation. I will add that
> > statement to
> > > > >> the
> > > > >> > > > public
> > > > >> > > > >>> interface section.
> > > > >> > > > >>>
> > > > >> > > > >>> LM3: Thank you for the high-level perspective review. I
> > think
> > > > >> > you're
> > > > >> > > > >> right;
> > > > >> > > > >>> our intention isn't very clear since it was placed at
> the
> > end
> > > > of
> > > > >> > the
> > > > >> > > > >>> section. I will try to rephrase that section to make it
> > more
> > > > >> > obvious.
> > > > >> > > > >>>
> > > > >> > > > >>> LM4: Regarding the definition of "session" in this KIP,
> I
> > > > >> believe
> > > > >> > it
> > > > >> > > > >> refers
> > > > >> > > > >>> to the period between the *first-time heartbeat* and
> when
> > the
> > > > >> > > *consumer
> > > > >> > > > >>> leaves the group* (whether through a graceful shutdown
> or
> > a
> > > > >> > heartbeat
> > > > >> > > > >>> timeout). The consumer should reuse its UUID if it has
> > been
> > > > >> > generated
> > > > >> > > > >>> before. The only situation in which it will regenerate
> the
> > > > UUID
> > > > >> is
> > > > >> > if
> > > > >> > > > the
> > > > >> > > > >>> coordinator finds that there is already a consumer with
> > the
> > > > same
> > > > >> > > UUID.
> > > > >> > > > >>> IIRC, the coordinator should compare the member epochs,
> > and
> > > > the
> > > > >> > > > >>> later-joined consumer should be fenced off by the
> > coordinator
> > > > >> due
> > > > >> > to
> > > > >> > > > >> having
> > > > >> > > > >>> a lower member epoch. Once the consumer receives a
> > > > >> > > > `FENCED_MEMBER_EPOCH`
> > > > >> > > > >>> error, it will generate a new UUID and attempt to
> rejoin.
> > I
> > > > will
> > > > >> > > > clarify
> > > > >> > > > >>> this in the KIP.
> > > > >> > > > >>>
> > > > >> > > > >>> Thanks again for your reviews, I really appreciate it.
> > > > >> > > > >>>
> > > > >> > > > >>> Best regards,
> > > > >> > > > >>> TengYao
> > > > >> > > > >>>
> > > > >> > > > >>> Lianet M. <li...@gmail.com> 於 2024年8月28日 週三
> 下午7:12寫道：
> > > > >> > > > >>>
> > > > >> > > > >>>> Hello TengYao! Thanks for taking on this issue, we've
> > been
> > > > >> going
> > > > >> > > > around
> > > > >> > > > >>> it
> > > > >> > > > >>>> for a while.
> > > > >> > > > >>>>
> > > > >> > > > >>>> LM1: About the title of the KIP: "Enable ID Generation
> > for
> > > > >> Clients
> > > > >> > > > over
> > > > >> > > > >>> the
> > > > >> > > > >>>> ConsumerGroupHeartbeat RPC". I find it confusing
> because
> > it
> > > > >> hints
> > > > >> > > that
> > > > >> > > > >>>> we're adding it as an alternative (which was discussed
> > and
> > > > >> > > discarded,
> > > > >> > > > >> in
> > > > >> > > > >>>> favour of really enforcing it). It's also missing the
> > core
> > > > >> change
> > > > >> > > imo,
> > > > >> > > > >>>> which is "where" the generation happens. So, maybe more
> > to
> > > > the
> > > > >> > point
> > > > >> > > > >> with
> > > > >> > > > >>>> something along the lines of "Client-side generated ID
> > for
> > > > >> clients
> > > > >> > > > over
> > > > >> > > > >>>> ConsumerGroupHeartbeat RPC"?
> > > > >> > > > >>>>
> > > > >> > > > >>>> LM2: On the public interfaces section, the KIP states
> > that
> > > > "the
> > > > >> > > server
> > > > >> > > > >>> will
> > > > >> > > > >>>> reject the request", but we should agree on the
> specific
> > > > error
> > > > >> > > type. I
> > > > >> > > > >>>> expect it should fail with an InvalidRequestException,
> is
> > > > that
> > > > >> the
> > > > >> > > > >>>> intention? (This was also suggested in the discussion
> > thread
> > > > >> > before
> > > > >> > > > but
> > > > >> > > > >>> is
> > > > >> > > > >>>> not in the KIP).
> > > > >> > > > >>>>
> > > > >> > > > >>>> LM3. Related to my previous point, I find that to be
> the
> > true
> > > > >> > > > >>> public-facing
> > > > >> > > > >>>> change (member ID mandatory at the protocol level), but
> > it's
> > > > >> only
> > > > >> > at
> > > > >> > > > >> the
> > > > >> > > > >>>> end of the Public interfaces changes, kind of lost
> among
> > > > >> details
> > > > >> > of
> > > > >> > > > how
> > > > >> > > > >>>> we're going to do it. Should we rephrase that section
> > with
> > > > the
> > > > >> > > actual
> > > > >> > > > >>>> change first, and the hows after (ex. Bumping the
> > version is
> > > > >> not
> > > > >> > the
> > > > >> > > > >>>> public-facing change in this case, it's just the
> > mechanism to
> > > > >> > > properly
> > > > >> > > > >>>> introduce our change)
> > > > >> > > > >>>>
> > > > >> > > > >>>> LM4. Regarding the lifetime of the UUID: the KIP states
> > we
> > > > will
> > > > >> > > > "Verify
> > > > >> > > > >>>> that the UUID remains consistent across all subsequent
> > > > >> heartbeats
> > > > >> > > > >> during
> > > > >> > > > >>>> the session". What is this "session" referring to
> here? I
> > > > would
> > > > >> > > expect
> > > > >> > > > >>> that
> > > > >> > > > >>>> the UUID is associated to a consumer instance
> (generated
> > for
> > > > >> the
> > > > >> > > > >> consumer
> > > > >> > > > >>>> the first time it needs to send a HB if it doesn't have
> > the
> > > > >> UUID
> > > > >> > > yet.
> > > > >> > > > >>> From
> > > > >> > > > >>>> there on, every time it needs to send a "first HB"
> > again, it
> > > > >> will
> > > > >> > > > reuse
> > > > >> > > > >>> its
> > > > >> > > > >>>> UUID, is that the intention? Note that we should
> consider
> > > > that
> > > > >> the
> > > > >> > > > same
> > > > >> > > > >>>> consumer instance may have many "first heartbeats",
> > meaning
> > > > >> > > heartbeats
> > > > >> > > > >> to
> > > > >> > > > >>>> join the group when it's not part of it (ex. consumer
> > > > >> unsubscribe
> > > > >> > +
> > > > >> > > > >>>> subscribe, fenced, stale). Is this the intention or are
> > you
> > > > >> > > > considering
> > > > >> > > > >>> the
> > > > >> > > > >>>> lifetime differently? We should clarify it in the KIP.
> > > > >> > > > >>>>
> > > > >> > > > >>>> Thanks!
> > > > >> > > > >>>>
> > > > >> > > > >>>> Lianet
> > > > >> > > > >>>>
> > > > >> > > > >>>> On Tue, Aug 27, 2024 at 2:27 AM TengYao Chi <
> > > > >> kitingiao@gmail.com>
> > > > >> > > > >> wrote:
> > > > >> > > > >>>>
> > > > >> > > > >>>>> Hi everyone,
> > > > >> > > > >>>>>
> > > > >> > > > >>>>> I have revised this KIP multiple times based on the
> > feedback
> > > > >> from
> > > > >> > > our
> > > > >> > > > >>>>> discussions.
> > > > >> > > > >>>>> I would greatly appreciate it if you could review it
> > when
> > > > you
> > > > >> > have
> > > > >> > > > >> the
> > > > >> > > > >>>>> time.
> > > > >> > > > >>>>> If there are no further comments or suggestions, I
> plan
> > to
> > > > >> > proceed
> > > > >> > > > >> with
> > > > >> > > > >>>>> initiating a vote soon.
> > > > >> > > > >>>>>
> > > > >> > > > >>>>> Best regards,
> > > > >> > > > >>>>> TengYao
> > > > >> > > > >>>>>
> > > > >> > > > >>>>> TengYao Chi <ki...@gmail.com> 於 2024年8月23日 週五
> > 下午2:43寫道：
> > > > >> > > > >>>>>
> > > > >> > > > >>>>>> Hi Andrew,
> > > > >> > > > >>>>>> Thank you for your previous feedback and insights.
> > > > >> > > > >>>>>> Your contributions have added valuable perspectives
> to
> > the
> > > > >> > > > >>> discussions.
> > > > >> > > > >>>>>> And we also benefit from the comparison of different
> > > > >> solutions.
> > > > >> > > > >>>>>> I’m also looking forward to seeing an initial version
> > in
> > > > >> > KIP-932,
> > > > >> > > > >> as
> > > > >> > > > >>> it
> > > > >> > > > >>>>>> will provide a good reference for future
> > implementations.
> > > > >> > > > >>>>>>
> > > > >> > > > >>>>>> Regarding your comment on AS2, I wanted to clarify
> > that my
> > > > >> > > > >>>> specification
> > > > >> > > > >>>>>> references org.apache.kafka.common.Uuid.
> > > > >> > > > >>>>>> I believe we’re referring to the same class, and it
> > might
> > > > >> just
> > > > >> > be
> > > > >> > > a
> > > > >> > > > >>>> small
> > > > >> > > > >>>>>> oversight due to the busy schedule.
> > > > >> > > > >>>>>>
> > > > >> > > > >>>>>> I want to express my gratitude once again for your
> many
> > > > >> > insightful
> > > > >> > > > >>>>>> comments, which have helped the discussion progress
> > > > smoothly.
> > > > >> > > > >>>>>>
> > > > >> > > > >>>>>> Best regards,
> > > > >> > > > >>>>>> TengYao
> > > > >> > > > >>>>>>
> > > > >> > > > >>>>>>
> > > > >> > > > >>>>>> Andrew Schofield <an...@live.com> 於
> > 2024年8月22日
> > > > 週四
> > > > >> > > > >>>> 下午11:28寫道：
> > > > >> > > > >>>>>>
> > > > >> > > > >>>>>>> Hi TengYao,
> > > > >> > > > >>>>>>> I’ve been reading through the comments and I’m happy
> > that
> > > > >> the
> > > > >> > > > >> lobby
> > > > >> > > > >>>>>>> approach has not gained support.
> > > > >> > > > >>>>>>>
> > > > >> > > > >>>>>>> Assuming that this KIP is voted, I will be happy to
> > change
> > > > >> > > KIP-932
> > > > >> > > > >>> so
> > > > >> > > > >>>>>>> that it only supports client-side member ID
> > generation.
> > > > >> Because
> > > > >> > > > >> that
> > > > >> > > > >>>> KIP
> > > > >> > > > >>>>>>> is still
> > > > >> > > > >>>>>>> under development, I can do this in the first
> version
> > of
> > > > >> > > > >>>>>>> ShareGroupHeartbeat.
> > > > >> > > > >>>>>>>
> > > > >> > > > >>>>>>> AS2: For the encoding section, I suppose the
> specific
> > > > >> encoding
> > > > >> > > > >> which
> > > > >> > > > >>>>>>> is used is what org.apache.kafka.utils.Uuid uses.
> > > > >> > > > >>>>>>>
> > > > >> > > > >>>>>>> Thanks,
> > > > >> > > > >>>>>>> Andrew
> > > > >> > > > >>>>>>>
> > > > >> > > > >>>>>>>> On 14 Aug 2024, at 17:00, TengYao Chi <
> > > > kitingiao@gmail.com
> > > > >> >
> > > > >> > > > >>> wrote:
> > > > >> > > > >>>>>>>>
> > > > >> > > > >>>>>>>> Hello Apoorv,
> > > > >> > > > >>>>>>>> Thank you for your feedback.
> > > > >> > > > >>>>>>>> Regarding the questions you raised, unfortunately,
> > this
> > > > KIP
> > > > >> > > > >> cannot
> > > > >> > > > >>>>>>>> guarantee the order of heartbeats. As with many
> > classic
> > > > >> > > > >>> distributed
> > > > >> > > > >>>>>>> system
> > > > >> > > > >>>>>>>> challenges, what we can do is make our best effort
> to
> > > > >> ensure
> > > > >> > > > >> that
> > > > >> > > > >>>>> there
> > > > >> > > > >>>>>>> are
> > > > >> > > > >>>>>>>> no idle members or stale assignments under normal
> > > > >> > circumstances.
> > > > >> > > > >>>>>>>>
> > > > >> > > > >>>>>>>> As for the lobby approach, I’m not a fan of it
> > because it
> > > > >> > > > >> requires
> > > > >> > > > >>>>>>> adding a
> > > > >> > > > >>>>>>>> mechanism to maintain client state within the
> > > > >> ConsumerGroup,
> > > > >> > > > >>> which,
> > > > >> > > > >>>> in
> > > > >> > > > >>>>>>> my
> > > > >> > > > >>>>>>>> view, resembles something like a two-phase commit.
> > This
> > > > >> would
> > > > >> > > > >>>>> introduce
> > > > >> > > > >>>>>>>> more complexity than the proposal in this KIP,
> which
> > is
> > > > >> > > > >> something
> > > > >> > > > >>> we
> > > > >> > > > >>>>>>> want
> > > > >> > > > >>>>>>>> to avoid. KIP-848 aims to simplify the existing
> > protocol,
> > > > >> and
> > > > >> > > > >>> while
> > > > >> > > > >>>>> the
> > > > >> > > > >>>>>>>> lobby approach is a good one, I believe it is not
> the
> > > > right
> > > > >> > fit
> > > > >> > > > >>> for
> > > > >> > > > >>>>> this
> > > > >> > > > >>>>>>>> particular situation.
> > > > >> > > > >>>>>>>>
> > > > >> > > > >>>>>>>> Best regards,
> > > > >> > > > >>>>>>>> TengYao
> > > > >> > > > >>>>>>>>
> > > > >> > > > >>>>>>>> TengYao Chi <ki...@gmail.com> 於 2024年8月14日 週三
> > > > >> 下午11:45寫道：
> > > > >> > > > >>>>>>>>
> > > > >> > > > >>>>>>>>> Hi David,
> > > > >> > > > >>>>>>>>>
> > > > >> > > > >>>>>>>>> I really appreciate your review and suggestions.
> As
> > I am
> > > > >> > still
> > > > >> > > > >>>>> gaining
> > > > >> > > > >>>>>>>>> experience in writing KIPs, your input has been
> > > > incredibly
> > > > >> > > > >>>> helpful. I
> > > > >> > > > >>>>>>> am
> > > > >> > > > >>>>>>>>> currently applying your suggestions to the KIP and
> > will
> > > > >> > > > >> complete
> > > > >> > > > >>> it
> > > > >> > > > >>>>> as
> > > > >> > > > >>>>>>> soon
> > > > >> > > > >>>>>>>>> as possible.
> > > > >> > > > >>>>>>>>> Regarding the UUID part, I think we haven’t
> reached
> > a
> > > > >> > > > >> conclusion
> > > > >> > > > >>>>>>> yet.(So
> > > > >> > > > >>>>>>>>> far according to this thread)
> > > > >> > > > >>>>>>>>> However, I will review the current implementation
> > in the
> > > > >> > Kafka
> > > > >> > > > >>>> `Uuid`
> > > > >> > > > >>>>>>>>> class and include a brief specification in the
> KIP.
> > > > >> > > > >>>>>>>>>
> > > > >> > > > >>>>>>>>> Once again, thank you so much for your help.
> > > > >> > > > >>>>>>>>>
> > > > >> > > > >>>>>>>>> Best regards,
> > > > >> > > > >>>>>>>>> TengYao
> > > > >> > > > >>>>>>>>>
> > > > >> > > > >>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月14日
> 週三
> > > > >> > 下午11:14寫道：
> > > > >> > > > >>>>>>>>>
> > > > >> > > > >>>>>>>>>> hi Apoorv
> > > > >> > > > >>>>>>>>>>
> > > > >> > > > >>>>>>>>>>> As the memberId is now known to the client, and
> > client
> > > > >> > might
> > > > >> > > > >>> send
> > > > >> > > > >>>>> the
> > > > >> > > > >>>>>>>>>> leave
> > > > >> > > > >>>>>>>>>> group heartbeat on shutdown prior to receiving
> the
> > > > >> initial
> > > > >> > > > >>>> heartbeat
> > > > >> > > > >>>>>>>>>> response. If that's true then how do we guarantee
> > that
> > > > >> the 2
> > > > >> > > > >>>>> requests
> > > > >> > > > >>>>>>> to
> > > > >> > > > >>>>>>>>>> join and leave will be processed in order, which
> > could
> > > > >> still
> > > > >> > > > >>> leave
> > > > >> > > > >>>>>>> stale
> > > > >> > > > >>>>>>>>>> members or throw unknown member id exceptions?
> > > > >> > > > >>>>>>>>>>
> > > > >> > > > >>>>>>>>>> This is definitely a good question. the short
> > answer:
> > > > no
> > > > >> > > > >>> guarantee
> > > > >> > > > >>>>> but
> > > > >> > > > >>>>>>>>>> best
> > > > >> > > > >>>>>>>>>> efforts
> > > > >> > > > >>>>>>>>>>
> > > > >> > > > >>>>>>>>>> Please notice the root cause is "we have no
> enough
> > time
> > > > >> to
> > > > >> > > > >> wait
> > > > >> > > > >>>>>>> member id
> > > > >> > > > >>>>>>>>>> (response) when closing consumer". Sadly, we can'
> > > > >> guarantee
> > > > >> > > > >> the
> > > > >> > > > >>>>>>> request
> > > > >> > > > >>>>>>>>>> order due to the same reason.
> > > > >> > > > >>>>>>>>>>
> > > > >> > > > >>>>>>>>>> However, in contrast to previous behavior, there
> > is one
> > > > >> big
> > > > >> > > > >>>> benefit
> > > > >> > > > >>>>>>> of new
> > > > >> > > > >>>>>>>>>> approach - we can try STONITH because we know the
> > > > member
> > > > >> id
> > > > >> > > > >>>>>>>>>>
> > > > >> > > > >>>>>>>>>> Best,
> > > > >> > > > >>>>>>>>>> Chia-Ping
> > > > >> > > > >>>>>>>>>>
> > > > >> > > > >>>>>>>>>>
> > > > >> > > > >>>>>>>>>> Apoorv Mittal <ap...@gmail.com> 於
> > 2024年8月14日
> > > > 週三
> > > > >> > > > >>>> 下午8:55寫道：
> > > > >> > > > >>>>>>>>>>
> > > > >> > > > >>>>>>>>>>> Hi TengYao,
> > > > >> > > > >>>>>>>>>>> Thanks for the KIP. Continuing on the point
> which
> > > > Andrew
> > > > >> > > > >>>> mentioned
> > > > >> > > > >>>>> as
> > > > >> > > > >>>>>>>>>> AS1.
> > > > >> > > > >>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>> As the memberId is now known to the client, and
> > client
> > > > >> > might
> > > > >> > > > >>> send
> > > > >> > > > >>>>> the
> > > > >> > > > >>>>>>>>>> leave
> > > > >> > > > >>>>>>>>>>> group heartbeat on shutdown prior to receiving
> the
> > > > >> initial
> > > > >> > > > >>>>> heartbeat
> > > > >> > > > >>>>>>>>>>> response. If that's true then how do we
> guarantee
> > that
> > > > >> the
> > > > >> > 2
> > > > >> > > > >>>>>>> requests to
> > > > >> > > > >>>>>>>>>>> join and leave will be processed in order, which
> > could
> > > > >> > still
> > > > >> > > > >>>> leave
> > > > >> > > > >>>>>>> stale
> > > > >> > > > >>>>>>>>>>> members or throw unknown member id exceptions?
> > > > >> > > > >>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>> Though the client side member id generation is
> > helpful
> > > > >> > which
> > > > >> > > > >>> will
> > > > >> > > > >>>>>>>>>> represent
> > > > >> > > > >>>>>>>>>>> the same group perspective as from client and
> > broker's
> > > > >> end.
> > > > >> > > > >>> But I
> > > > >> > > > >>>>>>> think
> > > > >> > > > >>>>>>>>>> the
> > > > >> > > > >>>>>>>>>>> major concern we want to solve here is Stale
> > Partition
> > > > >> > > > >>>> Assignments
> > > > >> > > > >>>>>>> which
> > > > >> > > > >>>>>>>>>>> might still exist with the new approach. I am
> > leaning
> > > > >> > towards
> > > > >> > > > >>> the
> > > > >> > > > >>>>>>>>>>> suggestion mentioned by Andrew where partition
> > > > >> assignment
> > > > >> > > > >>>> triggers
> > > > >> > > > >>>>> on
> > > > >> > > > >>>>>>>>>>> subsequent heartbeat when client acknowledges
> the
> > > > >> initial
> > > > >> > > > >>>>> heartbeat,
> > > > >> > > > >>>>>>>>>>> delayed partition assignment.
> > > > >> > > > >>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>> Though on a separate note, I have a different
> > > > question.
> > > > >> > What
> > > > >> > > > >>>>> happens
> > > > >> > > > >>>>>>>>>> when
> > > > >> > > > >>>>>>>>>>> there is an issue with the client which sends
> the
> > > > >> initial
> > > > >> > > > >>>> heartbeat
> > > > >> > > > >>>>>>>>>> without
> > > > >> > > > >>>>>>>>>>> memberId, then crashes and restarts? I think we
> > must
> > > > be
> > > > >> > > > >>>>> re-triggering
> > > > >> > > > >>>>>>>>>>> assignments and expiring members only after the
> > > > >> heartbeat
> > > > >> > > > >>> session
> > > > >> > > > >>>>>>>>>> timeout?
> > > > >> > > > >>>>>>>>>>> If that's true then shall delayed partition
> > assignment
> > > > >> can
> > > > >> > > > >> help
> > > > >> > > > >>>>>>> benefit
> > > > >> > > > >>>>>>>>>> us
> > > > >> > > > >>>>>>>>>>> from this situation as well?
> > > > >> > > > >>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>> Regards,
> > > > >> > > > >>>>>>>>>>> Apoorv Mittal
> > > > >> > > > >>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>> On Wed, Aug 14, 2024 at 12:51 PM David Jacot
> > > > >> > > > >>>>>>>>>> <dj...@confluent.io.invalid>
> > > > >> > > > >>>>>>>>>>> wrote:
> > > > >> > > > >>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>> Hi Andrew,
> > > > >> > > > >>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>> Personally, I don't like the lobby approach. It
> > makes
> > > > >> > things
> > > > >> > > > >>>> more
> > > > >> > > > >>>>>>>>>>>> complicated and it would require changing the
> > records
> > > > >> on
> > > > >> > the
> > > > >> > > > >>>>> server
> > > > >> > > > >>>>>>>>>> too.
> > > > >> > > > >>>>>>>>>>>> This is why I initially suggested the rejected
> > > > >> alternative
> > > > >> > > > >> #2
> > > > >> > > > >>>>> which
> > > > >> > > > >>>>>>> is
> > > > >> > > > >>>>>>>>>>>> pretty close but also not perfect.
> > > > >> > > > >>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>> I'd like to clarify one thing. The
> > > > >> ConsumerGroupHeartbeat
> > > > >> > > > >> API
> > > > >> > > > >>>>>>> already
> > > > >> > > > >>>>>>>>>>>> supports generating the member id on the client
> > so we
> > > > >> > don't
> > > > >> > > > >>> need
> > > > >> > > > >>>>> any
> > > > >> > > > >>>>>>>>>>>> conditional logic on the client side. This is
> > > > actually
> > > > >> > what
> > > > >> > > > >> we
> > > > >> > > > >>>>>>> wanted
> > > > >> > > > >>>>>>>>>> to
> > > > >> > > > >>>>>>>>>>> do
> > > > >> > > > >>>>>>>>>>>> in the first place but the idea got pushed back
> > by
> > > > >> Magnus
> > > > >> > > > >> back
> > > > >> > > > >>>>> then
> > > > >> > > > >>>>>>>>>>> because
> > > > >> > > > >>>>>>>>>>>> generating uuid from librdkafka required a new
> > > > >> dependency.
> > > > >> > > > >> It
> > > > >> > > > >>>>> turns
> > > > >> > > > >>>>>>>>>> out
> > > > >> > > > >>>>>>>>>>>> that librdkafka has that dependency today. In
> > > > >> retrospect,
> > > > >> > we
> > > > >> > > > >>>>> should
> > > > >> > > > >>>>>>>>>> have
> > > > >> > > > >>>>>>>>>>>> pushed back on this. Long story short, we can
> > just do
> > > > >> it.
> > > > >> > > > >> The
> > > > >> > > > >>>>>>>>>> proposal in
> > > > >> > > > >>>>>>>>>>>> this KIP is to make the member id required in
> > future
> > > > >> > > > >> versions.
> > > > >> > > > >>>> We
> > > > >> > > > >>>>>>>>>> could
> > > > >> > > > >>>>>>>>>>>> also decide not to do it and to keep supporting
> > both
> > > > >> > > > >>>> approaches. I
> > > > >> > > > >>>>>>>>>> would
> > > > >> > > > >>>>>>>>>>>> also be fine with this.
> > > > >> > > > >>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>> Best,
> > > > >> > > > >>>>>>>>>>>> David
> > > > >> > > > >>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>> On Wed, Aug 14, 2024 at 12:30 PM Andrew
> > Schofield <
> > > > >> > > > >>>>>>>>>>>> andrew_schofield@live.com>
> > > > >> > > > >>>>>>>>>>>> wrote:
> > > > >> > > > >>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>> Hi TengYao,
> > > > >> > > > >>>>>>>>>>>>> Thanks for your response. I’ll have just one
> > more
> > > > try
> > > > >> to
> > > > >> > > > >>>>> persuade.
> > > > >> > > > >>>>>>>>>>>>> I feel that I will need to follow the approach
> > with
> > > > >> > KIP-932
> > > > >> > > > >>>> when
> > > > >> > > > >>>>>>>>>> we’ve
> > > > >> > > > >>>>>>>>>>>>> made a decision, so I do have more than a
> > passing
> > > > >> > interest
> > > > >> > > > >> in
> > > > >> > > > >>>>> this.
> > > > >> > > > >>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>> A group member in the lobby is in the group,
> > but it
> > > > >> does
> > > > >> > > > >> not
> > > > >> > > > >>>> have
> > > > >> > > > >>>>>>>>>> any
> > > > >> > > > >>>>>>>>>>>>> assignments. A member of a consumer group can
> > have
> > > > no
> > > > >> > > > >>> assigned
> > > > >> > > > >>>>>>>>>>>>> partitions (such as 5 CG members subscribed
> to a
> > > > topic
> > > > >> > > > >> with 4
> > > > >> > > > >>>>>>>>>>>> partitions),
> > > > >> > > > >>>>>>>>>>>>> so it’s a situation that consumer group
> members
> > > > >> already
> > > > >> > > > >>> expect.
> > > > >> > > > >>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>> One of Kafka’s strengths is the way that we
> > handle
> > > > API
> > > > >> > > > >>>>> versioning.
> > > > >> > > > >>>>>>>>>>>>> But, there is a cost - the behaviour is
> > different
> > > > >> > depending
> > > > >> > > > >>> on
> > > > >> > > > >>>>> the
> > > > >> > > > >>>>>>>>>> RPC
> > > > >> > > > >>>>>>>>>>>>> version. KIP-848 is on the cusp of completion,
> > but
> > > > >> we’re
> > > > >> > > > >>>> already
> > > > >> > > > >>>>>>>>>> adding
> > > > >> > > > >>>>>>>>>>>>> conditional logic for v0/v1 for
> > > > >> ConsumerGroupHeartbeat.
> > > > >> > > > >>> That’s
> > > > >> > > > >>>> a
> > > > >> > > > >>>>>>>>>> pity.
> > > > >> > > > >>>>>>>>>>>>> Only a minor issue, but it’s unfortunate.
> > > > >> > > > >>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>> Thanks,
> > > > >> > > > >>>>>>>>>>>>> Andrew
> > > > >> > > > >>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>> On 14 Aug 2024, at 08:47, TengYao Chi <
> > > > >> > > > >> kitingiao@gmail.com>
> > > > >> > > > >>>>>>>>>> wrote:
> > > > >> > > > >>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>> Hello Andrew
> > > > >> > > > >>>>>>>>>>>>>> Thank you for your thoughtful suggestions and
> > > > getting
> > > > >> > the
> > > > >> > > > >>>>>>>>>> discussion
> > > > >> > > > >>>>>>>>>>>>> going.
> > > > >> > > > >>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>> To AS1:
> > > > >> > > > >>>>>>>>>>>>>> In the current scenario where the server
> > generates
> > > > >> the
> > > > >> > > > >> UUID,
> > > > >> > > > >>>> if
> > > > >> > > > >>>>>>>>>> the
> > > > >> > > > >>>>>>>>>>>>> client
> > > > >> > > > >>>>>>>>>>>>>> shuts down before receiving the memberId
> > generated
> > > > by
> > > > >> > the
> > > > >> > > > >> GC
> > > > >> > > > >>>>>>>>>>>> (regardless
> > > > >> > > > >>>>>>>>>>>>> of
> > > > >> > > > >>>>>>>>>>>>>> whether it’s a graceful shutdown or not), the
> > GC
> > > > will
> > > > >> > > > >> still
> > > > >> > > > >>>> have
> > > > >> > > > >>>>>>>>>> to
> > > > >> > > > >>>>>>>>>>>> wait
> > > > >> > > > >>>>>>>>>>>>>> for the heartbeat timeout because the client
> > > > doesn’t
> > > > >> > know
> > > > >> > > > >>> its
> > > > >> > > > >>>>>>>>>>> memberId.
> > > > >> > > > >>>>>>>>>>>>>> This KIP indeed cannot completely resolve the
> > > > >> > idempotency
> > > > >> > > > >>>> issue,
> > > > >> > > > >>>>>>>>>> but
> > > > >> > > > >>>>>>>>>>> it
> > > > >> > > > >>>>>>>>>>>>> can
> > > > >> > > > >>>>>>>>>>>>>> better handle shutdown scenarios under normal
> > > > >> > > > >> circumstances
> > > > >> > > > >>>>>>>>>> because
> > > > >> > > > >>>>>>>>>>> the
> > > > >> > > > >>>>>>>>>>>>>> client always knows its memberId. Even if the
> > > > client
> > > > >> > shuts
> > > > >> > > > >>>> down
> > > > >> > > > >>>>>>>>>>>>> immediately
> > > > >> > > > >>>>>>>>>>>>>> after the initial heartbeat, as long as it
> > > > performs a
> > > > >> > > > >>> graceful
> > > > >> > > > >>>>>>>>>>> shutdown
> > > > >> > > > >>>>>>>>>>>>> and
> > > > >> > > > >>>>>>>>>>>>>> sends a leave heartbeat, the GC can manage
> the
> > > > >> situation
> > > > >> > > > >> and
> > > > >> > > > >>>>>>>>>> remove
> > > > >> > > > >>>>>>>>>>> the
> > > > >> > > > >>>>>>>>>>>>>> member. Therefore, the goal of this KIP is to
> > > > address
> > > > >> > the
> > > > >> > > > >>>> issue
> > > > >> > > > >>>>>>>>>> where
> > > > >> > > > >>>>>>>>>>>> the
> > > > >> > > > >>>>>>>>>>>>>> GC has to wait for the heartbeat timeout due
> > to the
> > > > >> > client
> > > > >> > > > >>>>> leaving
> > > > >> > > > >>>>>>>>>>>>> without
> > > > >> > > > >>>>>>>>>>>>>> knowing its memberId, which leads to reduced
> > > > >> throughput
> > > > >> > > > >> and
> > > > >> > > > >>>>>>>>>> limited
> > > > >> > > > >>>>>>>>>>>>>> scalability.
> > > > >> > > > >>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>> The solution you suggest has also been
> > proposed by
> > > > >> > David.
> > > > >> > > > >>> The
> > > > >> > > > >>>>>>>>>> concern
> > > > >> > > > >>>>>>>>>>>>> with
> > > > >> > > > >>>>>>>>>>>>>> this approach is that it introduces
> additional
> > > > >> > complexity
> > > > >> > > > >>> for
> > > > >> > > > >>>>>>>>>>>>>> compatibility, as the new server would not
> > > > >> immediately
> > > > >> > add
> > > > >> > > > >>> the
> > > > >> > > > >>>>>>>>>> member
> > > > >> > > > >>>>>>>>>>>> to
> > > > >> > > > >>>>>>>>>>>>>> the group, while the old server would. This
> > > > requires
> > > > >> > > > >> clients
> > > > >> > > > >>>> to
> > > > >> > > > >>>>>>>>>>>>>> differentiate whether their memberId has been
> > added
> > > > >> to
> > > > >> > the
> > > > >> > > > >>>> group
> > > > >> > > > >>>>>>>>>> or
> > > > >> > > > >>>>>>>>>>>> not,
> > > > >> > > > >>>>>>>>>>>>>> which could result in unexpected logs.
> > > > >> > > > >>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>> Best Regards,
> > > > >> > > > >>>>>>>>>>>>>> TengYao
> > > > >> > > > >>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>> Andrew Schofield <an...@live.com>
> 於
> > > > >> > 2024年8月14日
> > > > >> > > > >>> 週三
> > > > >> > > > >>>>>>>>>>>> 上午12:29寫道：
> > > > >> > > > >>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>>> Hi TengYao,
> > > > >> > > > >>>>>>>>>>>>>>> Thanks for the KIP. I wonder if there’s a
> > > > different
> > > > >> way
> > > > >> > > > >> to
> > > > >> > > > >>>>> close
> > > > >> > > > >>>>>>>>>>> what
> > > > >> > > > >>>>>>>>>>>>>>> is quite a small window.
> > > > >> > > > >>>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>>> AS1: It is true that the initial heartbeat
> is
> > not
> > > > >> > > > >>> idempotent,
> > > > >> > > > >>>>> but
> > > > >> > > > >>>>>>>>>>> this
> > > > >> > > > >>>>>>>>>>>>>>> remains
> > > > >> > > > >>>>>>>>>>>>>>> true with this KIP. It’s just differently
> not
> > > > >> > idempotent.
> > > > >> > > > >>> If
> > > > >> > > > >>>>> the
> > > > >> > > > >>>>>>>>>>>> client
> > > > >> > > > >>>>>>>>>>>>>>> makes its
> > > > >> > > > >>>>>>>>>>>>>>> own member ID, sends a request and dies, the
> > GC
> > > > will
> > > > >> > > > >> still
> > > > >> > > > >>>> have
> > > > >> > > > >>>>>>>>>>> added
> > > > >> > > > >>>>>>>>>>>>>>> the member to the group and it will hang
> > around
> > > > >> until
> > > > >> > the
> > > > >> > > > >>>>> session
> > > > >> > > > >>>>>>>>>>>>> expires.
> > > > >> > > > >>>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>>> I wonder if the GC could still generate the
> > member
> > > > >> ID
> > > > >> > in
> > > > >> > > > >>>>>>>>>> response to
> > > > >> > > > >>>>>>>>>>>> the
> > > > >> > > > >>>>>>>>>>>>>>> first
> > > > >> > > > >>>>>>>>>>>>>>> heartbeat, and put the member in a special
> > PENDING
> > > > >> > state
> > > > >> > > > >>> with
> > > > >> > > > >>>>> no
> > > > >> > > > >>>>>>>>>>>>>>> assignments until the client sends the next
> > > > >> heartbeat,
> > > > >> > > > >> thus
> > > > >> > > > >>>>>>>>>>> confirming
> > > > >> > > > >>>>>>>>>>>>> it
> > > > >> > > > >>>>>>>>>>>>>>> has received the member ID. This would not
> be
> > a
> > > > >> > protocol
> > > > >> > > > >>>> change
> > > > >> > > > >>>>>>>>>> at
> > > > >> > > > >>>>>>>>>>>> all,
> > > > >> > > > >>>>>>>>>>>>>>> just
> > > > >> > > > >>>>>>>>>>>>>>> a change to the GC to keep a new member in
> the
> > > > lobby
> > > > >> > > > >> until
> > > > >> > > > >>>> it’s
> > > > >> > > > >>>>>>>>>>>>> comfirmed
> > > > >> > > > >>>>>>>>>>>>>>> it knows its member ID.
> > > > >> > > > >>>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>>> Thanks,
> > > > >> > > > >>>>>>>>>>>>>>> Andrew
> > > > >> > > > >>>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>>>> On 13 Aug 2024, at 15:59, TengYao Chi <
> > > > >> > > > >>> kitingiao@gmail.com>
> > > > >> > > > >>>>>>>>>> wrote:
> > > > >> > > > >>>>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>>>> Hi Chia-Ping,
> > > > >> > > > >>>>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>>>> Thanks for review and suggestions.
> > > > >> > > > >>>>>>>>>>>>>>>> I have updated the content of KIP
> > accordingly.
> > > > >> > > > >>>>>>>>>>>>>>>> Please take a look.
> > > > >> > > > >>>>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>>>> Best regards,
> > > > >> > > > >>>>>>>>>>>>>>>> TengYao
> > > > >> > > > >>>>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > > > 2024年8月13日
> > > > >> 週二
> > > > >> > > > >>>>> 下午9:45寫道：
> > > > >> > > > >>>>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>>>>> hi TengYao
> > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>>>>> thanks for this KIP.
> > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>>>>> 1) could you please describe the
> > before/after
> > > > >> > behavior
> > > > >> > > > >> in
> > > > >> > > > >>>> the
> > > > >> > > > >>>>>>>>>>>>> "Proposed
> > > > >> > > > >>>>>>>>>>>>>>>>> Changes" section? IIRC, current RPC allows
> > HB
> > > > >> having
> > > > >> > > > >>> member
> > > > >> > > > >>>>> id
> > > > >> > > > >>>>>>>>>>>>>>> generated by
> > > > >> > > > >>>>>>>>>>>>>>>>> client, right? If HB has no member ID,
> > server
> > > > will
> > > > >> > > > >>> generate
> > > > >> > > > >>>>> one
> > > > >> > > > >>>>>>>>>>> and
> > > > >> > > > >>>>>>>>>>>>> then
> > > > >> > > > >>>>>>>>>>>>>>>>> return. The new behavior will enforce HB
> > "must"
> > > > >> have
> > > > >> > > > >>> member
> > > > >> > > > >>>>> ID.
> > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>>>>> 2) could you please write the version
> number
> > > > >> > explicitly
> > > > >> > > > >>> in
> > > > >> > > > >>>>> the
> > > > >> > > > >>>>>>>>>> KIP
> > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>>>>> 3) how new client code handle the old HB?
> > Does
> > > > it
> > > > >> > > > >> always
> > > > >> > > > >>>>>>>>>> generate
> > > > >> > > > >>>>>>>>>>>>> member
> > > > >> > > > >>>>>>>>>>>>>>>>> ID on client-side even though that is not
> > > > >> restricted?
> > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>>>>> Best,
> > > > >> > > > >>>>>>>>>>>>>>>>> Chia-Ping
> > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>>>>> On 2024/08/13 06:20:42 TengYao Chi wrote:
> > > > >> > > > >>>>>>>>>>>>>>>>>> Hello everyone,
> > > > >> > > > >>>>>>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>>>>>> I would like to start a discussion thread
> > on
> > > > >> > KIP-1082,
> > > > >> > > > >>>> which
> > > > >> > > > >>>>>>>>>>>> proposes
> > > > >> > > > >>>>>>>>>>>>>>>>>> enabling id generation for clients over
> the
> > > > >> > > > >>>>>>>>>>> ConsumerGroupHeartbeat
> > > > >> > > > >>>>>>>>>>>>> RPC.
> > > > >> > > > >>>>>>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>>>>>> Here is the KIP Link: KIP-1082
> > > > >> > > > >>>>>>>>>>>>>>>>>> <
> > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>
> > > > >> > > > >>>>>>>
> > > > >> > > > >>>>>
> > > > >> > > > >>>>
> > > > >> > > > >>>
> > > > >> > > > >>
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
> > > > >> > > > >>>>>>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>>>>>> Please take a look and let me know what
> you
> > > > >> think,
> > > > >> > > > >> and I
> > > > >> > > > >>>>> would
> > > > >> > > > >>>>>>>>>>>>>>> appreciate
> > > > >> > > > >>>>>>>>>>>>>>>>>> any suggestions and feedback.
> > > > >> > > > >>>>>>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>>>>>> Best regards,
> > > > >> > > > >>>>>>>>>>>>>>>>>> TengYao
> > > > >> > > > >>>>>>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>>
> > > > >> > > > >>>>>>>>>>
> > > > >> > > > >>>>>>>>>
> > > > >> > > > >>>>>>>
> > > > >> > > > >>>>>>>
> > > > >> > > > >>>>>
> > > > >> > > > >>>>
> > > > >> > > > >>>
> > > > >> > > > >>
> > > > >> > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[TengYao Chi <ki...@gmail.com>]

Hi Chia-Ping,

Thanks for pointing out this issue.

I’m thinking that maybe we might need to define a new Exception to handle
this scenario.
Once the client receives this exception, it should consider the exception
as a serious error and stop the process, and the user will need to assign a
new memberId.
What do you think?

Best Regards,
TengYao

Chia-Ping Tsai <ch...@apache.org> 於 2024年9月20日 週五 下午7:00寫道：

> > This part is not clear either. It basically says that if a member joins
> with an existing member id but a different epoch, it will be fenced. Then
> it must rejoin with the same member id and epoch zero. This is already the
> current behavior and it does not help with detecting duplicates, right?
>
> The duplicates issue is interesting. The member id is generated on
> server-side by UUID before, so it seems to me the fenced member epoch
> happens only if the client miss the response with bumped epoch, and this
> case is recoverable.
>
> however, v1 brings a different fenced epoch scenario. Users, now, have
> responsibility to generate unique member id, hence clients may encounter
> infinite fenced error if there are >1 clients having same member id due to
> config error (or other engineer error)
>
> Maybe we can highlight this scenario if we start to requires clients to
> generate unique member id.
>
> Best,
> Chia-Ping
>
> On 2024/09/19 18:36:48 David Jacot wrote:
> > Hi,
> >
> > Thanks for the update. I have a few nits:
> >
> > > If the member ID is null or empty, the server will reject the request
> > with an InvalidRequestException.
> > We should clarify that this should only apply to version >= 1.
> >
> > > The consumer instance must generate a member ID, and this ID should
> > remain consistent for the duration of the consumer's session. Here, a
> > "session" is defined as the period from the consumer's first heartbeat
> > until it leaves the group, either through a graceful shutdown, a
> heartbeat
> > timeout, or the process stopping or dying. The consumer instance should
> > reuse the same member ID for all heartbeats and rejoin attempts to
> maintain
> > continuity within the group.
> >
> > This part is not clear to me. When the member leaves the group, it should
> > not reset the member id. I would rather say that the member must generate
> > its member id when it starts and it must keep it until the process stops.
> > It is basically an incarnation of the process.
> >
> > > If a conflict arises where the member ID generated by the client is
> > detected to be a duplicate within the same group (for example, the same
> > member ID is associated with another active member in the group), the
> > server will handle this by comparing the memberEpoch values of the
> > conflicting members. The member with the lower memberEpoch is considered
> > outdated and will be fenced off by the server. When this occurs, the
> server
> > responds with a FENCED_MEMBER_EPOCH error to the client, signaling it to
> > rejoin the group with the same member ID while resetting the memberEpoch
> to
> > zero. This ensures that the client properly resynchronizes and maintains
> > the continuity and consistency of the group membership.
> >
> > This part is not clear either. It basically says that if a member joins
> > with an existing member id but a different epoch, it will be fenced. Then
> > it must rejoin with the same member id and epoch zero. This is already
> the
> > current behavior and it does not help with detecting duplicates, right?
> > Should we just remove the paragraph?
> >
> > > A member ID mismatch occurs within a session: If the server detects a
> > mismatch between the provided member ID and the expected member ID for an
> > ongoing session, it should return a UNKNOWN_MEMBER_ID  error.
> >
> > How could we detect a mismatch between the provided and the expected
> member
> > id? My understanding is that we can only know whether the provided member
> > id exists or not. This is already implemented.
> >
> > Thanks,
> > David
> >
> > On Sat, Sep 14, 2024 at 9:31 AM TengYao Chi <ki...@gmail.com> wrote:
> >
> > > Hello everyone,
> > >
> > > Since this KIP has been fully discussed, I will initiate a vote for it
> next
> > > Monday.
> > > Thank you and have a nice weekend.
> > >
> > > Best regards,
> > > TengYao
> > >
> > > TengYao Chi <ki...@gmail.com> 於 2024年9月5日 週四 下午2:19寫道：
> > >
> > > > Hello everyone,
> > > >
> > > > KT2: It looks like everyone who has expressed an opinion supports the
> > > > second option: “Document a recommendation for clients to use UUIDs as
> > > > member IDs, without strictly enforcing it.”
> > > > I have updated the KIP accordingly.
> > > > Please take a look, and let me know if you have any thoughts or
> feedback.
> > > >
> > > > Thank you!
> > > >
> > > > Best regards,
> > > > TengYao
> > > >
> > > > Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月30日 週五 下午9:56寫道：
> > > >
> > > >> hi TengYao
> > > >>
> > > >> KT2: +1 to second approach
> > > >>
> > > >> Best,
> > > >> Chia-Ping
> > > >>
> > > >>
> > > >> David Jacot <dj...@confluent.io.invalid> 於 2024年8月30日 週五 下午9:15寫道：
> > > >>
> > > >> > Hi TengYao,
> > > >> >
> > > >> > KT2: I don't think that we can realistically validate the uuid on
> the
> > > >> > server. It is basically a string of chars. So I lean towards
> having a
> > > >> good
> > > >> > recommendation in the KIP and in the document of the field in the
> > > RPC's
> > > >> > definition.
> > > >> >
> > > >> > Best,
> > > >> > David
> > > >> >
> > > >> > On Fri, Aug 30, 2024 at 3:02 PM TengYao Chi <ki...@gmail.com>
> > > >> wrote:
> > > >> >
> > > >> > > Hello Kirk !
> > > >> > >
> > > >> > > Thank you for your comments !
> > > >> > >
> > > >> > > KT1: Yes, you are correct. The issue is not unique to the
> initial
> > > >> > > heartbeat; there can always be cases where the broker might lose
> > > >> > connection
> > > >> > > with a member.
> > > >> > >
> > > >> > > KT2: Currently, if the client doesn't have a member ID and the
> > > >> > memberEpoch
> > > >> > > equals 0, the coordinator will generate a UUID as the member ID
> for
> > > >> the
> > > >> > > client. However, at the RPC level, the member ID is sent as a
> > > literal
> > > >> > > string, meaning there are no restrictions on the format at this
> > > level.
> > > >> > > This also reminds me that we haven't reached a final conclusion
> on
> > > >> how to
> > > >> > > enforce the use of UUIDs.
> > > >> > > From our previous discussions, I recall two possible approaches:
> > > >> > > The first is to validate the UUID on the server side, and if
> it's
> > > not
> > > >> > > valid, throw an exception to the client.
> > > >> > > The second is to document a recommendation for clients to use
> UUIDs
> > > as
> > > >> > > member IDs, without strictly enforcing it.
> > > >> > > I think it's time to decide on the approach we want to take.
> > > >> > >
> > > >> > > KT3: Yes, "session" can be considered synonymous with
> "membership"
> > > in
> > > >> > this
> > > >> > > context.
> > > >> > >
> > > >> > > KT4: Thank you for pointing that out. I will update the wording
> to
> > > >> > > specifically say this behavior is for consumers.
> > > >> > >
> > > >> > > Thanks again for your comments.
> > > >> > >
> > > >> > > Best regards,
> > > >> > > TengYao
> > > >> > >
> > > >> > > Kirk True <ki...@kirktrue.pro> 於 2024年8月30日 週五 上午12:39寫道：
> > > >> > >
> > > >> > > > Hi TengYao!
> > > >> > > >
> > > >> > > > Sorry for being late to the discussion...
> > > >> > > >
> > > >> > > > After reading the thread and then the KIP, I had a few
> > > >> > > questions/comments:
> > > >> > > >
> > > >> > > > KT1: In Motivation, it states: "This scenario can result in
> the
> > > >> broker
> > > >> > > > registering a new member for which it will never receive a
> proper
> > > >> leave
> > > >> > > > request.” Just to be clear, the broker will always have cases
> > > where
> > > >> it
> > > >> > > > might lose connection with a member. That’s not unique to the
> > > >> initial
> > > >> > > > heartbeat, right?
> > > >> > > >
> > > >> > > > KT2: There was a bit of back and forth about format of the
> member
> > > >> ID.
> > > >> > > From
> > > >> > > > what I gathered in the thread, the member ID is still defined
> in
> > > the
> > > >> > RPC
> > > >> > > as
> > > >> > > > a string and not a UUID, right? The KIP states that the
> “client
> > > must
> > > >> > > > generate a UUID as the member ID” and that the “server will
> > > validate
> > > >> > > that a
> > > >> > > > valid UUID is provided.” Is that a change for the server, or
> is it
> > > >> > > already
> > > >> > > > enforced as a UUID?
> > > >> > > >
> > > >> > > > KT3: Lianet mentioned some confusion over the use of the word
> > > >> > “session.”
> > > >> > > > Isn’t “session” synonymous with “membership?”
> > > >> > > >
> > > >> > > > KT4: Under “Member ID Lifecycle,” it states: "The client
> should
> > > >> reuse
> > > >> > the
> > > >> > > > same UUID as the member ID for all heartbeats and rejoin
> attempts
> > > to
> > > >> > > > maintain continuity within the group.” Could we change the
> first
> > > >> part
> > > >> > of
> > > >> > > > that to “The Consumer instance should…” We do have lifetimes
> that
> > > >> > extend
> > > >> > > > past the lifetime of a client instance (such as the
> transaction
> > > ID).
> > > >> > > >
> > > >> > > > Thanks,
> > > >> > > > Kirk
> > > >> > > >
> > > >> > > > > On Aug 29, 2024, at 1:28 AM, TengYao Chi <
> kitingiao@gmail.com>
> > > >> > wrote:
> > > >> > > > >
> > > >> > > > > Hi David,
> > > >> > > > >
> > > >> > > > > Thank you for pointing that out.
> > > >> > > > > I have updated the content of the KIP based on Lianet's and
> your
> > > >> > > > feedback.
> > > >> > > > > Please take a look and let me know your thoughts.
> > > >> > > > >
> > > >> > > > > Best regards,
> > > >> > > > > TengYao
> > > >> > > > >
> > > >> > > > > David Jacot <dj...@confluent.io.invalid> 於 2024年8月29日 週四
> > > >> 下午3:20寫道：
> > > >> > > > >
> > > >> > > > >> Hi TengYao,
> > > >> > > > >>
> > > >> > > > >> Thanks for the update. I haven't fully read it yet but I
> will
> > > >> soon.
> > > >> > > > >>
> > > >> > > > >> LM4: This is incorrect. The consumer must keep its member
> id
> > > >> during
> > > >> > > its
> > > >> > > > >> entire lifetime (until the process stops or dies). The
> protocol
> > > >> > > > stipulates
> > > >> > > > >> that a member must rejoin with the same member id and the
> > > member
> > > >> > epoch
> > > >> > > > set
> > > >> > > > >> to zero when an FENCED_MEMBER_EPOCH occurs. This allows the
> > > >> member
> > > >> > to
> > > >> > > > >> resynchronize itself. We should not change this behavior. I
> > > think
> > > >> > that
> > > >> > > > we
> > > >> > > > >> should see the client side generation id as an incarnation
> id
> > > of
> > > >> the
> > > >> > > > >> application. It is generated once and kept until it stops
> or
> > > >> dies.
> > > >> > > > >>
> > > >> > > > >> Best,
> > > >> > > > >> David
> > > >> > > > >>
> > > >> > > > >> On Thu, Aug 29, 2024 at 6:21 AM TengYao Chi <
> > > kitingiao@gmail.com
> > > >> >
> > > >> > > > wrote:
> > > >> > > > >>
> > > >> > > > >>> Hello Lianet !
> > > >> > > > >>>
> > > >> > > > >>> Thanks for the reviews and suggestions!
> > > >> > > > >>>
> > > >> > > > >>> LM1: Indeed, we plan to enforce client-side ID generation
> in
> > > the
> > > >> > > > future,
> > > >> > > > >>> and it is not an alternative. I will change the title
> > > >> accordingly.
> > > >> > > > >>>
> > > >> > > > >>> LM2: Yes, that's the expectation. I will add that
> statement to
> > > >> the
> > > >> > > > public
> > > >> > > > >>> interface section.
> > > >> > > > >>>
> > > >> > > > >>> LM3: Thank you for the high-level perspective review. I
> think
> > > >> > you're
> > > >> > > > >> right;
> > > >> > > > >>> our intention isn't very clear since it was placed at the
> end
> > > of
> > > >> > the
> > > >> > > > >>> section. I will try to rephrase that section to make it
> more
> > > >> > obvious.
> > > >> > > > >>>
> > > >> > > > >>> LM4: Regarding the definition of "session" in this KIP, I
> > > >> believe
> > > >> > it
> > > >> > > > >> refers
> > > >> > > > >>> to the period between the *first-time heartbeat* and when
> the
> > > >> > > *consumer
> > > >> > > > >>> leaves the group* (whether through a graceful shutdown or
> a
> > > >> > heartbeat
> > > >> > > > >>> timeout). The consumer should reuse its UUID if it has
> been
> > > >> > generated
> > > >> > > > >>> before. The only situation in which it will regenerate the
> > > UUID
> > > >> is
> > > >> > if
> > > >> > > > the
> > > >> > > > >>> coordinator finds that there is already a consumer with
> the
> > > same
> > > >> > > UUID.
> > > >> > > > >>> IIRC, the coordinator should compare the member epochs,
> and
> > > the
> > > >> > > > >>> later-joined consumer should be fenced off by the
> coordinator
> > > >> due
> > > >> > to
> > > >> > > > >> having
> > > >> > > > >>> a lower member epoch. Once the consumer receives a
> > > >> > > > `FENCED_MEMBER_EPOCH`
> > > >> > > > >>> error, it will generate a new UUID and attempt to rejoin.
> I
> > > will
> > > >> > > > clarify
> > > >> > > > >>> this in the KIP.
> > > >> > > > >>>
> > > >> > > > >>> Thanks again for your reviews, I really appreciate it.
> > > >> > > > >>>
> > > >> > > > >>> Best regards,
> > > >> > > > >>> TengYao
> > > >> > > > >>>
> > > >> > > > >>> Lianet M. <li...@gmail.com> 於 2024年8月28日 週三 下午7:12寫道：
> > > >> > > > >>>
> > > >> > > > >>>> Hello TengYao! Thanks for taking on this issue, we've
> been
> > > >> going
> > > >> > > > around
> > > >> > > > >>> it
> > > >> > > > >>>> for a while.
> > > >> > > > >>>>
> > > >> > > > >>>> LM1: About the title of the KIP: "Enable ID Generation
> for
> > > >> Clients
> > > >> > > > over
> > > >> > > > >>> the
> > > >> > > > >>>> ConsumerGroupHeartbeat RPC". I find it confusing because
> it
> > > >> hints
> > > >> > > that
> > > >> > > > >>>> we're adding it as an alternative (which was discussed
> and
> > > >> > > discarded,
> > > >> > > > >> in
> > > >> > > > >>>> favour of really enforcing it). It's also missing the
> core
> > > >> change
> > > >> > > imo,
> > > >> > > > >>>> which is "where" the generation happens. So, maybe more
> to
> > > the
> > > >> > point
> > > >> > > > >> with
> > > >> > > > >>>> something along the lines of "Client-side generated ID
> for
> > > >> clients
> > > >> > > > over
> > > >> > > > >>>> ConsumerGroupHeartbeat RPC"?
> > > >> > > > >>>>
> > > >> > > > >>>> LM2: On the public interfaces section, the KIP states
> that
> > > "the
> > > >> > > server
> > > >> > > > >>> will
> > > >> > > > >>>> reject the request", but we should agree on the specific
> > > error
> > > >> > > type. I
> > > >> > > > >>>> expect it should fail with an InvalidRequestException, is
> > > that
> > > >> the
> > > >> > > > >>>> intention? (This was also suggested in the discussion
> thread
> > > >> > before
> > > >> > > > but
> > > >> > > > >>> is
> > > >> > > > >>>> not in the KIP).
> > > >> > > > >>>>
> > > >> > > > >>>> LM3. Related to my previous point, I find that to be the
> true
> > > >> > > > >>> public-facing
> > > >> > > > >>>> change (member ID mandatory at the protocol level), but
> it's
> > > >> only
> > > >> > at
> > > >> > > > >> the
> > > >> > > > >>>> end of the Public interfaces changes, kind of lost among
> > > >> details
> > > >> > of
> > > >> > > > how
> > > >> > > > >>>> we're going to do it. Should we rephrase that section
> with
> > > the
> > > >> > > actual
> > > >> > > > >>>> change first, and the hows after (ex. Bumping the
> version is
> > > >> not
> > > >> > the
> > > >> > > > >>>> public-facing change in this case, it's just the
> mechanism to
> > > >> > > properly
> > > >> > > > >>>> introduce our change)
> > > >> > > > >>>>
> > > >> > > > >>>> LM4. Regarding the lifetime of the UUID: the KIP states
> we
> > > will
> > > >> > > > "Verify
> > > >> > > > >>>> that the UUID remains consistent across all subsequent
> > > >> heartbeats
> > > >> > > > >> during
> > > >> > > > >>>> the session". What is this "session" referring to here? I
> > > would
> > > >> > > expect
> > > >> > > > >>> that
> > > >> > > > >>>> the UUID is associated to a consumer instance (generated
> for
> > > >> the
> > > >> > > > >> consumer
> > > >> > > > >>>> the first time it needs to send a HB if it doesn't have
> the
> > > >> UUID
> > > >> > > yet.
> > > >> > > > >>> From
> > > >> > > > >>>> there on, every time it needs to send a "first HB"
> again, it
> > > >> will
> > > >> > > > reuse
> > > >> > > > >>> its
> > > >> > > > >>>> UUID, is that the intention? Note that we should consider
> > > that
> > > >> the
> > > >> > > > same
> > > >> > > > >>>> consumer instance may have many "first heartbeats",
> meaning
> > > >> > > heartbeats
> > > >> > > > >> to
> > > >> > > > >>>> join the group when it's not part of it (ex. consumer
> > > >> unsubscribe
> > > >> > +
> > > >> > > > >>>> subscribe, fenced, stale). Is this the intention or are
> you
> > > >> > > > considering
> > > >> > > > >>> the
> > > >> > > > >>>> lifetime differently? We should clarify it in the KIP.
> > > >> > > > >>>>
> > > >> > > > >>>> Thanks!
> > > >> > > > >>>>
> > > >> > > > >>>> Lianet
> > > >> > > > >>>>
> > > >> > > > >>>> On Tue, Aug 27, 2024 at 2:27 AM TengYao Chi <
> > > >> kitingiao@gmail.com>
> > > >> > > > >> wrote:
> > > >> > > > >>>>
> > > >> > > > >>>>> Hi everyone,
> > > >> > > > >>>>>
> > > >> > > > >>>>> I have revised this KIP multiple times based on the
> feedback
> > > >> from
> > > >> > > our
> > > >> > > > >>>>> discussions.
> > > >> > > > >>>>> I would greatly appreciate it if you could review it
> when
> > > you
> > > >> > have
> > > >> > > > >> the
> > > >> > > > >>>>> time.
> > > >> > > > >>>>> If there are no further comments or suggestions, I plan
> to
> > > >> > proceed
> > > >> > > > >> with
> > > >> > > > >>>>> initiating a vote soon.
> > > >> > > > >>>>>
> > > >> > > > >>>>> Best regards,
> > > >> > > > >>>>> TengYao
> > > >> > > > >>>>>
> > > >> > > > >>>>> TengYao Chi <ki...@gmail.com> 於 2024年8月23日 週五
> 下午2:43寫道：
> > > >> > > > >>>>>
> > > >> > > > >>>>>> Hi Andrew,
> > > >> > > > >>>>>> Thank you for your previous feedback and insights.
> > > >> > > > >>>>>> Your contributions have added valuable perspectives to
> the
> > > >> > > > >>> discussions.
> > > >> > > > >>>>>> And we also benefit from the comparison of different
> > > >> solutions.
> > > >> > > > >>>>>> I’m also looking forward to seeing an initial version
> in
> > > >> > KIP-932,
> > > >> > > > >> as
> > > >> > > > >>> it
> > > >> > > > >>>>>> will provide a good reference for future
> implementations.
> > > >> > > > >>>>>>
> > > >> > > > >>>>>> Regarding your comment on AS2, I wanted to clarify
> that my
> > > >> > > > >>>> specification
> > > >> > > > >>>>>> references org.apache.kafka.common.Uuid.
> > > >> > > > >>>>>> I believe we’re referring to the same class, and it
> might
> > > >> just
> > > >> > be
> > > >> > > a
> > > >> > > > >>>> small
> > > >> > > > >>>>>> oversight due to the busy schedule.
> > > >> > > > >>>>>>
> > > >> > > > >>>>>> I want to express my gratitude once again for your many
> > > >> > insightful
> > > >> > > > >>>>>> comments, which have helped the discussion progress
> > > smoothly.
> > > >> > > > >>>>>>
> > > >> > > > >>>>>> Best regards,
> > > >> > > > >>>>>> TengYao
> > > >> > > > >>>>>>
> > > >> > > > >>>>>>
> > > >> > > > >>>>>> Andrew Schofield <an...@live.com> 於
> 2024年8月22日
> > > 週四
> > > >> > > > >>>> 下午11:28寫道：
> > > >> > > > >>>>>>
> > > >> > > > >>>>>>> Hi TengYao,
> > > >> > > > >>>>>>> I’ve been reading through the comments and I’m happy
> that
> > > >> the
> > > >> > > > >> lobby
> > > >> > > > >>>>>>> approach has not gained support.
> > > >> > > > >>>>>>>
> > > >> > > > >>>>>>> Assuming that this KIP is voted, I will be happy to
> change
> > > >> > > KIP-932
> > > >> > > > >>> so
> > > >> > > > >>>>>>> that it only supports client-side member ID
> generation.
> > > >> Because
> > > >> > > > >> that
> > > >> > > > >>>> KIP
> > > >> > > > >>>>>>> is still
> > > >> > > > >>>>>>> under development, I can do this in the first version
> of
> > > >> > > > >>>>>>> ShareGroupHeartbeat.
> > > >> > > > >>>>>>>
> > > >> > > > >>>>>>> AS2: For the encoding section, I suppose the specific
> > > >> encoding
> > > >> > > > >> which
> > > >> > > > >>>>>>> is used is what org.apache.kafka.utils.Uuid uses.
> > > >> > > > >>>>>>>
> > > >> > > > >>>>>>> Thanks,
> > > >> > > > >>>>>>> Andrew
> > > >> > > > >>>>>>>
> > > >> > > > >>>>>>>> On 14 Aug 2024, at 17:00, TengYao Chi <
> > > kitingiao@gmail.com
> > > >> >
> > > >> > > > >>> wrote:
> > > >> > > > >>>>>>>>
> > > >> > > > >>>>>>>> Hello Apoorv,
> > > >> > > > >>>>>>>> Thank you for your feedback.
> > > >> > > > >>>>>>>> Regarding the questions you raised, unfortunately,
> this
> > > KIP
> > > >> > > > >> cannot
> > > >> > > > >>>>>>>> guarantee the order of heartbeats. As with many
> classic
> > > >> > > > >>> distributed
> > > >> > > > >>>>>>> system
> > > >> > > > >>>>>>>> challenges, what we can do is make our best effort to
> > > >> ensure
> > > >> > > > >> that
> > > >> > > > >>>>> there
> > > >> > > > >>>>>>> are
> > > >> > > > >>>>>>>> no idle members or stale assignments under normal
> > > >> > circumstances.
> > > >> > > > >>>>>>>>
> > > >> > > > >>>>>>>> As for the lobby approach, I’m not a fan of it
> because it
> > > >> > > > >> requires
> > > >> > > > >>>>>>> adding a
> > > >> > > > >>>>>>>> mechanism to maintain client state within the
> > > >> ConsumerGroup,
> > > >> > > > >>> which,
> > > >> > > > >>>> in
> > > >> > > > >>>>>>> my
> > > >> > > > >>>>>>>> view, resembles something like a two-phase commit.
> This
> > > >> would
> > > >> > > > >>>>> introduce
> > > >> > > > >>>>>>>> more complexity than the proposal in this KIP, which
> is
> > > >> > > > >> something
> > > >> > > > >>> we
> > > >> > > > >>>>>>> want
> > > >> > > > >>>>>>>> to avoid. KIP-848 aims to simplify the existing
> protocol,
> > > >> and
> > > >> > > > >>> while
> > > >> > > > >>>>> the
> > > >> > > > >>>>>>>> lobby approach is a good one, I believe it is not the
> > > right
> > > >> > fit
> > > >> > > > >>> for
> > > >> > > > >>>>> this
> > > >> > > > >>>>>>>> particular situation.
> > > >> > > > >>>>>>>>
> > > >> > > > >>>>>>>> Best regards,
> > > >> > > > >>>>>>>> TengYao
> > > >> > > > >>>>>>>>
> > > >> > > > >>>>>>>> TengYao Chi <ki...@gmail.com> 於 2024年8月14日 週三
> > > >> 下午11:45寫道：
> > > >> > > > >>>>>>>>
> > > >> > > > >>>>>>>>> Hi David,
> > > >> > > > >>>>>>>>>
> > > >> > > > >>>>>>>>> I really appreciate your review and suggestions. As
> I am
> > > >> > still
> > > >> > > > >>>>> gaining
> > > >> > > > >>>>>>>>> experience in writing KIPs, your input has been
> > > incredibly
> > > >> > > > >>>> helpful. I
> > > >> > > > >>>>>>> am
> > > >> > > > >>>>>>>>> currently applying your suggestions to the KIP and
> will
> > > >> > > > >> complete
> > > >> > > > >>> it
> > > >> > > > >>>>> as
> > > >> > > > >>>>>>> soon
> > > >> > > > >>>>>>>>> as possible.
> > > >> > > > >>>>>>>>> Regarding the UUID part, I think we haven’t reached
> a
> > > >> > > > >> conclusion
> > > >> > > > >>>>>>> yet.(So
> > > >> > > > >>>>>>>>> far according to this thread)
> > > >> > > > >>>>>>>>> However, I will review the current implementation
> in the
> > > >> > Kafka
> > > >> > > > >>>> `Uuid`
> > > >> > > > >>>>>>>>> class and include a brief specification in the KIP.
> > > >> > > > >>>>>>>>>
> > > >> > > > >>>>>>>>> Once again, thank you so much for your help.
> > > >> > > > >>>>>>>>>
> > > >> > > > >>>>>>>>> Best regards,
> > > >> > > > >>>>>>>>> TengYao
> > > >> > > > >>>>>>>>>
> > > >> > > > >>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月14日 週三
> > > >> > 下午11:14寫道：
> > > >> > > > >>>>>>>>>
> > > >> > > > >>>>>>>>>> hi Apoorv
> > > >> > > > >>>>>>>>>>
> > > >> > > > >>>>>>>>>>> As the memberId is now known to the client, and
> client
> > > >> > might
> > > >> > > > >>> send
> > > >> > > > >>>>> the
> > > >> > > > >>>>>>>>>> leave
> > > >> > > > >>>>>>>>>> group heartbeat on shutdown prior to receiving the
> > > >> initial
> > > >> > > > >>>> heartbeat
> > > >> > > > >>>>>>>>>> response. If that's true then how do we guarantee
> that
> > > >> the 2
> > > >> > > > >>>>> requests
> > > >> > > > >>>>>>> to
> > > >> > > > >>>>>>>>>> join and leave will be processed in order, which
> could
> > > >> still
> > > >> > > > >>> leave
> > > >> > > > >>>>>>> stale
> > > >> > > > >>>>>>>>>> members or throw unknown member id exceptions?
> > > >> > > > >>>>>>>>>>
> > > >> > > > >>>>>>>>>> This is definitely a good question. the short
> answer:
> > > no
> > > >> > > > >>> guarantee
> > > >> > > > >>>>> but
> > > >> > > > >>>>>>>>>> best
> > > >> > > > >>>>>>>>>> efforts
> > > >> > > > >>>>>>>>>>
> > > >> > > > >>>>>>>>>> Please notice the root cause is "we have no enough
> time
> > > >> to
> > > >> > > > >> wait
> > > >> > > > >>>>>>> member id
> > > >> > > > >>>>>>>>>> (response) when closing consumer". Sadly, we can'
> > > >> guarantee
> > > >> > > > >> the
> > > >> > > > >>>>>>> request
> > > >> > > > >>>>>>>>>> order due to the same reason.
> > > >> > > > >>>>>>>>>>
> > > >> > > > >>>>>>>>>> However, in contrast to previous behavior, there
> is one
> > > >> big
> > > >> > > > >>>> benefit
> > > >> > > > >>>>>>> of new
> > > >> > > > >>>>>>>>>> approach - we can try STONITH because we know the
> > > member
> > > >> id
> > > >> > > > >>>>>>>>>>
> > > >> > > > >>>>>>>>>> Best,
> > > >> > > > >>>>>>>>>> Chia-Ping
> > > >> > > > >>>>>>>>>>
> > > >> > > > >>>>>>>>>>
> > > >> > > > >>>>>>>>>> Apoorv Mittal <ap...@gmail.com> 於
> 2024年8月14日
> > > 週三
> > > >> > > > >>>> 下午8:55寫道：
> > > >> > > > >>>>>>>>>>
> > > >> > > > >>>>>>>>>>> Hi TengYao,
> > > >> > > > >>>>>>>>>>> Thanks for the KIP. Continuing on the point which
> > > Andrew
> > > >> > > > >>>> mentioned
> > > >> > > > >>>>> as
> > > >> > > > >>>>>>>>>> AS1.
> > > >> > > > >>>>>>>>>>>
> > > >> > > > >>>>>>>>>>> As the memberId is now known to the client, and
> client
> > > >> > might
> > > >> > > > >>> send
> > > >> > > > >>>>> the
> > > >> > > > >>>>>>>>>> leave
> > > >> > > > >>>>>>>>>>> group heartbeat on shutdown prior to receiving the
> > > >> initial
> > > >> > > > >>>>> heartbeat
> > > >> > > > >>>>>>>>>>> response. If that's true then how do we guarantee
> that
> > > >> the
> > > >> > 2
> > > >> > > > >>>>>>> requests to
> > > >> > > > >>>>>>>>>>> join and leave will be processed in order, which
> could
> > > >> > still
> > > >> > > > >>>> leave
> > > >> > > > >>>>>>> stale
> > > >> > > > >>>>>>>>>>> members or throw unknown member id exceptions?
> > > >> > > > >>>>>>>>>>>
> > > >> > > > >>>>>>>>>>> Though the client side member id generation is
> helpful
> > > >> > which
> > > >> > > > >>> will
> > > >> > > > >>>>>>>>>> represent
> > > >> > > > >>>>>>>>>>> the same group perspective as from client and
> broker's
> > > >> end.
> > > >> > > > >>> But I
> > > >> > > > >>>>>>> think
> > > >> > > > >>>>>>>>>> the
> > > >> > > > >>>>>>>>>>> major concern we want to solve here is Stale
> Partition
> > > >> > > > >>>> Assignments
> > > >> > > > >>>>>>> which
> > > >> > > > >>>>>>>>>>> might still exist with the new approach. I am
> leaning
> > > >> > towards
> > > >> > > > >>> the
> > > >> > > > >>>>>>>>>>> suggestion mentioned by Andrew where partition
> > > >> assignment
> > > >> > > > >>>> triggers
> > > >> > > > >>>>> on
> > > >> > > > >>>>>>>>>>> subsequent heartbeat when client acknowledges the
> > > >> initial
> > > >> > > > >>>>> heartbeat,
> > > >> > > > >>>>>>>>>>> delayed partition assignment.
> > > >> > > > >>>>>>>>>>>
> > > >> > > > >>>>>>>>>>> Though on a separate note, I have a different
> > > question.
> > > >> > What
> > > >> > > > >>>>> happens
> > > >> > > > >>>>>>>>>> when
> > > >> > > > >>>>>>>>>>> there is an issue with the client which sends the
> > > >> initial
> > > >> > > > >>>> heartbeat
> > > >> > > > >>>>>>>>>> without
> > > >> > > > >>>>>>>>>>> memberId, then crashes and restarts? I think we
> must
> > > be
> > > >> > > > >>>>> re-triggering
> > > >> > > > >>>>>>>>>>> assignments and expiring members only after the
> > > >> heartbeat
> > > >> > > > >>> session
> > > >> > > > >>>>>>>>>> timeout?
> > > >> > > > >>>>>>>>>>> If that's true then shall delayed partition
> assignment
> > > >> can
> > > >> > > > >> help
> > > >> > > > >>>>>>> benefit
> > > >> > > > >>>>>>>>>> us
> > > >> > > > >>>>>>>>>>> from this situation as well?
> > > >> > > > >>>>>>>>>>>
> > > >> > > > >>>>>>>>>>> Regards,
> > > >> > > > >>>>>>>>>>> Apoorv Mittal
> > > >> > > > >>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>
> > > >> > > > >>>>>>>>>>> On Wed, Aug 14, 2024 at 12:51 PM David Jacot
> > > >> > > > >>>>>>>>>> <dj...@confluent.io.invalid>
> > > >> > > > >>>>>>>>>>> wrote:
> > > >> > > > >>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>> Hi Andrew,
> > > >> > > > >>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>> Personally, I don't like the lobby approach. It
> makes
> > > >> > things
> > > >> > > > >>>> more
> > > >> > > > >>>>>>>>>>>> complicated and it would require changing the
> records
> > > >> on
> > > >> > the
> > > >> > > > >>>>> server
> > > >> > > > >>>>>>>>>> too.
> > > >> > > > >>>>>>>>>>>> This is why I initially suggested the rejected
> > > >> alternative
> > > >> > > > >> #2
> > > >> > > > >>>>> which
> > > >> > > > >>>>>>> is
> > > >> > > > >>>>>>>>>>>> pretty close but also not perfect.
> > > >> > > > >>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>> I'd like to clarify one thing. The
> > > >> ConsumerGroupHeartbeat
> > > >> > > > >> API
> > > >> > > > >>>>>>> already
> > > >> > > > >>>>>>>>>>>> supports generating the member id on the client
> so we
> > > >> > don't
> > > >> > > > >>> need
> > > >> > > > >>>>> any
> > > >> > > > >>>>>>>>>>>> conditional logic on the client side. This is
> > > actually
> > > >> > what
> > > >> > > > >> we
> > > >> > > > >>>>>>> wanted
> > > >> > > > >>>>>>>>>> to
> > > >> > > > >>>>>>>>>>> do
> > > >> > > > >>>>>>>>>>>> in the first place but the idea got pushed back
> by
> > > >> Magnus
> > > >> > > > >> back
> > > >> > > > >>>>> then
> > > >> > > > >>>>>>>>>>> because
> > > >> > > > >>>>>>>>>>>> generating uuid from librdkafka required a new
> > > >> dependency.
> > > >> > > > >> It
> > > >> > > > >>>>> turns
> > > >> > > > >>>>>>>>>> out
> > > >> > > > >>>>>>>>>>>> that librdkafka has that dependency today. In
> > > >> retrospect,
> > > >> > we
> > > >> > > > >>>>> should
> > > >> > > > >>>>>>>>>> have
> > > >> > > > >>>>>>>>>>>> pushed back on this. Long story short, we can
> just do
> > > >> it.
> > > >> > > > >> The
> > > >> > > > >>>>>>>>>> proposal in
> > > >> > > > >>>>>>>>>>>> this KIP is to make the member id required in
> future
> > > >> > > > >> versions.
> > > >> > > > >>>> We
> > > >> > > > >>>>>>>>>> could
> > > >> > > > >>>>>>>>>>>> also decide not to do it and to keep supporting
> both
> > > >> > > > >>>> approaches. I
> > > >> > > > >>>>>>>>>> would
> > > >> > > > >>>>>>>>>>>> also be fine with this.
> > > >> > > > >>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>> Best,
> > > >> > > > >>>>>>>>>>>> David
> > > >> > > > >>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>> On Wed, Aug 14, 2024 at 12:30 PM Andrew
> Schofield <
> > > >> > > > >>>>>>>>>>>> andrew_schofield@live.com>
> > > >> > > > >>>>>>>>>>>> wrote:
> > > >> > > > >>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>> Hi TengYao,
> > > >> > > > >>>>>>>>>>>>> Thanks for your response. I’ll have just one
> more
> > > try
> > > >> to
> > > >> > > > >>>>> persuade.
> > > >> > > > >>>>>>>>>>>>> I feel that I will need to follow the approach
> with
> > > >> > KIP-932
> > > >> > > > >>>> when
> > > >> > > > >>>>>>>>>> we’ve
> > > >> > > > >>>>>>>>>>>>> made a decision, so I do have more than a
> passing
> > > >> > interest
> > > >> > > > >> in
> > > >> > > > >>>>> this.
> > > >> > > > >>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>> A group member in the lobby is in the group,
> but it
> > > >> does
> > > >> > > > >> not
> > > >> > > > >>>> have
> > > >> > > > >>>>>>>>>> any
> > > >> > > > >>>>>>>>>>>>> assignments. A member of a consumer group can
> have
> > > no
> > > >> > > > >>> assigned
> > > >> > > > >>>>>>>>>>>>> partitions (such as 5 CG members subscribed to a
> > > topic
> > > >> > > > >> with 4
> > > >> > > > >>>>>>>>>>>> partitions),
> > > >> > > > >>>>>>>>>>>>> so it’s a situation that consumer group members
> > > >> already
> > > >> > > > >>> expect.
> > > >> > > > >>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>> One of Kafka’s strengths is the way that we
> handle
> > > API
> > > >> > > > >>>>> versioning.
> > > >> > > > >>>>>>>>>>>>> But, there is a cost - the behaviour is
> different
> > > >> > depending
> > > >> > > > >>> on
> > > >> > > > >>>>> the
> > > >> > > > >>>>>>>>>> RPC
> > > >> > > > >>>>>>>>>>>>> version. KIP-848 is on the cusp of completion,
> but
> > > >> we’re
> > > >> > > > >>>> already
> > > >> > > > >>>>>>>>>> adding
> > > >> > > > >>>>>>>>>>>>> conditional logic for v0/v1 for
> > > >> ConsumerGroupHeartbeat.
> > > >> > > > >>> That’s
> > > >> > > > >>>> a
> > > >> > > > >>>>>>>>>> pity.
> > > >> > > > >>>>>>>>>>>>> Only a minor issue, but it’s unfortunate.
> > > >> > > > >>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>> Thanks,
> > > >> > > > >>>>>>>>>>>>> Andrew
> > > >> > > > >>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>> On 14 Aug 2024, at 08:47, TengYao Chi <
> > > >> > > > >> kitingiao@gmail.com>
> > > >> > > > >>>>>>>>>> wrote:
> > > >> > > > >>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>> Hello Andrew
> > > >> > > > >>>>>>>>>>>>>> Thank you for your thoughtful suggestions and
> > > getting
> > > >> > the
> > > >> > > > >>>>>>>>>> discussion
> > > >> > > > >>>>>>>>>>>>> going.
> > > >> > > > >>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>> To AS1:
> > > >> > > > >>>>>>>>>>>>>> In the current scenario where the server
> generates
> > > >> the
> > > >> > > > >> UUID,
> > > >> > > > >>>> if
> > > >> > > > >>>>>>>>>> the
> > > >> > > > >>>>>>>>>>>>> client
> > > >> > > > >>>>>>>>>>>>>> shuts down before receiving the memberId
> generated
> > > by
> > > >> > the
> > > >> > > > >> GC
> > > >> > > > >>>>>>>>>>>> (regardless
> > > >> > > > >>>>>>>>>>>>> of
> > > >> > > > >>>>>>>>>>>>>> whether it’s a graceful shutdown or not), the
> GC
> > > will
> > > >> > > > >> still
> > > >> > > > >>>> have
> > > >> > > > >>>>>>>>>> to
> > > >> > > > >>>>>>>>>>>> wait
> > > >> > > > >>>>>>>>>>>>>> for the heartbeat timeout because the client
> > > doesn’t
> > > >> > know
> > > >> > > > >>> its
> > > >> > > > >>>>>>>>>>> memberId.
> > > >> > > > >>>>>>>>>>>>>> This KIP indeed cannot completely resolve the
> > > >> > idempotency
> > > >> > > > >>>> issue,
> > > >> > > > >>>>>>>>>> but
> > > >> > > > >>>>>>>>>>> it
> > > >> > > > >>>>>>>>>>>>> can
> > > >> > > > >>>>>>>>>>>>>> better handle shutdown scenarios under normal
> > > >> > > > >> circumstances
> > > >> > > > >>>>>>>>>> because
> > > >> > > > >>>>>>>>>>> the
> > > >> > > > >>>>>>>>>>>>>> client always knows its memberId. Even if the
> > > client
> > > >> > shuts
> > > >> > > > >>>> down
> > > >> > > > >>>>>>>>>>>>> immediately
> > > >> > > > >>>>>>>>>>>>>> after the initial heartbeat, as long as it
> > > performs a
> > > >> > > > >>> graceful
> > > >> > > > >>>>>>>>>>> shutdown
> > > >> > > > >>>>>>>>>>>>> and
> > > >> > > > >>>>>>>>>>>>>> sends a leave heartbeat, the GC can manage the
> > > >> situation
> > > >> > > > >> and
> > > >> > > > >>>>>>>>>> remove
> > > >> > > > >>>>>>>>>>> the
> > > >> > > > >>>>>>>>>>>>>> member. Therefore, the goal of this KIP is to
> > > address
> > > >> > the
> > > >> > > > >>>> issue
> > > >> > > > >>>>>>>>>> where
> > > >> > > > >>>>>>>>>>>> the
> > > >> > > > >>>>>>>>>>>>>> GC has to wait for the heartbeat timeout due
> to the
> > > >> > client
> > > >> > > > >>>>> leaving
> > > >> > > > >>>>>>>>>>>>> without
> > > >> > > > >>>>>>>>>>>>>> knowing its memberId, which leads to reduced
> > > >> throughput
> > > >> > > > >> and
> > > >> > > > >>>>>>>>>> limited
> > > >> > > > >>>>>>>>>>>>>> scalability.
> > > >> > > > >>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>> The solution you suggest has also been
> proposed by
> > > >> > David.
> > > >> > > > >>> The
> > > >> > > > >>>>>>>>>> concern
> > > >> > > > >>>>>>>>>>>>> with
> > > >> > > > >>>>>>>>>>>>>> this approach is that it introduces additional
> > > >> > complexity
> > > >> > > > >>> for
> > > >> > > > >>>>>>>>>>>>>> compatibility, as the new server would not
> > > >> immediately
> > > >> > add
> > > >> > > > >>> the
> > > >> > > > >>>>>>>>>> member
> > > >> > > > >>>>>>>>>>>> to
> > > >> > > > >>>>>>>>>>>>>> the group, while the old server would. This
> > > requires
> > > >> > > > >> clients
> > > >> > > > >>>> to
> > > >> > > > >>>>>>>>>>>>>> differentiate whether their memberId has been
> added
> > > >> to
> > > >> > the
> > > >> > > > >>>> group
> > > >> > > > >>>>>>>>>> or
> > > >> > > > >>>>>>>>>>>> not,
> > > >> > > > >>>>>>>>>>>>>> which could result in unexpected logs.
> > > >> > > > >>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>> Best Regards,
> > > >> > > > >>>>>>>>>>>>>> TengYao
> > > >> > > > >>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>> Andrew Schofield <an...@live.com> 於
> > > >> > 2024年8月14日
> > > >> > > > >>> 週三
> > > >> > > > >>>>>>>>>>>> 上午12:29寫道：
> > > >> > > > >>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>>> Hi TengYao,
> > > >> > > > >>>>>>>>>>>>>>> Thanks for the KIP. I wonder if there’s a
> > > different
> > > >> way
> > > >> > > > >> to
> > > >> > > > >>>>> close
> > > >> > > > >>>>>>>>>>> what
> > > >> > > > >>>>>>>>>>>>>>> is quite a small window.
> > > >> > > > >>>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>>> AS1: It is true that the initial heartbeat is
> not
> > > >> > > > >>> idempotent,
> > > >> > > > >>>>> but
> > > >> > > > >>>>>>>>>>> this
> > > >> > > > >>>>>>>>>>>>>>> remains
> > > >> > > > >>>>>>>>>>>>>>> true with this KIP. It’s just differently not
> > > >> > idempotent.
> > > >> > > > >>> If
> > > >> > > > >>>>> the
> > > >> > > > >>>>>>>>>>>> client
> > > >> > > > >>>>>>>>>>>>>>> makes its
> > > >> > > > >>>>>>>>>>>>>>> own member ID, sends a request and dies, the
> GC
> > > will
> > > >> > > > >> still
> > > >> > > > >>>> have
> > > >> > > > >>>>>>>>>>> added
> > > >> > > > >>>>>>>>>>>>>>> the member to the group and it will hang
> around
> > > >> until
> > > >> > the
> > > >> > > > >>>>> session
> > > >> > > > >>>>>>>>>>>>> expires.
> > > >> > > > >>>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>>> I wonder if the GC could still generate the
> member
> > > >> ID
> > > >> > in
> > > >> > > > >>>>>>>>>> response to
> > > >> > > > >>>>>>>>>>>> the
> > > >> > > > >>>>>>>>>>>>>>> first
> > > >> > > > >>>>>>>>>>>>>>> heartbeat, and put the member in a special
> PENDING
> > > >> > state
> > > >> > > > >>> with
> > > >> > > > >>>>> no
> > > >> > > > >>>>>>>>>>>>>>> assignments until the client sends the next
> > > >> heartbeat,
> > > >> > > > >> thus
> > > >> > > > >>>>>>>>>>> confirming
> > > >> > > > >>>>>>>>>>>>> it
> > > >> > > > >>>>>>>>>>>>>>> has received the member ID. This would not be
> a
> > > >> > protocol
> > > >> > > > >>>> change
> > > >> > > > >>>>>>>>>> at
> > > >> > > > >>>>>>>>>>>> all,
> > > >> > > > >>>>>>>>>>>>>>> just
> > > >> > > > >>>>>>>>>>>>>>> a change to the GC to keep a new member in the
> > > lobby
> > > >> > > > >> until
> > > >> > > > >>>> it’s
> > > >> > > > >>>>>>>>>>>>> comfirmed
> > > >> > > > >>>>>>>>>>>>>>> it knows its member ID.
> > > >> > > > >>>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>>> Thanks,
> > > >> > > > >>>>>>>>>>>>>>> Andrew
> > > >> > > > >>>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>>>> On 13 Aug 2024, at 15:59, TengYao Chi <
> > > >> > > > >>> kitingiao@gmail.com>
> > > >> > > > >>>>>>>>>> wrote:
> > > >> > > > >>>>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>>>> Hi Chia-Ping,
> > > >> > > > >>>>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>>>> Thanks for review and suggestions.
> > > >> > > > >>>>>>>>>>>>>>>> I have updated the content of KIP
> accordingly.
> > > >> > > > >>>>>>>>>>>>>>>> Please take a look.
> > > >> > > > >>>>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>>>> Best regards,
> > > >> > > > >>>>>>>>>>>>>>>> TengYao
> > > >> > > > >>>>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > > 2024年8月13日
> > > >> 週二
> > > >> > > > >>>>> 下午9:45寫道：
> > > >> > > > >>>>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>>>>> hi TengYao
> > > >> > > > >>>>>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>>>>> thanks for this KIP.
> > > >> > > > >>>>>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>>>>> 1) could you please describe the
> before/after
> > > >> > behavior
> > > >> > > > >> in
> > > >> > > > >>>> the
> > > >> > > > >>>>>>>>>>>>> "Proposed
> > > >> > > > >>>>>>>>>>>>>>>>> Changes" section? IIRC, current RPC allows
> HB
> > > >> having
> > > >> > > > >>> member
> > > >> > > > >>>>> id
> > > >> > > > >>>>>>>>>>>>>>> generated by
> > > >> > > > >>>>>>>>>>>>>>>>> client, right? If HB has no member ID,
> server
> > > will
> > > >> > > > >>> generate
> > > >> > > > >>>>> one
> > > >> > > > >>>>>>>>>>> and
> > > >> > > > >>>>>>>>>>>>> then
> > > >> > > > >>>>>>>>>>>>>>>>> return. The new behavior will enforce HB
> "must"
> > > >> have
> > > >> > > > >>> member
> > > >> > > > >>>>> ID.
> > > >> > > > >>>>>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>>>>> 2) could you please write the version number
> > > >> > explicitly
> > > >> > > > >>> in
> > > >> > > > >>>>> the
> > > >> > > > >>>>>>>>>> KIP
> > > >> > > > >>>>>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>>>>> 3) how new client code handle the old HB?
> Does
> > > it
> > > >> > > > >> always
> > > >> > > > >>>>>>>>>> generate
> > > >> > > > >>>>>>>>>>>>> member
> > > >> > > > >>>>>>>>>>>>>>>>> ID on client-side even though that is not
> > > >> restricted?
> > > >> > > > >>>>>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>>>>> Best,
> > > >> > > > >>>>>>>>>>>>>>>>> Chia-Ping
> > > >> > > > >>>>>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>>>>> On 2024/08/13 06:20:42 TengYao Chi wrote:
> > > >> > > > >>>>>>>>>>>>>>>>>> Hello everyone,
> > > >> > > > >>>>>>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>>>>>> I would like to start a discussion thread
> on
> > > >> > KIP-1082,
> > > >> > > > >>>> which
> > > >> > > > >>>>>>>>>>>> proposes
> > > >> > > > >>>>>>>>>>>>>>>>>> enabling id generation for clients over the
> > > >> > > > >>>>>>>>>>> ConsumerGroupHeartbeat
> > > >> > > > >>>>>>>>>>>>> RPC.
> > > >> > > > >>>>>>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>>>>>> Here is the KIP Link: KIP-1082
> > > >> > > > >>>>>>>>>>>>>>>>>> <
> > > >> > > > >>>>>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>
> > > >> > > > >>>>>>>>>>
> > > >> > > > >>>>>>>
> > > >> > > > >>>>>
> > > >> > > > >>>>
> > > >> > > > >>>
> > > >> > > > >>
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
> > > >> > > > >>>>>>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>>>>>> Please take a look and let me know what you
> > > >> think,
> > > >> > > > >> and I
> > > >> > > > >>>>> would
> > > >> > > > >>>>>>>>>>>>>>> appreciate
> > > >> > > > >>>>>>>>>>>>>>>>>> any suggestions and feedback.
> > > >> > > > >>>>>>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>>>>>> Best regards,
> > > >> > > > >>>>>>>>>>>>>>>>>> TengYao
> > > >> > > > >>>>>>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>>
> > > >> > > > >>>>>>>>>>>
> > > >> > > > >>>>>>>>>>
> > > >> > > > >>>>>>>>>
> > > >> > > > >>>>>>>
> > > >> > > > >>>>>>>
> > > >> > > > >>>>>
> > > >> > > > >>>>
> > > >> > > > >>>
> > > >> > > > >>
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[Chia-Ping Tsai <ch...@apache.org>]

> This part is not clear either. It basically says that if a member joins with an existing member id but a different epoch, it will be fenced. Then it must rejoin with the same member id and epoch zero. This is already the current behavior and it does not help with detecting duplicates, right?

The duplicates issue is interesting. The member id is generated on server-side by UUID before, so it seems to me the fenced member epoch happens only if the client miss the response with bumped epoch, and this case is recoverable.

however, v1 brings a different fenced epoch scenario. Users, now, have responsibility to generate unique member id, hence clients may encounter infinite fenced error if there are >1 clients having same member id due to config error (or other engineer error)

Maybe we can highlight this scenario if we start to requires clients to generate unique member id.

Best,
Chia-Ping

On 2024/09/19 18:36:48 David Jacot wrote:
> Hi,
> 
> Thanks for the update. I have a few nits:
> 
> > If the member ID is null or empty, the server will reject the request
> with an InvalidRequestException.
> We should clarify that this should only apply to version >= 1.
> 
> > The consumer instance must generate a member ID, and this ID should
> remain consistent for the duration of the consumer's session. Here, a
> "session" is defined as the period from the consumer's first heartbeat
> until it leaves the group, either through a graceful shutdown, a heartbeat
> timeout, or the process stopping or dying. The consumer instance should
> reuse the same member ID for all heartbeats and rejoin attempts to maintain
> continuity within the group.
> 
> This part is not clear to me. When the member leaves the group, it should
> not reset the member id. I would rather say that the member must generate
> its member id when it starts and it must keep it until the process stops.
> It is basically an incarnation of the process.
> 
> > If a conflict arises where the member ID generated by the client is
> detected to be a duplicate within the same group (for example, the same
> member ID is associated with another active member in the group), the
> server will handle this by comparing the memberEpoch values of the
> conflicting members. The member with the lower memberEpoch is considered
> outdated and will be fenced off by the server. When this occurs, the server
> responds with a FENCED_MEMBER_EPOCH error to the client, signaling it to
> rejoin the group with the same member ID while resetting the memberEpoch to
> zero. This ensures that the client properly resynchronizes and maintains
> the continuity and consistency of the group membership.
> 
> This part is not clear either. It basically says that if a member joins
> with an existing member id but a different epoch, it will be fenced. Then
> it must rejoin with the same member id and epoch zero. This is already the
> current behavior and it does not help with detecting duplicates, right?
> Should we just remove the paragraph?
> 
> > A member ID mismatch occurs within a session: If the server detects a
> mismatch between the provided member ID and the expected member ID for an
> ongoing session, it should return a UNKNOWN_MEMBER_ID  error.
> 
> How could we detect a mismatch between the provided and the expected member
> id? My understanding is that we can only know whether the provided member
> id exists or not. This is already implemented.
> 
> Thanks,
> David
> 
> On Sat, Sep 14, 2024 at 9:31 AM TengYao Chi <ki...@gmail.com> wrote:
> 
> > Hello everyone,
> >
> > Since this KIP has been fully discussed, I will initiate a vote for it next
> > Monday.
> > Thank you and have a nice weekend.
> >
> > Best regards,
> > TengYao
> >
> > TengYao Chi <ki...@gmail.com> 於 2024年9月5日 週四 下午2:19寫道：
> >
> > > Hello everyone,
> > >
> > > KT2: It looks like everyone who has expressed an opinion supports the
> > > second option: “Document a recommendation for clients to use UUIDs as
> > > member IDs, without strictly enforcing it.”
> > > I have updated the KIP accordingly.
> > > Please take a look, and let me know if you have any thoughts or feedback.
> > >
> > > Thank you!
> > >
> > > Best regards,
> > > TengYao
> > >
> > > Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月30日 週五 下午9:56寫道：
> > >
> > >> hi TengYao
> > >>
> > >> KT2: +1 to second approach
> > >>
> > >> Best,
> > >> Chia-Ping
> > >>
> > >>
> > >> David Jacot <dj...@confluent.io.invalid> 於 2024年8月30日 週五 下午9:15寫道：
> > >>
> > >> > Hi TengYao,
> > >> >
> > >> > KT2: I don't think that we can realistically validate the uuid on the
> > >> > server. It is basically a string of chars. So I lean towards having a
> > >> good
> > >> > recommendation in the KIP and in the document of the field in the
> > RPC's
> > >> > definition.
> > >> >
> > >> > Best,
> > >> > David
> > >> >
> > >> > On Fri, Aug 30, 2024 at 3:02 PM TengYao Chi <ki...@gmail.com>
> > >> wrote:
> > >> >
> > >> > > Hello Kirk !
> > >> > >
> > >> > > Thank you for your comments !
> > >> > >
> > >> > > KT1: Yes, you are correct. The issue is not unique to the initial
> > >> > > heartbeat; there can always be cases where the broker might lose
> > >> > connection
> > >> > > with a member.
> > >> > >
> > >> > > KT2: Currently, if the client doesn't have a member ID and the
> > >> > memberEpoch
> > >> > > equals 0, the coordinator will generate a UUID as the member ID for
> > >> the
> > >> > > client. However, at the RPC level, the member ID is sent as a
> > literal
> > >> > > string, meaning there are no restrictions on the format at this
> > level.
> > >> > > This also reminds me that we haven't reached a final conclusion on
> > >> how to
> > >> > > enforce the use of UUIDs.
> > >> > > From our previous discussions, I recall two possible approaches:
> > >> > > The first is to validate the UUID on the server side, and if it's
> > not
> > >> > > valid, throw an exception to the client.
> > >> > > The second is to document a recommendation for clients to use UUIDs
> > as
> > >> > > member IDs, without strictly enforcing it.
> > >> > > I think it's time to decide on the approach we want to take.
> > >> > >
> > >> > > KT3: Yes, "session" can be considered synonymous with "membership"
> > in
> > >> > this
> > >> > > context.
> > >> > >
> > >> > > KT4: Thank you for pointing that out. I will update the wording to
> > >> > > specifically say this behavior is for consumers.
> > >> > >
> > >> > > Thanks again for your comments.
> > >> > >
> > >> > > Best regards,
> > >> > > TengYao
> > >> > >
> > >> > > Kirk True <ki...@kirktrue.pro> 於 2024年8月30日 週五 上午12:39寫道：
> > >> > >
> > >> > > > Hi TengYao!
> > >> > > >
> > >> > > > Sorry for being late to the discussion...
> > >> > > >
> > >> > > > After reading the thread and then the KIP, I had a few
> > >> > > questions/comments:
> > >> > > >
> > >> > > > KT1: In Motivation, it states: "This scenario can result in the
> > >> broker
> > >> > > > registering a new member for which it will never receive a proper
> > >> leave
> > >> > > > request.” Just to be clear, the broker will always have cases
> > where
> > >> it
> > >> > > > might lose connection with a member. That’s not unique to the
> > >> initial
> > >> > > > heartbeat, right?
> > >> > > >
> > >> > > > KT2: There was a bit of back and forth about format of the member
> > >> ID.
> > >> > > From
> > >> > > > what I gathered in the thread, the member ID is still defined in
> > the
> > >> > RPC
> > >> > > as
> > >> > > > a string and not a UUID, right? The KIP states that the “client
> > must
> > >> > > > generate a UUID as the member ID” and that the “server will
> > validate
> > >> > > that a
> > >> > > > valid UUID is provided.” Is that a change for the server, or is it
> > >> > > already
> > >> > > > enforced as a UUID?
> > >> > > >
> > >> > > > KT3: Lianet mentioned some confusion over the use of the word
> > >> > “session.”
> > >> > > > Isn’t “session” synonymous with “membership?”
> > >> > > >
> > >> > > > KT4: Under “Member ID Lifecycle,” it states: "The client should
> > >> reuse
> > >> > the
> > >> > > > same UUID as the member ID for all heartbeats and rejoin attempts
> > to
> > >> > > > maintain continuity within the group.” Could we change the first
> > >> part
> > >> > of
> > >> > > > that to “The Consumer instance should…” We do have lifetimes that
> > >> > extend
> > >> > > > past the lifetime of a client instance (such as the transaction
> > ID).
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Kirk
> > >> > > >
> > >> > > > > On Aug 29, 2024, at 1:28 AM, TengYao Chi <ki...@gmail.com>
> > >> > wrote:
> > >> > > > >
> > >> > > > > Hi David,
> > >> > > > >
> > >> > > > > Thank you for pointing that out.
> > >> > > > > I have updated the content of the KIP based on Lianet's and your
> > >> > > > feedback.
> > >> > > > > Please take a look and let me know your thoughts.
> > >> > > > >
> > >> > > > > Best regards,
> > >> > > > > TengYao
> > >> > > > >
> > >> > > > > David Jacot <dj...@confluent.io.invalid> 於 2024年8月29日 週四
> > >> 下午3:20寫道：
> > >> > > > >
> > >> > > > >> Hi TengYao,
> > >> > > > >>
> > >> > > > >> Thanks for the update. I haven't fully read it yet but I will
> > >> soon.
> > >> > > > >>
> > >> > > > >> LM4: This is incorrect. The consumer must keep its member id
> > >> during
> > >> > > its
> > >> > > > >> entire lifetime (until the process stops or dies). The protocol
> > >> > > > stipulates
> > >> > > > >> that a member must rejoin with the same member id and the
> > member
> > >> > epoch
> > >> > > > set
> > >> > > > >> to zero when an FENCED_MEMBER_EPOCH occurs. This allows the
> > >> member
> > >> > to
> > >> > > > >> resynchronize itself. We should not change this behavior. I
> > think
> > >> > that
> > >> > > > we
> > >> > > > >> should see the client side generation id as an incarnation id
> > of
> > >> the
> > >> > > > >> application. It is generated once and kept until it stops or
> > >> dies.
> > >> > > > >>
> > >> > > > >> Best,
> > >> > > > >> David
> > >> > > > >>
> > >> > > > >> On Thu, Aug 29, 2024 at 6:21 AM TengYao Chi <
> > kitingiao@gmail.com
> > >> >
> > >> > > > wrote:
> > >> > > > >>
> > >> > > > >>> Hello Lianet !
> > >> > > > >>>
> > >> > > > >>> Thanks for the reviews and suggestions!
> > >> > > > >>>
> > >> > > > >>> LM1: Indeed, we plan to enforce client-side ID generation in
> > the
> > >> > > > future,
> > >> > > > >>> and it is not an alternative. I will change the title
> > >> accordingly.
> > >> > > > >>>
> > >> > > > >>> LM2: Yes, that's the expectation. I will add that statement to
> > >> the
> > >> > > > public
> > >> > > > >>> interface section.
> > >> > > > >>>
> > >> > > > >>> LM3: Thank you for the high-level perspective review. I think
> > >> > you're
> > >> > > > >> right;
> > >> > > > >>> our intention isn't very clear since it was placed at the end
> > of
> > >> > the
> > >> > > > >>> section. I will try to rephrase that section to make it more
> > >> > obvious.
> > >> > > > >>>
> > >> > > > >>> LM4: Regarding the definition of "session" in this KIP, I
> > >> believe
> > >> > it
> > >> > > > >> refers
> > >> > > > >>> to the period between the *first-time heartbeat* and when the
> > >> > > *consumer
> > >> > > > >>> leaves the group* (whether through a graceful shutdown or a
> > >> > heartbeat
> > >> > > > >>> timeout). The consumer should reuse its UUID if it has been
> > >> > generated
> > >> > > > >>> before. The only situation in which it will regenerate the
> > UUID
> > >> is
> > >> > if
> > >> > > > the
> > >> > > > >>> coordinator finds that there is already a consumer with the
> > same
> > >> > > UUID.
> > >> > > > >>> IIRC, the coordinator should compare the member epochs, and
> > the
> > >> > > > >>> later-joined consumer should be fenced off by the coordinator
> > >> due
> > >> > to
> > >> > > > >> having
> > >> > > > >>> a lower member epoch. Once the consumer receives a
> > >> > > > `FENCED_MEMBER_EPOCH`
> > >> > > > >>> error, it will generate a new UUID and attempt to rejoin. I
> > will
> > >> > > > clarify
> > >> > > > >>> this in the KIP.
> > >> > > > >>>
> > >> > > > >>> Thanks again for your reviews, I really appreciate it.
> > >> > > > >>>
> > >> > > > >>> Best regards,
> > >> > > > >>> TengYao
> > >> > > > >>>
> > >> > > > >>> Lianet M. <li...@gmail.com> 於 2024年8月28日 週三 下午7:12寫道：
> > >> > > > >>>
> > >> > > > >>>> Hello TengYao! Thanks for taking on this issue, we've been
> > >> going
> > >> > > > around
> > >> > > > >>> it
> > >> > > > >>>> for a while.
> > >> > > > >>>>
> > >> > > > >>>> LM1: About the title of the KIP: "Enable ID Generation for
> > >> Clients
> > >> > > > over
> > >> > > > >>> the
> > >> > > > >>>> ConsumerGroupHeartbeat RPC". I find it confusing because it
> > >> hints
> > >> > > that
> > >> > > > >>>> we're adding it as an alternative (which was discussed and
> > >> > > discarded,
> > >> > > > >> in
> > >> > > > >>>> favour of really enforcing it). It's also missing the core
> > >> change
> > >> > > imo,
> > >> > > > >>>> which is "where" the generation happens. So, maybe more to
> > the
> > >> > point
> > >> > > > >> with
> > >> > > > >>>> something along the lines of "Client-side generated ID for
> > >> clients
> > >> > > > over
> > >> > > > >>>> ConsumerGroupHeartbeat RPC"?
> > >> > > > >>>>
> > >> > > > >>>> LM2: On the public interfaces section, the KIP states that
> > "the
> > >> > > server
> > >> > > > >>> will
> > >> > > > >>>> reject the request", but we should agree on the specific
> > error
> > >> > > type. I
> > >> > > > >>>> expect it should fail with an InvalidRequestException, is
> > that
> > >> the
> > >> > > > >>>> intention? (This was also suggested in the discussion thread
> > >> > before
> > >> > > > but
> > >> > > > >>> is
> > >> > > > >>>> not in the KIP).
> > >> > > > >>>>
> > >> > > > >>>> LM3. Related to my previous point, I find that to be the true
> > >> > > > >>> public-facing
> > >> > > > >>>> change (member ID mandatory at the protocol level), but it's
> > >> only
> > >> > at
> > >> > > > >> the
> > >> > > > >>>> end of the Public interfaces changes, kind of lost among
> > >> details
> > >> > of
> > >> > > > how
> > >> > > > >>>> we're going to do it. Should we rephrase that section with
> > the
> > >> > > actual
> > >> > > > >>>> change first, and the hows after (ex. Bumping the version is
> > >> not
> > >> > the
> > >> > > > >>>> public-facing change in this case, it's just the mechanism to
> > >> > > properly
> > >> > > > >>>> introduce our change)
> > >> > > > >>>>
> > >> > > > >>>> LM4. Regarding the lifetime of the UUID: the KIP states we
> > will
> > >> > > > "Verify
> > >> > > > >>>> that the UUID remains consistent across all subsequent
> > >> heartbeats
> > >> > > > >> during
> > >> > > > >>>> the session". What is this "session" referring to here? I
> > would
> > >> > > expect
> > >> > > > >>> that
> > >> > > > >>>> the UUID is associated to a consumer instance (generated for
> > >> the
> > >> > > > >> consumer
> > >> > > > >>>> the first time it needs to send a HB if it doesn't have the
> > >> UUID
> > >> > > yet.
> > >> > > > >>> From
> > >> > > > >>>> there on, every time it needs to send a "first HB" again, it
> > >> will
> > >> > > > reuse
> > >> > > > >>> its
> > >> > > > >>>> UUID, is that the intention? Note that we should consider
> > that
> > >> the
> > >> > > > same
> > >> > > > >>>> consumer instance may have many "first heartbeats", meaning
> > >> > > heartbeats
> > >> > > > >> to
> > >> > > > >>>> join the group when it's not part of it (ex. consumer
> > >> unsubscribe
> > >> > +
> > >> > > > >>>> subscribe, fenced, stale). Is this the intention or are you
> > >> > > > considering
> > >> > > > >>> the
> > >> > > > >>>> lifetime differently? We should clarify it in the KIP.
> > >> > > > >>>>
> > >> > > > >>>> Thanks!
> > >> > > > >>>>
> > >> > > > >>>> Lianet
> > >> > > > >>>>
> > >> > > > >>>> On Tue, Aug 27, 2024 at 2:27 AM TengYao Chi <
> > >> kitingiao@gmail.com>
> > >> > > > >> wrote:
> > >> > > > >>>>
> > >> > > > >>>>> Hi everyone,
> > >> > > > >>>>>
> > >> > > > >>>>> I have revised this KIP multiple times based on the feedback
> > >> from
> > >> > > our
> > >> > > > >>>>> discussions.
> > >> > > > >>>>> I would greatly appreciate it if you could review it when
> > you
> > >> > have
> > >> > > > >> the
> > >> > > > >>>>> time.
> > >> > > > >>>>> If there are no further comments or suggestions, I plan to
> > >> > proceed
> > >> > > > >> with
> > >> > > > >>>>> initiating a vote soon.
> > >> > > > >>>>>
> > >> > > > >>>>> Best regards,
> > >> > > > >>>>> TengYao
> > >> > > > >>>>>
> > >> > > > >>>>> TengYao Chi <ki...@gmail.com> 於 2024年8月23日 週五 下午2:43寫道：
> > >> > > > >>>>>
> > >> > > > >>>>>> Hi Andrew,
> > >> > > > >>>>>> Thank you for your previous feedback and insights.
> > >> > > > >>>>>> Your contributions have added valuable perspectives to the
> > >> > > > >>> discussions.
> > >> > > > >>>>>> And we also benefit from the comparison of different
> > >> solutions.
> > >> > > > >>>>>> I’m also looking forward to seeing an initial version in
> > >> > KIP-932,
> > >> > > > >> as
> > >> > > > >>> it
> > >> > > > >>>>>> will provide a good reference for future implementations.
> > >> > > > >>>>>>
> > >> > > > >>>>>> Regarding your comment on AS2, I wanted to clarify that my
> > >> > > > >>>> specification
> > >> > > > >>>>>> references org.apache.kafka.common.Uuid.
> > >> > > > >>>>>> I believe we’re referring to the same class, and it might
> > >> just
> > >> > be
> > >> > > a
> > >> > > > >>>> small
> > >> > > > >>>>>> oversight due to the busy schedule.
> > >> > > > >>>>>>
> > >> > > > >>>>>> I want to express my gratitude once again for your many
> > >> > insightful
> > >> > > > >>>>>> comments, which have helped the discussion progress
> > smoothly.
> > >> > > > >>>>>>
> > >> > > > >>>>>> Best regards,
> > >> > > > >>>>>> TengYao
> > >> > > > >>>>>>
> > >> > > > >>>>>>
> > >> > > > >>>>>> Andrew Schofield <an...@live.com> 於 2024年8月22日
> > 週四
> > >> > > > >>>> 下午11:28寫道：
> > >> > > > >>>>>>
> > >> > > > >>>>>>> Hi TengYao,
> > >> > > > >>>>>>> I’ve been reading through the comments and I’m happy that
> > >> the
> > >> > > > >> lobby
> > >> > > > >>>>>>> approach has not gained support.
> > >> > > > >>>>>>>
> > >> > > > >>>>>>> Assuming that this KIP is voted, I will be happy to change
> > >> > > KIP-932
> > >> > > > >>> so
> > >> > > > >>>>>>> that it only supports client-side member ID generation.
> > >> Because
> > >> > > > >> that
> > >> > > > >>>> KIP
> > >> > > > >>>>>>> is still
> > >> > > > >>>>>>> under development, I can do this in the first version of
> > >> > > > >>>>>>> ShareGroupHeartbeat.
> > >> > > > >>>>>>>
> > >> > > > >>>>>>> AS2: For the encoding section, I suppose the specific
> > >> encoding
> > >> > > > >> which
> > >> > > > >>>>>>> is used is what org.apache.kafka.utils.Uuid uses.
> > >> > > > >>>>>>>
> > >> > > > >>>>>>> Thanks,
> > >> > > > >>>>>>> Andrew
> > >> > > > >>>>>>>
> > >> > > > >>>>>>>> On 14 Aug 2024, at 17:00, TengYao Chi <
> > kitingiao@gmail.com
> > >> >
> > >> > > > >>> wrote:
> > >> > > > >>>>>>>>
> > >> > > > >>>>>>>> Hello Apoorv,
> > >> > > > >>>>>>>> Thank you for your feedback.
> > >> > > > >>>>>>>> Regarding the questions you raised, unfortunately, this
> > KIP
> > >> > > > >> cannot
> > >> > > > >>>>>>>> guarantee the order of heartbeats. As with many classic
> > >> > > > >>> distributed
> > >> > > > >>>>>>> system
> > >> > > > >>>>>>>> challenges, what we can do is make our best effort to
> > >> ensure
> > >> > > > >> that
> > >> > > > >>>>> there
> > >> > > > >>>>>>> are
> > >> > > > >>>>>>>> no idle members or stale assignments under normal
> > >> > circumstances.
> > >> > > > >>>>>>>>
> > >> > > > >>>>>>>> As for the lobby approach, I’m not a fan of it because it
> > >> > > > >> requires
> > >> > > > >>>>>>> adding a
> > >> > > > >>>>>>>> mechanism to maintain client state within the
> > >> ConsumerGroup,
> > >> > > > >>> which,
> > >> > > > >>>> in
> > >> > > > >>>>>>> my
> > >> > > > >>>>>>>> view, resembles something like a two-phase commit. This
> > >> would
> > >> > > > >>>>> introduce
> > >> > > > >>>>>>>> more complexity than the proposal in this KIP, which is
> > >> > > > >> something
> > >> > > > >>> we
> > >> > > > >>>>>>> want
> > >> > > > >>>>>>>> to avoid. KIP-848 aims to simplify the existing protocol,
> > >> and
> > >> > > > >>> while
> > >> > > > >>>>> the
> > >> > > > >>>>>>>> lobby approach is a good one, I believe it is not the
> > right
> > >> > fit
> > >> > > > >>> for
> > >> > > > >>>>> this
> > >> > > > >>>>>>>> particular situation.
> > >> > > > >>>>>>>>
> > >> > > > >>>>>>>> Best regards,
> > >> > > > >>>>>>>> TengYao
> > >> > > > >>>>>>>>
> > >> > > > >>>>>>>> TengYao Chi <ki...@gmail.com> 於 2024年8月14日 週三
> > >> 下午11:45寫道：
> > >> > > > >>>>>>>>
> > >> > > > >>>>>>>>> Hi David,
> > >> > > > >>>>>>>>>
> > >> > > > >>>>>>>>> I really appreciate your review and suggestions. As I am
> > >> > still
> > >> > > > >>>>> gaining
> > >> > > > >>>>>>>>> experience in writing KIPs, your input has been
> > incredibly
> > >> > > > >>>> helpful. I
> > >> > > > >>>>>>> am
> > >> > > > >>>>>>>>> currently applying your suggestions to the KIP and will
> > >> > > > >> complete
> > >> > > > >>> it
> > >> > > > >>>>> as
> > >> > > > >>>>>>> soon
> > >> > > > >>>>>>>>> as possible.
> > >> > > > >>>>>>>>> Regarding the UUID part, I think we haven’t reached a
> > >> > > > >> conclusion
> > >> > > > >>>>>>> yet.(So
> > >> > > > >>>>>>>>> far according to this thread)
> > >> > > > >>>>>>>>> However, I will review the current implementation in the
> > >> > Kafka
> > >> > > > >>>> `Uuid`
> > >> > > > >>>>>>>>> class and include a brief specification in the KIP.
> > >> > > > >>>>>>>>>
> > >> > > > >>>>>>>>> Once again, thank you so much for your help.
> > >> > > > >>>>>>>>>
> > >> > > > >>>>>>>>> Best regards,
> > >> > > > >>>>>>>>> TengYao
> > >> > > > >>>>>>>>>
> > >> > > > >>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月14日 週三
> > >> > 下午11:14寫道：
> > >> > > > >>>>>>>>>
> > >> > > > >>>>>>>>>> hi Apoorv
> > >> > > > >>>>>>>>>>
> > >> > > > >>>>>>>>>>> As the memberId is now known to the client, and client
> > >> > might
> > >> > > > >>> send
> > >> > > > >>>>> the
> > >> > > > >>>>>>>>>> leave
> > >> > > > >>>>>>>>>> group heartbeat on shutdown prior to receiving the
> > >> initial
> > >> > > > >>>> heartbeat
> > >> > > > >>>>>>>>>> response. If that's true then how do we guarantee that
> > >> the 2
> > >> > > > >>>>> requests
> > >> > > > >>>>>>> to
> > >> > > > >>>>>>>>>> join and leave will be processed in order, which could
> > >> still
> > >> > > > >>> leave
> > >> > > > >>>>>>> stale
> > >> > > > >>>>>>>>>> members or throw unknown member id exceptions?
> > >> > > > >>>>>>>>>>
> > >> > > > >>>>>>>>>> This is definitely a good question. the short answer:
> > no
> > >> > > > >>> guarantee
> > >> > > > >>>>> but
> > >> > > > >>>>>>>>>> best
> > >> > > > >>>>>>>>>> efforts
> > >> > > > >>>>>>>>>>
> > >> > > > >>>>>>>>>> Please notice the root cause is "we have no enough time
> > >> to
> > >> > > > >> wait
> > >> > > > >>>>>>> member id
> > >> > > > >>>>>>>>>> (response) when closing consumer". Sadly, we can'
> > >> guarantee
> > >> > > > >> the
> > >> > > > >>>>>>> request
> > >> > > > >>>>>>>>>> order due to the same reason.
> > >> > > > >>>>>>>>>>
> > >> > > > >>>>>>>>>> However, in contrast to previous behavior, there is one
> > >> big
> > >> > > > >>>> benefit
> > >> > > > >>>>>>> of new
> > >> > > > >>>>>>>>>> approach - we can try STONITH because we know the
> > member
> > >> id
> > >> > > > >>>>>>>>>>
> > >> > > > >>>>>>>>>> Best,
> > >> > > > >>>>>>>>>> Chia-Ping
> > >> > > > >>>>>>>>>>
> > >> > > > >>>>>>>>>>
> > >> > > > >>>>>>>>>> Apoorv Mittal <ap...@gmail.com> 於 2024年8月14日
> > 週三
> > >> > > > >>>> 下午8:55寫道：
> > >> > > > >>>>>>>>>>
> > >> > > > >>>>>>>>>>> Hi TengYao,
> > >> > > > >>>>>>>>>>> Thanks for the KIP. Continuing on the point which
> > Andrew
> > >> > > > >>>> mentioned
> > >> > > > >>>>> as
> > >> > > > >>>>>>>>>> AS1.
> > >> > > > >>>>>>>>>>>
> > >> > > > >>>>>>>>>>> As the memberId is now known to the client, and client
> > >> > might
> > >> > > > >>> send
> > >> > > > >>>>> the
> > >> > > > >>>>>>>>>> leave
> > >> > > > >>>>>>>>>>> group heartbeat on shutdown prior to receiving the
> > >> initial
> > >> > > > >>>>> heartbeat
> > >> > > > >>>>>>>>>>> response. If that's true then how do we guarantee that
> > >> the
> > >> > 2
> > >> > > > >>>>>>> requests to
> > >> > > > >>>>>>>>>>> join and leave will be processed in order, which could
> > >> > still
> > >> > > > >>>> leave
> > >> > > > >>>>>>> stale
> > >> > > > >>>>>>>>>>> members or throw unknown member id exceptions?
> > >> > > > >>>>>>>>>>>
> > >> > > > >>>>>>>>>>> Though the client side member id generation is helpful
> > >> > which
> > >> > > > >>> will
> > >> > > > >>>>>>>>>> represent
> > >> > > > >>>>>>>>>>> the same group perspective as from client and broker's
> > >> end.
> > >> > > > >>> But I
> > >> > > > >>>>>>> think
> > >> > > > >>>>>>>>>> the
> > >> > > > >>>>>>>>>>> major concern we want to solve here is Stale Partition
> > >> > > > >>>> Assignments
> > >> > > > >>>>>>> which
> > >> > > > >>>>>>>>>>> might still exist with the new approach. I am leaning
> > >> > towards
> > >> > > > >>> the
> > >> > > > >>>>>>>>>>> suggestion mentioned by Andrew where partition
> > >> assignment
> > >> > > > >>>> triggers
> > >> > > > >>>>> on
> > >> > > > >>>>>>>>>>> subsequent heartbeat when client acknowledges the
> > >> initial
> > >> > > > >>>>> heartbeat,
> > >> > > > >>>>>>>>>>> delayed partition assignment.
> > >> > > > >>>>>>>>>>>
> > >> > > > >>>>>>>>>>> Though on a separate note, I have a different
> > question.
> > >> > What
> > >> > > > >>>>> happens
> > >> > > > >>>>>>>>>> when
> > >> > > > >>>>>>>>>>> there is an issue with the client which sends the
> > >> initial
> > >> > > > >>>> heartbeat
> > >> > > > >>>>>>>>>> without
> > >> > > > >>>>>>>>>>> memberId, then crashes and restarts? I think we must
> > be
> > >> > > > >>>>> re-triggering
> > >> > > > >>>>>>>>>>> assignments and expiring members only after the
> > >> heartbeat
> > >> > > > >>> session
> > >> > > > >>>>>>>>>> timeout?
> > >> > > > >>>>>>>>>>> If that's true then shall delayed partition assignment
> > >> can
> > >> > > > >> help
> > >> > > > >>>>>>> benefit
> > >> > > > >>>>>>>>>> us
> > >> > > > >>>>>>>>>>> from this situation as well?
> > >> > > > >>>>>>>>>>>
> > >> > > > >>>>>>>>>>> Regards,
> > >> > > > >>>>>>>>>>> Apoorv Mittal
> > >> > > > >>>>>>>>>>>
> > >> > > > >>>>>>>>>>>
> > >> > > > >>>>>>>>>>> On Wed, Aug 14, 2024 at 12:51 PM David Jacot
> > >> > > > >>>>>>>>>> <dj...@confluent.io.invalid>
> > >> > > > >>>>>>>>>>> wrote:
> > >> > > > >>>>>>>>>>>
> > >> > > > >>>>>>>>>>>> Hi Andrew,
> > >> > > > >>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>> Personally, I don't like the lobby approach. It makes
> > >> > things
> > >> > > > >>>> more
> > >> > > > >>>>>>>>>>>> complicated and it would require changing the records
> > >> on
> > >> > the
> > >> > > > >>>>> server
> > >> > > > >>>>>>>>>> too.
> > >> > > > >>>>>>>>>>>> This is why I initially suggested the rejected
> > >> alternative
> > >> > > > >> #2
> > >> > > > >>>>> which
> > >> > > > >>>>>>> is
> > >> > > > >>>>>>>>>>>> pretty close but also not perfect.
> > >> > > > >>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>> I'd like to clarify one thing. The
> > >> ConsumerGroupHeartbeat
> > >> > > > >> API
> > >> > > > >>>>>>> already
> > >> > > > >>>>>>>>>>>> supports generating the member id on the client so we
> > >> > don't
> > >> > > > >>> need
> > >> > > > >>>>> any
> > >> > > > >>>>>>>>>>>> conditional logic on the client side. This is
> > actually
> > >> > what
> > >> > > > >> we
> > >> > > > >>>>>>> wanted
> > >> > > > >>>>>>>>>> to
> > >> > > > >>>>>>>>>>> do
> > >> > > > >>>>>>>>>>>> in the first place but the idea got pushed back by
> > >> Magnus
> > >> > > > >> back
> > >> > > > >>>>> then
> > >> > > > >>>>>>>>>>> because
> > >> > > > >>>>>>>>>>>> generating uuid from librdkafka required a new
> > >> dependency.
> > >> > > > >> It
> > >> > > > >>>>> turns
> > >> > > > >>>>>>>>>> out
> > >> > > > >>>>>>>>>>>> that librdkafka has that dependency today. In
> > >> retrospect,
> > >> > we
> > >> > > > >>>>> should
> > >> > > > >>>>>>>>>> have
> > >> > > > >>>>>>>>>>>> pushed back on this. Long story short, we can just do
> > >> it.
> > >> > > > >> The
> > >> > > > >>>>>>>>>> proposal in
> > >> > > > >>>>>>>>>>>> this KIP is to make the member id required in future
> > >> > > > >> versions.
> > >> > > > >>>> We
> > >> > > > >>>>>>>>>> could
> > >> > > > >>>>>>>>>>>> also decide not to do it and to keep supporting both
> > >> > > > >>>> approaches. I
> > >> > > > >>>>>>>>>> would
> > >> > > > >>>>>>>>>>>> also be fine with this.
> > >> > > > >>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>> Best,
> > >> > > > >>>>>>>>>>>> David
> > >> > > > >>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>> On Wed, Aug 14, 2024 at 12:30 PM Andrew Schofield <
> > >> > > > >>>>>>>>>>>> andrew_schofield@live.com>
> > >> > > > >>>>>>>>>>>> wrote:
> > >> > > > >>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>> Hi TengYao,
> > >> > > > >>>>>>>>>>>>> Thanks for your response. I’ll have just one more
> > try
> > >> to
> > >> > > > >>>>> persuade.
> > >> > > > >>>>>>>>>>>>> I feel that I will need to follow the approach with
> > >> > KIP-932
> > >> > > > >>>> when
> > >> > > > >>>>>>>>>> we’ve
> > >> > > > >>>>>>>>>>>>> made a decision, so I do have more than a passing
> > >> > interest
> > >> > > > >> in
> > >> > > > >>>>> this.
> > >> > > > >>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>> A group member in the lobby is in the group, but it
> > >> does
> > >> > > > >> not
> > >> > > > >>>> have
> > >> > > > >>>>>>>>>> any
> > >> > > > >>>>>>>>>>>>> assignments. A member of a consumer group can have
> > no
> > >> > > > >>> assigned
> > >> > > > >>>>>>>>>>>>> partitions (such as 5 CG members subscribed to a
> > topic
> > >> > > > >> with 4
> > >> > > > >>>>>>>>>>>> partitions),
> > >> > > > >>>>>>>>>>>>> so it’s a situation that consumer group members
> > >> already
> > >> > > > >>> expect.
> > >> > > > >>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>> One of Kafka’s strengths is the way that we handle
> > API
> > >> > > > >>>>> versioning.
> > >> > > > >>>>>>>>>>>>> But, there is a cost - the behaviour is different
> > >> > depending
> > >> > > > >>> on
> > >> > > > >>>>> the
> > >> > > > >>>>>>>>>> RPC
> > >> > > > >>>>>>>>>>>>> version. KIP-848 is on the cusp of completion, but
> > >> we’re
> > >> > > > >>>> already
> > >> > > > >>>>>>>>>> adding
> > >> > > > >>>>>>>>>>>>> conditional logic for v0/v1 for
> > >> ConsumerGroupHeartbeat.
> > >> > > > >>> That’s
> > >> > > > >>>> a
> > >> > > > >>>>>>>>>> pity.
> > >> > > > >>>>>>>>>>>>> Only a minor issue, but it’s unfortunate.
> > >> > > > >>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>> Thanks,
> > >> > > > >>>>>>>>>>>>> Andrew
> > >> > > > >>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>> On 14 Aug 2024, at 08:47, TengYao Chi <
> > >> > > > >> kitingiao@gmail.com>
> > >> > > > >>>>>>>>>> wrote:
> > >> > > > >>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>> Hello Andrew
> > >> > > > >>>>>>>>>>>>>> Thank you for your thoughtful suggestions and
> > getting
> > >> > the
> > >> > > > >>>>>>>>>> discussion
> > >> > > > >>>>>>>>>>>>> going.
> > >> > > > >>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>> To AS1:
> > >> > > > >>>>>>>>>>>>>> In the current scenario where the server generates
> > >> the
> > >> > > > >> UUID,
> > >> > > > >>>> if
> > >> > > > >>>>>>>>>> the
> > >> > > > >>>>>>>>>>>>> client
> > >> > > > >>>>>>>>>>>>>> shuts down before receiving the memberId generated
> > by
> > >> > the
> > >> > > > >> GC
> > >> > > > >>>>>>>>>>>> (regardless
> > >> > > > >>>>>>>>>>>>> of
> > >> > > > >>>>>>>>>>>>>> whether it’s a graceful shutdown or not), the GC
> > will
> > >> > > > >> still
> > >> > > > >>>> have
> > >> > > > >>>>>>>>>> to
> > >> > > > >>>>>>>>>>>> wait
> > >> > > > >>>>>>>>>>>>>> for the heartbeat timeout because the client
> > doesn’t
> > >> > know
> > >> > > > >>> its
> > >> > > > >>>>>>>>>>> memberId.
> > >> > > > >>>>>>>>>>>>>> This KIP indeed cannot completely resolve the
> > >> > idempotency
> > >> > > > >>>> issue,
> > >> > > > >>>>>>>>>> but
> > >> > > > >>>>>>>>>>> it
> > >> > > > >>>>>>>>>>>>> can
> > >> > > > >>>>>>>>>>>>>> better handle shutdown scenarios under normal
> > >> > > > >> circumstances
> > >> > > > >>>>>>>>>> because
> > >> > > > >>>>>>>>>>> the
> > >> > > > >>>>>>>>>>>>>> client always knows its memberId. Even if the
> > client
> > >> > shuts
> > >> > > > >>>> down
> > >> > > > >>>>>>>>>>>>> immediately
> > >> > > > >>>>>>>>>>>>>> after the initial heartbeat, as long as it
> > performs a
> > >> > > > >>> graceful
> > >> > > > >>>>>>>>>>> shutdown
> > >> > > > >>>>>>>>>>>>> and
> > >> > > > >>>>>>>>>>>>>> sends a leave heartbeat, the GC can manage the
> > >> situation
> > >> > > > >> and
> > >> > > > >>>>>>>>>> remove
> > >> > > > >>>>>>>>>>> the
> > >> > > > >>>>>>>>>>>>>> member. Therefore, the goal of this KIP is to
> > address
> > >> > the
> > >> > > > >>>> issue
> > >> > > > >>>>>>>>>> where
> > >> > > > >>>>>>>>>>>> the
> > >> > > > >>>>>>>>>>>>>> GC has to wait for the heartbeat timeout due to the
> > >> > client
> > >> > > > >>>>> leaving
> > >> > > > >>>>>>>>>>>>> without
> > >> > > > >>>>>>>>>>>>>> knowing its memberId, which leads to reduced
> > >> throughput
> > >> > > > >> and
> > >> > > > >>>>>>>>>> limited
> > >> > > > >>>>>>>>>>>>>> scalability.
> > >> > > > >>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>> The solution you suggest has also been proposed by
> > >> > David.
> > >> > > > >>> The
> > >> > > > >>>>>>>>>> concern
> > >> > > > >>>>>>>>>>>>> with
> > >> > > > >>>>>>>>>>>>>> this approach is that it introduces additional
> > >> > complexity
> > >> > > > >>> for
> > >> > > > >>>>>>>>>>>>>> compatibility, as the new server would not
> > >> immediately
> > >> > add
> > >> > > > >>> the
> > >> > > > >>>>>>>>>> member
> > >> > > > >>>>>>>>>>>> to
> > >> > > > >>>>>>>>>>>>>> the group, while the old server would. This
> > requires
> > >> > > > >> clients
> > >> > > > >>>> to
> > >> > > > >>>>>>>>>>>>>> differentiate whether their memberId has been added
> > >> to
> > >> > the
> > >> > > > >>>> group
> > >> > > > >>>>>>>>>> or
> > >> > > > >>>>>>>>>>>> not,
> > >> > > > >>>>>>>>>>>>>> which could result in unexpected logs.
> > >> > > > >>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>> Best Regards,
> > >> > > > >>>>>>>>>>>>>> TengYao
> > >> > > > >>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>> Andrew Schofield <an...@live.com> 於
> > >> > 2024年8月14日
> > >> > > > >>> 週三
> > >> > > > >>>>>>>>>>>> 上午12:29寫道：
> > >> > > > >>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>> Hi TengYao,
> > >> > > > >>>>>>>>>>>>>>> Thanks for the KIP. I wonder if there’s a
> > different
> > >> way
> > >> > > > >> to
> > >> > > > >>>>> close
> > >> > > > >>>>>>>>>>> what
> > >> > > > >>>>>>>>>>>>>>> is quite a small window.
> > >> > > > >>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>> AS1: It is true that the initial heartbeat is not
> > >> > > > >>> idempotent,
> > >> > > > >>>>> but
> > >> > > > >>>>>>>>>>> this
> > >> > > > >>>>>>>>>>>>>>> remains
> > >> > > > >>>>>>>>>>>>>>> true with this KIP. It’s just differently not
> > >> > idempotent.
> > >> > > > >>> If
> > >> > > > >>>>> the
> > >> > > > >>>>>>>>>>>> client
> > >> > > > >>>>>>>>>>>>>>> makes its
> > >> > > > >>>>>>>>>>>>>>> own member ID, sends a request and dies, the GC
> > will
> > >> > > > >> still
> > >> > > > >>>> have
> > >> > > > >>>>>>>>>>> added
> > >> > > > >>>>>>>>>>>>>>> the member to the group and it will hang around
> > >> until
> > >> > the
> > >> > > > >>>>> session
> > >> > > > >>>>>>>>>>>>> expires.
> > >> > > > >>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>> I wonder if the GC could still generate the member
> > >> ID
> > >> > in
> > >> > > > >>>>>>>>>> response to
> > >> > > > >>>>>>>>>>>> the
> > >> > > > >>>>>>>>>>>>>>> first
> > >> > > > >>>>>>>>>>>>>>> heartbeat, and put the member in a special PENDING
> > >> > state
> > >> > > > >>> with
> > >> > > > >>>>> no
> > >> > > > >>>>>>>>>>>>>>> assignments until the client sends the next
> > >> heartbeat,
> > >> > > > >> thus
> > >> > > > >>>>>>>>>>> confirming
> > >> > > > >>>>>>>>>>>>> it
> > >> > > > >>>>>>>>>>>>>>> has received the member ID. This would not be a
> > >> > protocol
> > >> > > > >>>> change
> > >> > > > >>>>>>>>>> at
> > >> > > > >>>>>>>>>>>> all,
> > >> > > > >>>>>>>>>>>>>>> just
> > >> > > > >>>>>>>>>>>>>>> a change to the GC to keep a new member in the
> > lobby
> > >> > > > >> until
> > >> > > > >>>> it’s
> > >> > > > >>>>>>>>>>>>> comfirmed
> > >> > > > >>>>>>>>>>>>>>> it knows its member ID.
> > >> > > > >>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>> Thanks,
> > >> > > > >>>>>>>>>>>>>>> Andrew
> > >> > > > >>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>> On 13 Aug 2024, at 15:59, TengYao Chi <
> > >> > > > >>> kitingiao@gmail.com>
> > >> > > > >>>>>>>>>> wrote:
> > >> > > > >>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>> Hi Chia-Ping,
> > >> > > > >>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>> Thanks for review and suggestions.
> > >> > > > >>>>>>>>>>>>>>>> I have updated the content of KIP accordingly.
> > >> > > > >>>>>>>>>>>>>>>> Please take a look.
> > >> > > > >>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>> Best regards,
> > >> > > > >>>>>>>>>>>>>>>> TengYao
> > >> > > > >>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > 2024年8月13日
> > >> 週二
> > >> > > > >>>>> 下午9:45寫道：
> > >> > > > >>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>>> hi TengYao
> > >> > > > >>>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>>> thanks for this KIP.
> > >> > > > >>>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>>> 1) could you please describe the before/after
> > >> > behavior
> > >> > > > >> in
> > >> > > > >>>> the
> > >> > > > >>>>>>>>>>>>> "Proposed
> > >> > > > >>>>>>>>>>>>>>>>> Changes" section? IIRC, current RPC allows HB
> > >> having
> > >> > > > >>> member
> > >> > > > >>>>> id
> > >> > > > >>>>>>>>>>>>>>> generated by
> > >> > > > >>>>>>>>>>>>>>>>> client, right? If HB has no member ID, server
> > will
> > >> > > > >>> generate
> > >> > > > >>>>> one
> > >> > > > >>>>>>>>>>> and
> > >> > > > >>>>>>>>>>>>> then
> > >> > > > >>>>>>>>>>>>>>>>> return. The new behavior will enforce HB "must"
> > >> have
> > >> > > > >>> member
> > >> > > > >>>>> ID.
> > >> > > > >>>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>>> 2) could you please write the version number
> > >> > explicitly
> > >> > > > >>> in
> > >> > > > >>>>> the
> > >> > > > >>>>>>>>>> KIP
> > >> > > > >>>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>>> 3) how new client code handle the old HB? Does
> > it
> > >> > > > >> always
> > >> > > > >>>>>>>>>> generate
> > >> > > > >>>>>>>>>>>>> member
> > >> > > > >>>>>>>>>>>>>>>>> ID on client-side even though that is not
> > >> restricted?
> > >> > > > >>>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>>> Best,
> > >> > > > >>>>>>>>>>>>>>>>> Chia-Ping
> > >> > > > >>>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>>> On 2024/08/13 06:20:42 TengYao Chi wrote:
> > >> > > > >>>>>>>>>>>>>>>>>> Hello everyone,
> > >> > > > >>>>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>>>> I would like to start a discussion thread on
> > >> > KIP-1082,
> > >> > > > >>>> which
> > >> > > > >>>>>>>>>>>> proposes
> > >> > > > >>>>>>>>>>>>>>>>>> enabling id generation for clients over the
> > >> > > > >>>>>>>>>>> ConsumerGroupHeartbeat
> > >> > > > >>>>>>>>>>>>> RPC.
> > >> > > > >>>>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>>>> Here is the KIP Link: KIP-1082
> > >> > > > >>>>>>>>>>>>>>>>>> <
> > >> > > > >>>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>
> > >> > > > >>>>>>>>>>
> > >> > > > >>>>>>>
> > >> > > > >>>>>
> > >> > > > >>>>
> > >> > > > >>>
> > >> > > > >>
> > >> > > >
> > >> > >
> > >> >
> > >>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
> > >> > > > >>>>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>>>> Please take a look and let me know what you
> > >> think,
> > >> > > > >> and I
> > >> > > > >>>>> would
> > >> > > > >>>>>>>>>>>>>>> appreciate
> > >> > > > >>>>>>>>>>>>>>>>>> any suggestions and feedback.
> > >> > > > >>>>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>>>> Best regards,
> > >> > > > >>>>>>>>>>>>>>>>>> TengYao
> > >> > > > >>>>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>
> > >> > > > >>>>>>>>>>
> > >> > > > >>>>>>>>>
> > >> > > > >>>>>>>
> > >> > > > >>>>>>>
> > >> > > > >>>>>
> > >> > > > >>>>
> > >> > > > >>>
> > >> > > > >>
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
> 

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[TengYao Chi <ki...@gmail.com>]

Hi David,

Thanks for your helpful suggestions, which make this KIP clearer.

I have revised the content according to your feedback.
Regarding the fourth point, I agree that the original description was
imprecise and could lead to misunderstandings.
Since this is not new behavior, I have removed it.

Please take a look and let me know what you think.

Best regards,
TengYao

David Jacot <dj...@confluent.io.invalid> 於 2024年9月20日 週五 上午2:37寫道：

> Hi,
>
> Thanks for the update. I have a few nits:
>
> > If the member ID is null or empty, the server will reject the request
> with an InvalidRequestException.
> We should clarify that this should only apply to version >= 1.
>
> > The consumer instance must generate a member ID, and this ID should
> remain consistent for the duration of the consumer's session. Here, a
> "session" is defined as the period from the consumer's first heartbeat
> until it leaves the group, either through a graceful shutdown, a heartbeat
> timeout, or the process stopping or dying. The consumer instance should
> reuse the same member ID for all heartbeats and rejoin attempts to maintain
> continuity within the group.
>
> This part is not clear to me. When the member leaves the group, it should
> not reset the member id. I would rather say that the member must generate
> its member id when it starts and it must keep it until the process stops.
> It is basically an incarnation of the process.
>
> > If a conflict arises where the member ID generated by the client is
> detected to be a duplicate within the same group (for example, the same
> member ID is associated with another active member in the group), the
> server will handle this by comparing the memberEpoch values of the
> conflicting members. The member with the lower memberEpoch is considered
> outdated and will be fenced off by the server. When this occurs, the server
> responds with a FENCED_MEMBER_EPOCH error to the client, signaling it to
> rejoin the group with the same member ID while resetting the memberEpoch to
> zero. This ensures that the client properly resynchronizes and maintains
> the continuity and consistency of the group membership.
>
> This part is not clear either. It basically says that if a member joins
> with an existing member id but a different epoch, it will be fenced. Then
> it must rejoin with the same member id and epoch zero. This is already the
> current behavior and it does not help with detecting duplicates, right?
> Should we just remove the paragraph?
>
> > A member ID mismatch occurs within a session: If the server detects a
> mismatch between the provided member ID and the expected member ID for an
> ongoing session, it should return a UNKNOWN_MEMBER_ID  error.
>
> How could we detect a mismatch between the provided and the expected member
> id? My understanding is that we can only know whether the provided member
> id exists or not. This is already implemented.
>
> Thanks,
> David
>
> On Sat, Sep 14, 2024 at 9:31 AM TengYao Chi <ki...@gmail.com> wrote:
>
> > Hello everyone,
> >
> > Since this KIP has been fully discussed, I will initiate a vote for it
> next
> > Monday.
> > Thank you and have a nice weekend.
> >
> > Best regards,
> > TengYao
> >
> > TengYao Chi <ki...@gmail.com> 於 2024年9月5日 週四 下午2:19寫道：
> >
> > > Hello everyone,
> > >
> > > KT2: It looks like everyone who has expressed an opinion supports the
> > > second option: “Document a recommendation for clients to use UUIDs as
> > > member IDs, without strictly enforcing it.”
> > > I have updated the KIP accordingly.
> > > Please take a look, and let me know if you have any thoughts or
> feedback.
> > >
> > > Thank you!
> > >
> > > Best regards,
> > > TengYao
> > >
> > > Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月30日 週五 下午9:56寫道：
> > >
> > >> hi TengYao
> > >>
> > >> KT2: +1 to second approach
> > >>
> > >> Best,
> > >> Chia-Ping
> > >>
> > >>
> > >> David Jacot <dj...@confluent.io.invalid> 於 2024年8月30日 週五 下午9:15寫道：
> > >>
> > >> > Hi TengYao,
> > >> >
> > >> > KT2: I don't think that we can realistically validate the uuid on
> the
> > >> > server. It is basically a string of chars. So I lean towards having
> a
> > >> good
> > >> > recommendation in the KIP and in the document of the field in the
> > RPC's
> > >> > definition.
> > >> >
> > >> > Best,
> > >> > David
> > >> >
> > >> > On Fri, Aug 30, 2024 at 3:02 PM TengYao Chi <ki...@gmail.com>
> > >> wrote:
> > >> >
> > >> > > Hello Kirk !
> > >> > >
> > >> > > Thank you for your comments !
> > >> > >
> > >> > > KT1: Yes, you are correct. The issue is not unique to the initial
> > >> > > heartbeat; there can always be cases where the broker might lose
> > >> > connection
> > >> > > with a member.
> > >> > >
> > >> > > KT2: Currently, if the client doesn't have a member ID and the
> > >> > memberEpoch
> > >> > > equals 0, the coordinator will generate a UUID as the member ID
> for
> > >> the
> > >> > > client. However, at the RPC level, the member ID is sent as a
> > literal
> > >> > > string, meaning there are no restrictions on the format at this
> > level.
> > >> > > This also reminds me that we haven't reached a final conclusion on
> > >> how to
> > >> > > enforce the use of UUIDs.
> > >> > > From our previous discussions, I recall two possible approaches:
> > >> > > The first is to validate the UUID on the server side, and if it's
> > not
> > >> > > valid, throw an exception to the client.
> > >> > > The second is to document a recommendation for clients to use
> UUIDs
> > as
> > >> > > member IDs, without strictly enforcing it.
> > >> > > I think it's time to decide on the approach we want to take.
> > >> > >
> > >> > > KT3: Yes, "session" can be considered synonymous with "membership"
> > in
> > >> > this
> > >> > > context.
> > >> > >
> > >> > > KT4: Thank you for pointing that out. I will update the wording to
> > >> > > specifically say this behavior is for consumers.
> > >> > >
> > >> > > Thanks again for your comments.
> > >> > >
> > >> > > Best regards,
> > >> > > TengYao
> > >> > >
> > >> > > Kirk True <ki...@kirktrue.pro> 於 2024年8月30日 週五 上午12:39寫道：
> > >> > >
> > >> > > > Hi TengYao!
> > >> > > >
> > >> > > > Sorry for being late to the discussion...
> > >> > > >
> > >> > > > After reading the thread and then the KIP, I had a few
> > >> > > questions/comments:
> > >> > > >
> > >> > > > KT1: In Motivation, it states: "This scenario can result in the
> > >> broker
> > >> > > > registering a new member for which it will never receive a
> proper
> > >> leave
> > >> > > > request.” Just to be clear, the broker will always have cases
> > where
> > >> it
> > >> > > > might lose connection with a member. That’s not unique to the
> > >> initial
> > >> > > > heartbeat, right?
> > >> > > >
> > >> > > > KT2: There was a bit of back and forth about format of the
> member
> > >> ID.
> > >> > > From
> > >> > > > what I gathered in the thread, the member ID is still defined in
> > the
> > >> > RPC
> > >> > > as
> > >> > > > a string and not a UUID, right? The KIP states that the “client
> > must
> > >> > > > generate a UUID as the member ID” and that the “server will
> > validate
> > >> > > that a
> > >> > > > valid UUID is provided.” Is that a change for the server, or is
> it
> > >> > > already
> > >> > > > enforced as a UUID?
> > >> > > >
> > >> > > > KT3: Lianet mentioned some confusion over the use of the word
> > >> > “session.”
> > >> > > > Isn’t “session” synonymous with “membership?”
> > >> > > >
> > >> > > > KT4: Under “Member ID Lifecycle,” it states: "The client should
> > >> reuse
> > >> > the
> > >> > > > same UUID as the member ID for all heartbeats and rejoin
> attempts
> > to
> > >> > > > maintain continuity within the group.” Could we change the first
> > >> part
> > >> > of
> > >> > > > that to “The Consumer instance should…” We do have lifetimes
> that
> > >> > extend
> > >> > > > past the lifetime of a client instance (such as the transaction
> > ID).
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Kirk
> > >> > > >
> > >> > > > > On Aug 29, 2024, at 1:28 AM, TengYao Chi <kitingiao@gmail.com
> >
> > >> > wrote:
> > >> > > > >
> > >> > > > > Hi David,
> > >> > > > >
> > >> > > > > Thank you for pointing that out.
> > >> > > > > I have updated the content of the KIP based on Lianet's and
> your
> > >> > > > feedback.
> > >> > > > > Please take a look and let me know your thoughts.
> > >> > > > >
> > >> > > > > Best regards,
> > >> > > > > TengYao
> > >> > > > >
> > >> > > > > David Jacot <dj...@confluent.io.invalid> 於 2024年8月29日 週四
> > >> 下午3:20寫道：
> > >> > > > >
> > >> > > > >> Hi TengYao,
> > >> > > > >>
> > >> > > > >> Thanks for the update. I haven't fully read it yet but I will
> > >> soon.
> > >> > > > >>
> > >> > > > >> LM4: This is incorrect. The consumer must keep its member id
> > >> during
> > >> > > its
> > >> > > > >> entire lifetime (until the process stops or dies). The
> protocol
> > >> > > > stipulates
> > >> > > > >> that a member must rejoin with the same member id and the
> > member
> > >> > epoch
> > >> > > > set
> > >> > > > >> to zero when an FENCED_MEMBER_EPOCH occurs. This allows the
> > >> member
> > >> > to
> > >> > > > >> resynchronize itself. We should not change this behavior. I
> > think
> > >> > that
> > >> > > > we
> > >> > > > >> should see the client side generation id as an incarnation id
> > of
> > >> the
> > >> > > > >> application. It is generated once and kept until it stops or
> > >> dies.
> > >> > > > >>
> > >> > > > >> Best,
> > >> > > > >> David
> > >> > > > >>
> > >> > > > >> On Thu, Aug 29, 2024 at 6:21 AM TengYao Chi <
> > kitingiao@gmail.com
> > >> >
> > >> > > > wrote:
> > >> > > > >>
> > >> > > > >>> Hello Lianet !
> > >> > > > >>>
> > >> > > > >>> Thanks for the reviews and suggestions!
> > >> > > > >>>
> > >> > > > >>> LM1: Indeed, we plan to enforce client-side ID generation in
> > the
> > >> > > > future,
> > >> > > > >>> and it is not an alternative. I will change the title
> > >> accordingly.
> > >> > > > >>>
> > >> > > > >>> LM2: Yes, that's the expectation. I will add that statement
> to
> > >> the
> > >> > > > public
> > >> > > > >>> interface section.
> > >> > > > >>>
> > >> > > > >>> LM3: Thank you for the high-level perspective review. I
> think
> > >> > you're
> > >> > > > >> right;
> > >> > > > >>> our intention isn't very clear since it was placed at the
> end
> > of
> > >> > the
> > >> > > > >>> section. I will try to rephrase that section to make it more
> > >> > obvious.
> > >> > > > >>>
> > >> > > > >>> LM4: Regarding the definition of "session" in this KIP, I
> > >> believe
> > >> > it
> > >> > > > >> refers
> > >> > > > >>> to the period between the *first-time heartbeat* and when
> the
> > >> > > *consumer
> > >> > > > >>> leaves the group* (whether through a graceful shutdown or a
> > >> > heartbeat
> > >> > > > >>> timeout). The consumer should reuse its UUID if it has been
> > >> > generated
> > >> > > > >>> before. The only situation in which it will regenerate the
> > UUID
> > >> is
> > >> > if
> > >> > > > the
> > >> > > > >>> coordinator finds that there is already a consumer with the
> > same
> > >> > > UUID.
> > >> > > > >>> IIRC, the coordinator should compare the member epochs, and
> > the
> > >> > > > >>> later-joined consumer should be fenced off by the
> coordinator
> > >> due
> > >> > to
> > >> > > > >> having
> > >> > > > >>> a lower member epoch. Once the consumer receives a
> > >> > > > `FENCED_MEMBER_EPOCH`
> > >> > > > >>> error, it will generate a new UUID and attempt to rejoin. I
> > will
> > >> > > > clarify
> > >> > > > >>> this in the KIP.
> > >> > > > >>>
> > >> > > > >>> Thanks again for your reviews, I really appreciate it.
> > >> > > > >>>
> > >> > > > >>> Best regards,
> > >> > > > >>> TengYao
> > >> > > > >>>
> > >> > > > >>> Lianet M. <li...@gmail.com> 於 2024年8月28日 週三 下午7:12寫道：
> > >> > > > >>>
> > >> > > > >>>> Hello TengYao! Thanks for taking on this issue, we've been
> > >> going
> > >> > > > around
> > >> > > > >>> it
> > >> > > > >>>> for a while.
> > >> > > > >>>>
> > >> > > > >>>> LM1: About the title of the KIP: "Enable ID Generation for
> > >> Clients
> > >> > > > over
> > >> > > > >>> the
> > >> > > > >>>> ConsumerGroupHeartbeat RPC". I find it confusing because it
> > >> hints
> > >> > > that
> > >> > > > >>>> we're adding it as an alternative (which was discussed and
> > >> > > discarded,
> > >> > > > >> in
> > >> > > > >>>> favour of really enforcing it). It's also missing the core
> > >> change
> > >> > > imo,
> > >> > > > >>>> which is "where" the generation happens. So, maybe more to
> > the
> > >> > point
> > >> > > > >> with
> > >> > > > >>>> something along the lines of "Client-side generated ID for
> > >> clients
> > >> > > > over
> > >> > > > >>>> ConsumerGroupHeartbeat RPC"?
> > >> > > > >>>>
> > >> > > > >>>> LM2: On the public interfaces section, the KIP states that
> > "the
> > >> > > server
> > >> > > > >>> will
> > >> > > > >>>> reject the request", but we should agree on the specific
> > error
> > >> > > type. I
> > >> > > > >>>> expect it should fail with an InvalidRequestException, is
> > that
> > >> the
> > >> > > > >>>> intention? (This was also suggested in the discussion
> thread
> > >> > before
> > >> > > > but
> > >> > > > >>> is
> > >> > > > >>>> not in the KIP).
> > >> > > > >>>>
> > >> > > > >>>> LM3. Related to my previous point, I find that to be the
> true
> > >> > > > >>> public-facing
> > >> > > > >>>> change (member ID mandatory at the protocol level), but
> it's
> > >> only
> > >> > at
> > >> > > > >> the
> > >> > > > >>>> end of the Public interfaces changes, kind of lost among
> > >> details
> > >> > of
> > >> > > > how
> > >> > > > >>>> we're going to do it. Should we rephrase that section with
> > the
> > >> > > actual
> > >> > > > >>>> change first, and the hows after (ex. Bumping the version
> is
> > >> not
> > >> > the
> > >> > > > >>>> public-facing change in this case, it's just the mechanism
> to
> > >> > > properly
> > >> > > > >>>> introduce our change)
> > >> > > > >>>>
> > >> > > > >>>> LM4. Regarding the lifetime of the UUID: the KIP states we
> > will
> > >> > > > "Verify
> > >> > > > >>>> that the UUID remains consistent across all subsequent
> > >> heartbeats
> > >> > > > >> during
> > >> > > > >>>> the session". What is this "session" referring to here? I
> > would
> > >> > > expect
> > >> > > > >>> that
> > >> > > > >>>> the UUID is associated to a consumer instance (generated
> for
> > >> the
> > >> > > > >> consumer
> > >> > > > >>>> the first time it needs to send a HB if it doesn't have the
> > >> UUID
> > >> > > yet.
> > >> > > > >>> From
> > >> > > > >>>> there on, every time it needs to send a "first HB" again,
> it
> > >> will
> > >> > > > reuse
> > >> > > > >>> its
> > >> > > > >>>> UUID, is that the intention? Note that we should consider
> > that
> > >> the
> > >> > > > same
> > >> > > > >>>> consumer instance may have many "first heartbeats", meaning
> > >> > > heartbeats
> > >> > > > >> to
> > >> > > > >>>> join the group when it's not part of it (ex. consumer
> > >> unsubscribe
> > >> > +
> > >> > > > >>>> subscribe, fenced, stale). Is this the intention or are you
> > >> > > > considering
> > >> > > > >>> the
> > >> > > > >>>> lifetime differently? We should clarify it in the KIP.
> > >> > > > >>>>
> > >> > > > >>>> Thanks!
> > >> > > > >>>>
> > >> > > > >>>> Lianet
> > >> > > > >>>>
> > >> > > > >>>> On Tue, Aug 27, 2024 at 2:27 AM TengYao Chi <
> > >> kitingiao@gmail.com>
> > >> > > > >> wrote:
> > >> > > > >>>>
> > >> > > > >>>>> Hi everyone,
> > >> > > > >>>>>
> > >> > > > >>>>> I have revised this KIP multiple times based on the
> feedback
> > >> from
> > >> > > our
> > >> > > > >>>>> discussions.
> > >> > > > >>>>> I would greatly appreciate it if you could review it when
> > you
> > >> > have
> > >> > > > >> the
> > >> > > > >>>>> time.
> > >> > > > >>>>> If there are no further comments or suggestions, I plan to
> > >> > proceed
> > >> > > > >> with
> > >> > > > >>>>> initiating a vote soon.
> > >> > > > >>>>>
> > >> > > > >>>>> Best regards,
> > >> > > > >>>>> TengYao
> > >> > > > >>>>>
> > >> > > > >>>>> TengYao Chi <ki...@gmail.com> 於 2024年8月23日 週五
> 下午2:43寫道：
> > >> > > > >>>>>
> > >> > > > >>>>>> Hi Andrew,
> > >> > > > >>>>>> Thank you for your previous feedback and insights.
> > >> > > > >>>>>> Your contributions have added valuable perspectives to
> the
> > >> > > > >>> discussions.
> > >> > > > >>>>>> And we also benefit from the comparison of different
> > >> solutions.
> > >> > > > >>>>>> I’m also looking forward to seeing an initial version in
> > >> > KIP-932,
> > >> > > > >> as
> > >> > > > >>> it
> > >> > > > >>>>>> will provide a good reference for future implementations.
> > >> > > > >>>>>>
> > >> > > > >>>>>> Regarding your comment on AS2, I wanted to clarify that
> my
> > >> > > > >>>> specification
> > >> > > > >>>>>> references org.apache.kafka.common.Uuid.
> > >> > > > >>>>>> I believe we’re referring to the same class, and it might
> > >> just
> > >> > be
> > >> > > a
> > >> > > > >>>> small
> > >> > > > >>>>>> oversight due to the busy schedule.
> > >> > > > >>>>>>
> > >> > > > >>>>>> I want to express my gratitude once again for your many
> > >> > insightful
> > >> > > > >>>>>> comments, which have helped the discussion progress
> > smoothly.
> > >> > > > >>>>>>
> > >> > > > >>>>>> Best regards,
> > >> > > > >>>>>> TengYao
> > >> > > > >>>>>>
> > >> > > > >>>>>>
> > >> > > > >>>>>> Andrew Schofield <an...@live.com> 於
> 2024年8月22日
> > 週四
> > >> > > > >>>> 下午11:28寫道：
> > >> > > > >>>>>>
> > >> > > > >>>>>>> Hi TengYao,
> > >> > > > >>>>>>> I’ve been reading through the comments and I’m happy
> that
> > >> the
> > >> > > > >> lobby
> > >> > > > >>>>>>> approach has not gained support.
> > >> > > > >>>>>>>
> > >> > > > >>>>>>> Assuming that this KIP is voted, I will be happy to
> change
> > >> > > KIP-932
> > >> > > > >>> so
> > >> > > > >>>>>>> that it only supports client-side member ID generation.
> > >> Because
> > >> > > > >> that
> > >> > > > >>>> KIP
> > >> > > > >>>>>>> is still
> > >> > > > >>>>>>> under development, I can do this in the first version of
> > >> > > > >>>>>>> ShareGroupHeartbeat.
> > >> > > > >>>>>>>
> > >> > > > >>>>>>> AS2: For the encoding section, I suppose the specific
> > >> encoding
> > >> > > > >> which
> > >> > > > >>>>>>> is used is what org.apache.kafka.utils.Uuid uses.
> > >> > > > >>>>>>>
> > >> > > > >>>>>>> Thanks,
> > >> > > > >>>>>>> Andrew
> > >> > > > >>>>>>>
> > >> > > > >>>>>>>> On 14 Aug 2024, at 17:00, TengYao Chi <
> > kitingiao@gmail.com
> > >> >
> > >> > > > >>> wrote:
> > >> > > > >>>>>>>>
> > >> > > > >>>>>>>> Hello Apoorv,
> > >> > > > >>>>>>>> Thank you for your feedback.
> > >> > > > >>>>>>>> Regarding the questions you raised, unfortunately, this
> > KIP
> > >> > > > >> cannot
> > >> > > > >>>>>>>> guarantee the order of heartbeats. As with many classic
> > >> > > > >>> distributed
> > >> > > > >>>>>>> system
> > >> > > > >>>>>>>> challenges, what we can do is make our best effort to
> > >> ensure
> > >> > > > >> that
> > >> > > > >>>>> there
> > >> > > > >>>>>>> are
> > >> > > > >>>>>>>> no idle members or stale assignments under normal
> > >> > circumstances.
> > >> > > > >>>>>>>>
> > >> > > > >>>>>>>> As for the lobby approach, I’m not a fan of it because
> it
> > >> > > > >> requires
> > >> > > > >>>>>>> adding a
> > >> > > > >>>>>>>> mechanism to maintain client state within the
> > >> ConsumerGroup,
> > >> > > > >>> which,
> > >> > > > >>>> in
> > >> > > > >>>>>>> my
> > >> > > > >>>>>>>> view, resembles something like a two-phase commit. This
> > >> would
> > >> > > > >>>>> introduce
> > >> > > > >>>>>>>> more complexity than the proposal in this KIP, which is
> > >> > > > >> something
> > >> > > > >>> we
> > >> > > > >>>>>>> want
> > >> > > > >>>>>>>> to avoid. KIP-848 aims to simplify the existing
> protocol,
> > >> and
> > >> > > > >>> while
> > >> > > > >>>>> the
> > >> > > > >>>>>>>> lobby approach is a good one, I believe it is not the
> > right
> > >> > fit
> > >> > > > >>> for
> > >> > > > >>>>> this
> > >> > > > >>>>>>>> particular situation.
> > >> > > > >>>>>>>>
> > >> > > > >>>>>>>> Best regards,
> > >> > > > >>>>>>>> TengYao
> > >> > > > >>>>>>>>
> > >> > > > >>>>>>>> TengYao Chi <ki...@gmail.com> 於 2024年8月14日 週三
> > >> 下午11:45寫道：
> > >> > > > >>>>>>>>
> > >> > > > >>>>>>>>> Hi David,
> > >> > > > >>>>>>>>>
> > >> > > > >>>>>>>>> I really appreciate your review and suggestions. As I
> am
> > >> > still
> > >> > > > >>>>> gaining
> > >> > > > >>>>>>>>> experience in writing KIPs, your input has been
> > incredibly
> > >> > > > >>>> helpful. I
> > >> > > > >>>>>>> am
> > >> > > > >>>>>>>>> currently applying your suggestions to the KIP and
> will
> > >> > > > >> complete
> > >> > > > >>> it
> > >> > > > >>>>> as
> > >> > > > >>>>>>> soon
> > >> > > > >>>>>>>>> as possible.
> > >> > > > >>>>>>>>> Regarding the UUID part, I think we haven’t reached a
> > >> > > > >> conclusion
> > >> > > > >>>>>>> yet.(So
> > >> > > > >>>>>>>>> far according to this thread)
> > >> > > > >>>>>>>>> However, I will review the current implementation in
> the
> > >> > Kafka
> > >> > > > >>>> `Uuid`
> > >> > > > >>>>>>>>> class and include a brief specification in the KIP.
> > >> > > > >>>>>>>>>
> > >> > > > >>>>>>>>> Once again, thank you so much for your help.
> > >> > > > >>>>>>>>>
> > >> > > > >>>>>>>>> Best regards,
> > >> > > > >>>>>>>>> TengYao
> > >> > > > >>>>>>>>>
> > >> > > > >>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月14日 週三
> > >> > 下午11:14寫道：
> > >> > > > >>>>>>>>>
> > >> > > > >>>>>>>>>> hi Apoorv
> > >> > > > >>>>>>>>>>
> > >> > > > >>>>>>>>>>> As the memberId is now known to the client, and
> client
> > >> > might
> > >> > > > >>> send
> > >> > > > >>>>> the
> > >> > > > >>>>>>>>>> leave
> > >> > > > >>>>>>>>>> group heartbeat on shutdown prior to receiving the
> > >> initial
> > >> > > > >>>> heartbeat
> > >> > > > >>>>>>>>>> response. If that's true then how do we guarantee
> that
> > >> the 2
> > >> > > > >>>>> requests
> > >> > > > >>>>>>> to
> > >> > > > >>>>>>>>>> join and leave will be processed in order, which
> could
> > >> still
> > >> > > > >>> leave
> > >> > > > >>>>>>> stale
> > >> > > > >>>>>>>>>> members or throw unknown member id exceptions?
> > >> > > > >>>>>>>>>>
> > >> > > > >>>>>>>>>> This is definitely a good question. the short answer:
> > no
> > >> > > > >>> guarantee
> > >> > > > >>>>> but
> > >> > > > >>>>>>>>>> best
> > >> > > > >>>>>>>>>> efforts
> > >> > > > >>>>>>>>>>
> > >> > > > >>>>>>>>>> Please notice the root cause is "we have no enough
> time
> > >> to
> > >> > > > >> wait
> > >> > > > >>>>>>> member id
> > >> > > > >>>>>>>>>> (response) when closing consumer". Sadly, we can'
> > >> guarantee
> > >> > > > >> the
> > >> > > > >>>>>>> request
> > >> > > > >>>>>>>>>> order due to the same reason.
> > >> > > > >>>>>>>>>>
> > >> > > > >>>>>>>>>> However, in contrast to previous behavior, there is
> one
> > >> big
> > >> > > > >>>> benefit
> > >> > > > >>>>>>> of new
> > >> > > > >>>>>>>>>> approach - we can try STONITH because we know the
> > member
> > >> id
> > >> > > > >>>>>>>>>>
> > >> > > > >>>>>>>>>> Best,
> > >> > > > >>>>>>>>>> Chia-Ping
> > >> > > > >>>>>>>>>>
> > >> > > > >>>>>>>>>>
> > >> > > > >>>>>>>>>> Apoorv Mittal <ap...@gmail.com> 於
> 2024年8月14日
> > 週三
> > >> > > > >>>> 下午8:55寫道：
> > >> > > > >>>>>>>>>>
> > >> > > > >>>>>>>>>>> Hi TengYao,
> > >> > > > >>>>>>>>>>> Thanks for the KIP. Continuing on the point which
> > Andrew
> > >> > > > >>>> mentioned
> > >> > > > >>>>> as
> > >> > > > >>>>>>>>>> AS1.
> > >> > > > >>>>>>>>>>>
> > >> > > > >>>>>>>>>>> As the memberId is now known to the client, and
> client
> > >> > might
> > >> > > > >>> send
> > >> > > > >>>>> the
> > >> > > > >>>>>>>>>> leave
> > >> > > > >>>>>>>>>>> group heartbeat on shutdown prior to receiving the
> > >> initial
> > >> > > > >>>>> heartbeat
> > >> > > > >>>>>>>>>>> response. If that's true then how do we guarantee
> that
> > >> the
> > >> > 2
> > >> > > > >>>>>>> requests to
> > >> > > > >>>>>>>>>>> join and leave will be processed in order, which
> could
> > >> > still
> > >> > > > >>>> leave
> > >> > > > >>>>>>> stale
> > >> > > > >>>>>>>>>>> members or throw unknown member id exceptions?
> > >> > > > >>>>>>>>>>>
> > >> > > > >>>>>>>>>>> Though the client side member id generation is
> helpful
> > >> > which
> > >> > > > >>> will
> > >> > > > >>>>>>>>>> represent
> > >> > > > >>>>>>>>>>> the same group perspective as from client and
> broker's
> > >> end.
> > >> > > > >>> But I
> > >> > > > >>>>>>> think
> > >> > > > >>>>>>>>>> the
> > >> > > > >>>>>>>>>>> major concern we want to solve here is Stale
> Partition
> > >> > > > >>>> Assignments
> > >> > > > >>>>>>> which
> > >> > > > >>>>>>>>>>> might still exist with the new approach. I am
> leaning
> > >> > towards
> > >> > > > >>> the
> > >> > > > >>>>>>>>>>> suggestion mentioned by Andrew where partition
> > >> assignment
> > >> > > > >>>> triggers
> > >> > > > >>>>> on
> > >> > > > >>>>>>>>>>> subsequent heartbeat when client acknowledges the
> > >> initial
> > >> > > > >>>>> heartbeat,
> > >> > > > >>>>>>>>>>> delayed partition assignment.
> > >> > > > >>>>>>>>>>>
> > >> > > > >>>>>>>>>>> Though on a separate note, I have a different
> > question.
> > >> > What
> > >> > > > >>>>> happens
> > >> > > > >>>>>>>>>> when
> > >> > > > >>>>>>>>>>> there is an issue with the client which sends the
> > >> initial
> > >> > > > >>>> heartbeat
> > >> > > > >>>>>>>>>> without
> > >> > > > >>>>>>>>>>> memberId, then crashes and restarts? I think we must
> > be
> > >> > > > >>>>> re-triggering
> > >> > > > >>>>>>>>>>> assignments and expiring members only after the
> > >> heartbeat
> > >> > > > >>> session
> > >> > > > >>>>>>>>>> timeout?
> > >> > > > >>>>>>>>>>> If that's true then shall delayed partition
> assignment
> > >> can
> > >> > > > >> help
> > >> > > > >>>>>>> benefit
> > >> > > > >>>>>>>>>> us
> > >> > > > >>>>>>>>>>> from this situation as well?
> > >> > > > >>>>>>>>>>>
> > >> > > > >>>>>>>>>>> Regards,
> > >> > > > >>>>>>>>>>> Apoorv Mittal
> > >> > > > >>>>>>>>>>>
> > >> > > > >>>>>>>>>>>
> > >> > > > >>>>>>>>>>> On Wed, Aug 14, 2024 at 12:51 PM David Jacot
> > >> > > > >>>>>>>>>> <dj...@confluent.io.invalid>
> > >> > > > >>>>>>>>>>> wrote:
> > >> > > > >>>>>>>>>>>
> > >> > > > >>>>>>>>>>>> Hi Andrew,
> > >> > > > >>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>> Personally, I don't like the lobby approach. It
> makes
> > >> > things
> > >> > > > >>>> more
> > >> > > > >>>>>>>>>>>> complicated and it would require changing the
> records
> > >> on
> > >> > the
> > >> > > > >>>>> server
> > >> > > > >>>>>>>>>> too.
> > >> > > > >>>>>>>>>>>> This is why I initially suggested the rejected
> > >> alternative
> > >> > > > >> #2
> > >> > > > >>>>> which
> > >> > > > >>>>>>> is
> > >> > > > >>>>>>>>>>>> pretty close but also not perfect.
> > >> > > > >>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>> I'd like to clarify one thing. The
> > >> ConsumerGroupHeartbeat
> > >> > > > >> API
> > >> > > > >>>>>>> already
> > >> > > > >>>>>>>>>>>> supports generating the member id on the client so
> we
> > >> > don't
> > >> > > > >>> need
> > >> > > > >>>>> any
> > >> > > > >>>>>>>>>>>> conditional logic on the client side. This is
> > actually
> > >> > what
> > >> > > > >> we
> > >> > > > >>>>>>> wanted
> > >> > > > >>>>>>>>>> to
> > >> > > > >>>>>>>>>>> do
> > >> > > > >>>>>>>>>>>> in the first place but the idea got pushed back by
> > >> Magnus
> > >> > > > >> back
> > >> > > > >>>>> then
> > >> > > > >>>>>>>>>>> because
> > >> > > > >>>>>>>>>>>> generating uuid from librdkafka required a new
> > >> dependency.
> > >> > > > >> It
> > >> > > > >>>>> turns
> > >> > > > >>>>>>>>>> out
> > >> > > > >>>>>>>>>>>> that librdkafka has that dependency today. In
> > >> retrospect,
> > >> > we
> > >> > > > >>>>> should
> > >> > > > >>>>>>>>>> have
> > >> > > > >>>>>>>>>>>> pushed back on this. Long story short, we can just
> do
> > >> it.
> > >> > > > >> The
> > >> > > > >>>>>>>>>> proposal in
> > >> > > > >>>>>>>>>>>> this KIP is to make the member id required in
> future
> > >> > > > >> versions.
> > >> > > > >>>> We
> > >> > > > >>>>>>>>>> could
> > >> > > > >>>>>>>>>>>> also decide not to do it and to keep supporting
> both
> > >> > > > >>>> approaches. I
> > >> > > > >>>>>>>>>> would
> > >> > > > >>>>>>>>>>>> also be fine with this.
> > >> > > > >>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>> Best,
> > >> > > > >>>>>>>>>>>> David
> > >> > > > >>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>> On Wed, Aug 14, 2024 at 12:30 PM Andrew Schofield <
> > >> > > > >>>>>>>>>>>> andrew_schofield@live.com>
> > >> > > > >>>>>>>>>>>> wrote:
> > >> > > > >>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>> Hi TengYao,
> > >> > > > >>>>>>>>>>>>> Thanks for your response. I’ll have just one more
> > try
> > >> to
> > >> > > > >>>>> persuade.
> > >> > > > >>>>>>>>>>>>> I feel that I will need to follow the approach
> with
> > >> > KIP-932
> > >> > > > >>>> when
> > >> > > > >>>>>>>>>> we’ve
> > >> > > > >>>>>>>>>>>>> made a decision, so I do have more than a passing
> > >> > interest
> > >> > > > >> in
> > >> > > > >>>>> this.
> > >> > > > >>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>> A group member in the lobby is in the group, but
> it
> > >> does
> > >> > > > >> not
> > >> > > > >>>> have
> > >> > > > >>>>>>>>>> any
> > >> > > > >>>>>>>>>>>>> assignments. A member of a consumer group can have
> > no
> > >> > > > >>> assigned
> > >> > > > >>>>>>>>>>>>> partitions (such as 5 CG members subscribed to a
> > topic
> > >> > > > >> with 4
> > >> > > > >>>>>>>>>>>> partitions),
> > >> > > > >>>>>>>>>>>>> so it’s a situation that consumer group members
> > >> already
> > >> > > > >>> expect.
> > >> > > > >>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>> One of Kafka’s strengths is the way that we handle
> > API
> > >> > > > >>>>> versioning.
> > >> > > > >>>>>>>>>>>>> But, there is a cost - the behaviour is different
> > >> > depending
> > >> > > > >>> on
> > >> > > > >>>>> the
> > >> > > > >>>>>>>>>> RPC
> > >> > > > >>>>>>>>>>>>> version. KIP-848 is on the cusp of completion, but
> > >> we’re
> > >> > > > >>>> already
> > >> > > > >>>>>>>>>> adding
> > >> > > > >>>>>>>>>>>>> conditional logic for v0/v1 for
> > >> ConsumerGroupHeartbeat.
> > >> > > > >>> That’s
> > >> > > > >>>> a
> > >> > > > >>>>>>>>>> pity.
> > >> > > > >>>>>>>>>>>>> Only a minor issue, but it’s unfortunate.
> > >> > > > >>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>> Thanks,
> > >> > > > >>>>>>>>>>>>> Andrew
> > >> > > > >>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>> On 14 Aug 2024, at 08:47, TengYao Chi <
> > >> > > > >> kitingiao@gmail.com>
> > >> > > > >>>>>>>>>> wrote:
> > >> > > > >>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>> Hello Andrew
> > >> > > > >>>>>>>>>>>>>> Thank you for your thoughtful suggestions and
> > getting
> > >> > the
> > >> > > > >>>>>>>>>> discussion
> > >> > > > >>>>>>>>>>>>> going.
> > >> > > > >>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>> To AS1:
> > >> > > > >>>>>>>>>>>>>> In the current scenario where the server
> generates
> > >> the
> > >> > > > >> UUID,
> > >> > > > >>>> if
> > >> > > > >>>>>>>>>> the
> > >> > > > >>>>>>>>>>>>> client
> > >> > > > >>>>>>>>>>>>>> shuts down before receiving the memberId
> generated
> > by
> > >> > the
> > >> > > > >> GC
> > >> > > > >>>>>>>>>>>> (regardless
> > >> > > > >>>>>>>>>>>>> of
> > >> > > > >>>>>>>>>>>>>> whether it’s a graceful shutdown or not), the GC
> > will
> > >> > > > >> still
> > >> > > > >>>> have
> > >> > > > >>>>>>>>>> to
> > >> > > > >>>>>>>>>>>> wait
> > >> > > > >>>>>>>>>>>>>> for the heartbeat timeout because the client
> > doesn’t
> > >> > know
> > >> > > > >>> its
> > >> > > > >>>>>>>>>>> memberId.
> > >> > > > >>>>>>>>>>>>>> This KIP indeed cannot completely resolve the
> > >> > idempotency
> > >> > > > >>>> issue,
> > >> > > > >>>>>>>>>> but
> > >> > > > >>>>>>>>>>> it
> > >> > > > >>>>>>>>>>>>> can
> > >> > > > >>>>>>>>>>>>>> better handle shutdown scenarios under normal
> > >> > > > >> circumstances
> > >> > > > >>>>>>>>>> because
> > >> > > > >>>>>>>>>>> the
> > >> > > > >>>>>>>>>>>>>> client always knows its memberId. Even if the
> > client
> > >> > shuts
> > >> > > > >>>> down
> > >> > > > >>>>>>>>>>>>> immediately
> > >> > > > >>>>>>>>>>>>>> after the initial heartbeat, as long as it
> > performs a
> > >> > > > >>> graceful
> > >> > > > >>>>>>>>>>> shutdown
> > >> > > > >>>>>>>>>>>>> and
> > >> > > > >>>>>>>>>>>>>> sends a leave heartbeat, the GC can manage the
> > >> situation
> > >> > > > >> and
> > >> > > > >>>>>>>>>> remove
> > >> > > > >>>>>>>>>>> the
> > >> > > > >>>>>>>>>>>>>> member. Therefore, the goal of this KIP is to
> > address
> > >> > the
> > >> > > > >>>> issue
> > >> > > > >>>>>>>>>> where
> > >> > > > >>>>>>>>>>>> the
> > >> > > > >>>>>>>>>>>>>> GC has to wait for the heartbeat timeout due to
> the
> > >> > client
> > >> > > > >>>>> leaving
> > >> > > > >>>>>>>>>>>>> without
> > >> > > > >>>>>>>>>>>>>> knowing its memberId, which leads to reduced
> > >> throughput
> > >> > > > >> and
> > >> > > > >>>>>>>>>> limited
> > >> > > > >>>>>>>>>>>>>> scalability.
> > >> > > > >>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>> The solution you suggest has also been proposed
> by
> > >> > David.
> > >> > > > >>> The
> > >> > > > >>>>>>>>>> concern
> > >> > > > >>>>>>>>>>>>> with
> > >> > > > >>>>>>>>>>>>>> this approach is that it introduces additional
> > >> > complexity
> > >> > > > >>> for
> > >> > > > >>>>>>>>>>>>>> compatibility, as the new server would not
> > >> immediately
> > >> > add
> > >> > > > >>> the
> > >> > > > >>>>>>>>>> member
> > >> > > > >>>>>>>>>>>> to
> > >> > > > >>>>>>>>>>>>>> the group, while the old server would. This
> > requires
> > >> > > > >> clients
> > >> > > > >>>> to
> > >> > > > >>>>>>>>>>>>>> differentiate whether their memberId has been
> added
> > >> to
> > >> > the
> > >> > > > >>>> group
> > >> > > > >>>>>>>>>> or
> > >> > > > >>>>>>>>>>>> not,
> > >> > > > >>>>>>>>>>>>>> which could result in unexpected logs.
> > >> > > > >>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>> Best Regards,
> > >> > > > >>>>>>>>>>>>>> TengYao
> > >> > > > >>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>> Andrew Schofield <an...@live.com> 於
> > >> > 2024年8月14日
> > >> > > > >>> 週三
> > >> > > > >>>>>>>>>>>> 上午12:29寫道：
> > >> > > > >>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>> Hi TengYao,
> > >> > > > >>>>>>>>>>>>>>> Thanks for the KIP. I wonder if there’s a
> > different
> > >> way
> > >> > > > >> to
> > >> > > > >>>>> close
> > >> > > > >>>>>>>>>>> what
> > >> > > > >>>>>>>>>>>>>>> is quite a small window.
> > >> > > > >>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>> AS1: It is true that the initial heartbeat is
> not
> > >> > > > >>> idempotent,
> > >> > > > >>>>> but
> > >> > > > >>>>>>>>>>> this
> > >> > > > >>>>>>>>>>>>>>> remains
> > >> > > > >>>>>>>>>>>>>>> true with this KIP. It’s just differently not
> > >> > idempotent.
> > >> > > > >>> If
> > >> > > > >>>>> the
> > >> > > > >>>>>>>>>>>> client
> > >> > > > >>>>>>>>>>>>>>> makes its
> > >> > > > >>>>>>>>>>>>>>> own member ID, sends a request and dies, the GC
> > will
> > >> > > > >> still
> > >> > > > >>>> have
> > >> > > > >>>>>>>>>>> added
> > >> > > > >>>>>>>>>>>>>>> the member to the group and it will hang around
> > >> until
> > >> > the
> > >> > > > >>>>> session
> > >> > > > >>>>>>>>>>>>> expires.
> > >> > > > >>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>> I wonder if the GC could still generate the
> member
> > >> ID
> > >> > in
> > >> > > > >>>>>>>>>> response to
> > >> > > > >>>>>>>>>>>> the
> > >> > > > >>>>>>>>>>>>>>> first
> > >> > > > >>>>>>>>>>>>>>> heartbeat, and put the member in a special
> PENDING
> > >> > state
> > >> > > > >>> with
> > >> > > > >>>>> no
> > >> > > > >>>>>>>>>>>>>>> assignments until the client sends the next
> > >> heartbeat,
> > >> > > > >> thus
> > >> > > > >>>>>>>>>>> confirming
> > >> > > > >>>>>>>>>>>>> it
> > >> > > > >>>>>>>>>>>>>>> has received the member ID. This would not be a
> > >> > protocol
> > >> > > > >>>> change
> > >> > > > >>>>>>>>>> at
> > >> > > > >>>>>>>>>>>> all,
> > >> > > > >>>>>>>>>>>>>>> just
> > >> > > > >>>>>>>>>>>>>>> a change to the GC to keep a new member in the
> > lobby
> > >> > > > >> until
> > >> > > > >>>> it’s
> > >> > > > >>>>>>>>>>>>> comfirmed
> > >> > > > >>>>>>>>>>>>>>> it knows its member ID.
> > >> > > > >>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>> Thanks,
> > >> > > > >>>>>>>>>>>>>>> Andrew
> > >> > > > >>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>> On 13 Aug 2024, at 15:59, TengYao Chi <
> > >> > > > >>> kitingiao@gmail.com>
> > >> > > > >>>>>>>>>> wrote:
> > >> > > > >>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>> Hi Chia-Ping,
> > >> > > > >>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>> Thanks for review and suggestions.
> > >> > > > >>>>>>>>>>>>>>>> I have updated the content of KIP accordingly.
> > >> > > > >>>>>>>>>>>>>>>> Please take a look.
> > >> > > > >>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>> Best regards,
> > >> > > > >>>>>>>>>>>>>>>> TengYao
> > >> > > > >>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > 2024年8月13日
> > >> 週二
> > >> > > > >>>>> 下午9:45寫道：
> > >> > > > >>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>>> hi TengYao
> > >> > > > >>>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>>> thanks for this KIP.
> > >> > > > >>>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>>> 1) could you please describe the before/after
> > >> > behavior
> > >> > > > >> in
> > >> > > > >>>> the
> > >> > > > >>>>>>>>>>>>> "Proposed
> > >> > > > >>>>>>>>>>>>>>>>> Changes" section? IIRC, current RPC allows HB
> > >> having
> > >> > > > >>> member
> > >> > > > >>>>> id
> > >> > > > >>>>>>>>>>>>>>> generated by
> > >> > > > >>>>>>>>>>>>>>>>> client, right? If HB has no member ID, server
> > will
> > >> > > > >>> generate
> > >> > > > >>>>> one
> > >> > > > >>>>>>>>>>> and
> > >> > > > >>>>>>>>>>>>> then
> > >> > > > >>>>>>>>>>>>>>>>> return. The new behavior will enforce HB
> "must"
> > >> have
> > >> > > > >>> member
> > >> > > > >>>>> ID.
> > >> > > > >>>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>>> 2) could you please write the version number
> > >> > explicitly
> > >> > > > >>> in
> > >> > > > >>>>> the
> > >> > > > >>>>>>>>>> KIP
> > >> > > > >>>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>>> 3) how new client code handle the old HB? Does
> > it
> > >> > > > >> always
> > >> > > > >>>>>>>>>> generate
> > >> > > > >>>>>>>>>>>>> member
> > >> > > > >>>>>>>>>>>>>>>>> ID on client-side even though that is not
> > >> restricted?
> > >> > > > >>>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>>> Best,
> > >> > > > >>>>>>>>>>>>>>>>> Chia-Ping
> > >> > > > >>>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>>> On 2024/08/13 06:20:42 TengYao Chi wrote:
> > >> > > > >>>>>>>>>>>>>>>>>> Hello everyone,
> > >> > > > >>>>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>>>> I would like to start a discussion thread on
> > >> > KIP-1082,
> > >> > > > >>>> which
> > >> > > > >>>>>>>>>>>> proposes
> > >> > > > >>>>>>>>>>>>>>>>>> enabling id generation for clients over the
> > >> > > > >>>>>>>>>>> ConsumerGroupHeartbeat
> > >> > > > >>>>>>>>>>>>> RPC.
> > >> > > > >>>>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>>>> Here is the KIP Link: KIP-1082
> > >> > > > >>>>>>>>>>>>>>>>>> <
> > >> > > > >>>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>
> > >> > > > >>>>>>>>>>
> > >> > > > >>>>>>>
> > >> > > > >>>>>
> > >> > > > >>>>
> > >> > > > >>>
> > >> > > > >>
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
> > >> > > > >>>>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>>>> Please take a look and let me know what you
> > >> think,
> > >> > > > >> and I
> > >> > > > >>>>> would
> > >> > > > >>>>>>>>>>>>>>> appreciate
> > >> > > > >>>>>>>>>>>>>>>>>> any suggestions and feedback.
> > >> > > > >>>>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>>>> Best regards,
> > >> > > > >>>>>>>>>>>>>>>>>> TengYao
> > >> > > > >>>>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>>
> > >> > > > >>>>>>>>>>>
> > >> > > > >>>>>>>>>>
> > >> > > > >>>>>>>>>
> > >> > > > >>>>>>>
> > >> > > > >>>>>>>
> > >> > > > >>>>>
> > >> > > > >>>>
> > >> > > > >>>
> > >> > > > >>
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[David Jacot <dj...@confluent.io.INVALID>]

Hi,

Thanks for the update. I have a few nits:

> If the member ID is null or empty, the server will reject the request
with an InvalidRequestException.
We should clarify that this should only apply to version >= 1.

> The consumer instance must generate a member ID, and this ID should
remain consistent for the duration of the consumer's session. Here, a
"session" is defined as the period from the consumer's first heartbeat
until it leaves the group, either through a graceful shutdown, a heartbeat
timeout, or the process stopping or dying. The consumer instance should
reuse the same member ID for all heartbeats and rejoin attempts to maintain
continuity within the group.

This part is not clear to me. When the member leaves the group, it should
not reset the member id. I would rather say that the member must generate
its member id when it starts and it must keep it until the process stops.
It is basically an incarnation of the process.

> If a conflict arises where the member ID generated by the client is
detected to be a duplicate within the same group (for example, the same
member ID is associated with another active member in the group), the
server will handle this by comparing the memberEpoch values of the
conflicting members. The member with the lower memberEpoch is considered
outdated and will be fenced off by the server. When this occurs, the server
responds with a FENCED_MEMBER_EPOCH error to the client, signaling it to
rejoin the group with the same member ID while resetting the memberEpoch to
zero. This ensures that the client properly resynchronizes and maintains
the continuity and consistency of the group membership.

This part is not clear either. It basically says that if a member joins
with an existing member id but a different epoch, it will be fenced. Then
it must rejoin with the same member id and epoch zero. This is already the
current behavior and it does not help with detecting duplicates, right?
Should we just remove the paragraph?

> A member ID mismatch occurs within a session: If the server detects a
mismatch between the provided member ID and the expected member ID for an
ongoing session, it should return a UNKNOWN_MEMBER_ID  error.

How could we detect a mismatch between the provided and the expected member
id? My understanding is that we can only know whether the provided member
id exists or not. This is already implemented.

Thanks,
David

On Sat, Sep 14, 2024 at 9:31 AM TengYao Chi <ki...@gmail.com> wrote:

> Hello everyone,
>
> Since this KIP has been fully discussed, I will initiate a vote for it next
> Monday.
> Thank you and have a nice weekend.
>
> Best regards,
> TengYao
>
> TengYao Chi <ki...@gmail.com> 於 2024年9月5日 週四 下午2:19寫道：
>
> > Hello everyone,
> >
> > KT2: It looks like everyone who has expressed an opinion supports the
> > second option: “Document a recommendation for clients to use UUIDs as
> > member IDs, without strictly enforcing it.”
> > I have updated the KIP accordingly.
> > Please take a look, and let me know if you have any thoughts or feedback.
> >
> > Thank you!
> >
> > Best regards,
> > TengYao
> >
> > Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月30日 週五 下午9:56寫道：
> >
> >> hi TengYao
> >>
> >> KT2: +1 to second approach
> >>
> >> Best,
> >> Chia-Ping
> >>
> >>
> >> David Jacot <dj...@confluent.io.invalid> 於 2024年8月30日 週五 下午9:15寫道：
> >>
> >> > Hi TengYao,
> >> >
> >> > KT2: I don't think that we can realistically validate the uuid on the
> >> > server. It is basically a string of chars. So I lean towards having a
> >> good
> >> > recommendation in the KIP and in the document of the field in the
> RPC's
> >> > definition.
> >> >
> >> > Best,
> >> > David
> >> >
> >> > On Fri, Aug 30, 2024 at 3:02 PM TengYao Chi <ki...@gmail.com>
> >> wrote:
> >> >
> >> > > Hello Kirk !
> >> > >
> >> > > Thank you for your comments !
> >> > >
> >> > > KT1: Yes, you are correct. The issue is not unique to the initial
> >> > > heartbeat; there can always be cases where the broker might lose
> >> > connection
> >> > > with a member.
> >> > >
> >> > > KT2: Currently, if the client doesn't have a member ID and the
> >> > memberEpoch
> >> > > equals 0, the coordinator will generate a UUID as the member ID for
> >> the
> >> > > client. However, at the RPC level, the member ID is sent as a
> literal
> >> > > string, meaning there are no restrictions on the format at this
> level.
> >> > > This also reminds me that we haven't reached a final conclusion on
> >> how to
> >> > > enforce the use of UUIDs.
> >> > > From our previous discussions, I recall two possible approaches:
> >> > > The first is to validate the UUID on the server side, and if it's
> not
> >> > > valid, throw an exception to the client.
> >> > > The second is to document a recommendation for clients to use UUIDs
> as
> >> > > member IDs, without strictly enforcing it.
> >> > > I think it's time to decide on the approach we want to take.
> >> > >
> >> > > KT3: Yes, "session" can be considered synonymous with "membership"
> in
> >> > this
> >> > > context.
> >> > >
> >> > > KT4: Thank you for pointing that out. I will update the wording to
> >> > > specifically say this behavior is for consumers.
> >> > >
> >> > > Thanks again for your comments.
> >> > >
> >> > > Best regards,
> >> > > TengYao
> >> > >
> >> > > Kirk True <ki...@kirktrue.pro> 於 2024年8月30日 週五 上午12:39寫道：
> >> > >
> >> > > > Hi TengYao!
> >> > > >
> >> > > > Sorry for being late to the discussion...
> >> > > >
> >> > > > After reading the thread and then the KIP, I had a few
> >> > > questions/comments:
> >> > > >
> >> > > > KT1: In Motivation, it states: "This scenario can result in the
> >> broker
> >> > > > registering a new member for which it will never receive a proper
> >> leave
> >> > > > request.” Just to be clear, the broker will always have cases
> where
> >> it
> >> > > > might lose connection with a member. That’s not unique to the
> >> initial
> >> > > > heartbeat, right?
> >> > > >
> >> > > > KT2: There was a bit of back and forth about format of the member
> >> ID.
> >> > > From
> >> > > > what I gathered in the thread, the member ID is still defined in
> the
> >> > RPC
> >> > > as
> >> > > > a string and not a UUID, right? The KIP states that the “client
> must
> >> > > > generate a UUID as the member ID” and that the “server will
> validate
> >> > > that a
> >> > > > valid UUID is provided.” Is that a change for the server, or is it
> >> > > already
> >> > > > enforced as a UUID?
> >> > > >
> >> > > > KT3: Lianet mentioned some confusion over the use of the word
> >> > “session.”
> >> > > > Isn’t “session” synonymous with “membership?”
> >> > > >
> >> > > > KT4: Under “Member ID Lifecycle,” it states: "The client should
> >> reuse
> >> > the
> >> > > > same UUID as the member ID for all heartbeats and rejoin attempts
> to
> >> > > > maintain continuity within the group.” Could we change the first
> >> part
> >> > of
> >> > > > that to “The Consumer instance should…” We do have lifetimes that
> >> > extend
> >> > > > past the lifetime of a client instance (such as the transaction
> ID).
> >> > > >
> >> > > > Thanks,
> >> > > > Kirk
> >> > > >
> >> > > > > On Aug 29, 2024, at 1:28 AM, TengYao Chi <ki...@gmail.com>
> >> > wrote:
> >> > > > >
> >> > > > > Hi David,
> >> > > > >
> >> > > > > Thank you for pointing that out.
> >> > > > > I have updated the content of the KIP based on Lianet's and your
> >> > > > feedback.
> >> > > > > Please take a look and let me know your thoughts.
> >> > > > >
> >> > > > > Best regards,
> >> > > > > TengYao
> >> > > > >
> >> > > > > David Jacot <dj...@confluent.io.invalid> 於 2024年8月29日 週四
> >> 下午3:20寫道：
> >> > > > >
> >> > > > >> Hi TengYao,
> >> > > > >>
> >> > > > >> Thanks for the update. I haven't fully read it yet but I will
> >> soon.
> >> > > > >>
> >> > > > >> LM4: This is incorrect. The consumer must keep its member id
> >> during
> >> > > its
> >> > > > >> entire lifetime (until the process stops or dies). The protocol
> >> > > > stipulates
> >> > > > >> that a member must rejoin with the same member id and the
> member
> >> > epoch
> >> > > > set
> >> > > > >> to zero when an FENCED_MEMBER_EPOCH occurs. This allows the
> >> member
> >> > to
> >> > > > >> resynchronize itself. We should not change this behavior. I
> think
> >> > that
> >> > > > we
> >> > > > >> should see the client side generation id as an incarnation id
> of
> >> the
> >> > > > >> application. It is generated once and kept until it stops or
> >> dies.
> >> > > > >>
> >> > > > >> Best,
> >> > > > >> David
> >> > > > >>
> >> > > > >> On Thu, Aug 29, 2024 at 6:21 AM TengYao Chi <
> kitingiao@gmail.com
> >> >
> >> > > > wrote:
> >> > > > >>
> >> > > > >>> Hello Lianet !
> >> > > > >>>
> >> > > > >>> Thanks for the reviews and suggestions!
> >> > > > >>>
> >> > > > >>> LM1: Indeed, we plan to enforce client-side ID generation in
> the
> >> > > > future,
> >> > > > >>> and it is not an alternative. I will change the title
> >> accordingly.
> >> > > > >>>
> >> > > > >>> LM2: Yes, that's the expectation. I will add that statement to
> >> the
> >> > > > public
> >> > > > >>> interface section.
> >> > > > >>>
> >> > > > >>> LM3: Thank you for the high-level perspective review. I think
> >> > you're
> >> > > > >> right;
> >> > > > >>> our intention isn't very clear since it was placed at the end
> of
> >> > the
> >> > > > >>> section. I will try to rephrase that section to make it more
> >> > obvious.
> >> > > > >>>
> >> > > > >>> LM4: Regarding the definition of "session" in this KIP, I
> >> believe
> >> > it
> >> > > > >> refers
> >> > > > >>> to the period between the *first-time heartbeat* and when the
> >> > > *consumer
> >> > > > >>> leaves the group* (whether through a graceful shutdown or a
> >> > heartbeat
> >> > > > >>> timeout). The consumer should reuse its UUID if it has been
> >> > generated
> >> > > > >>> before. The only situation in which it will regenerate the
> UUID
> >> is
> >> > if
> >> > > > the
> >> > > > >>> coordinator finds that there is already a consumer with the
> same
> >> > > UUID.
> >> > > > >>> IIRC, the coordinator should compare the member epochs, and
> the
> >> > > > >>> later-joined consumer should be fenced off by the coordinator
> >> due
> >> > to
> >> > > > >> having
> >> > > > >>> a lower member epoch. Once the consumer receives a
> >> > > > `FENCED_MEMBER_EPOCH`
> >> > > > >>> error, it will generate a new UUID and attempt to rejoin. I
> will
> >> > > > clarify
> >> > > > >>> this in the KIP.
> >> > > > >>>
> >> > > > >>> Thanks again for your reviews, I really appreciate it.
> >> > > > >>>
> >> > > > >>> Best regards,
> >> > > > >>> TengYao
> >> > > > >>>
> >> > > > >>> Lianet M. <li...@gmail.com> 於 2024年8月28日 週三 下午7:12寫道：
> >> > > > >>>
> >> > > > >>>> Hello TengYao! Thanks for taking on this issue, we've been
> >> going
> >> > > > around
> >> > > > >>> it
> >> > > > >>>> for a while.
> >> > > > >>>>
> >> > > > >>>> LM1: About the title of the KIP: "Enable ID Generation for
> >> Clients
> >> > > > over
> >> > > > >>> the
> >> > > > >>>> ConsumerGroupHeartbeat RPC". I find it confusing because it
> >> hints
> >> > > that
> >> > > > >>>> we're adding it as an alternative (which was discussed and
> >> > > discarded,
> >> > > > >> in
> >> > > > >>>> favour of really enforcing it). It's also missing the core
> >> change
> >> > > imo,
> >> > > > >>>> which is "where" the generation happens. So, maybe more to
> the
> >> > point
> >> > > > >> with
> >> > > > >>>> something along the lines of "Client-side generated ID for
> >> clients
> >> > > > over
> >> > > > >>>> ConsumerGroupHeartbeat RPC"?
> >> > > > >>>>
> >> > > > >>>> LM2: On the public interfaces section, the KIP states that
> "the
> >> > > server
> >> > > > >>> will
> >> > > > >>>> reject the request", but we should agree on the specific
> error
> >> > > type. I
> >> > > > >>>> expect it should fail with an InvalidRequestException, is
> that
> >> the
> >> > > > >>>> intention? (This was also suggested in the discussion thread
> >> > before
> >> > > > but
> >> > > > >>> is
> >> > > > >>>> not in the KIP).
> >> > > > >>>>
> >> > > > >>>> LM3. Related to my previous point, I find that to be the true
> >> > > > >>> public-facing
> >> > > > >>>> change (member ID mandatory at the protocol level), but it's
> >> only
> >> > at
> >> > > > >> the
> >> > > > >>>> end of the Public interfaces changes, kind of lost among
> >> details
> >> > of
> >> > > > how
> >> > > > >>>> we're going to do it. Should we rephrase that section with
> the
> >> > > actual
> >> > > > >>>> change first, and the hows after (ex. Bumping the version is
> >> not
> >> > the
> >> > > > >>>> public-facing change in this case, it's just the mechanism to
> >> > > properly
> >> > > > >>>> introduce our change)
> >> > > > >>>>
> >> > > > >>>> LM4. Regarding the lifetime of the UUID: the KIP states we
> will
> >> > > > "Verify
> >> > > > >>>> that the UUID remains consistent across all subsequent
> >> heartbeats
> >> > > > >> during
> >> > > > >>>> the session". What is this "session" referring to here? I
> would
> >> > > expect
> >> > > > >>> that
> >> > > > >>>> the UUID is associated to a consumer instance (generated for
> >> the
> >> > > > >> consumer
> >> > > > >>>> the first time it needs to send a HB if it doesn't have the
> >> UUID
> >> > > yet.
> >> > > > >>> From
> >> > > > >>>> there on, every time it needs to send a "first HB" again, it
> >> will
> >> > > > reuse
> >> > > > >>> its
> >> > > > >>>> UUID, is that the intention? Note that we should consider
> that
> >> the
> >> > > > same
> >> > > > >>>> consumer instance may have many "first heartbeats", meaning
> >> > > heartbeats
> >> > > > >> to
> >> > > > >>>> join the group when it's not part of it (ex. consumer
> >> unsubscribe
> >> > +
> >> > > > >>>> subscribe, fenced, stale). Is this the intention or are you
> >> > > > considering
> >> > > > >>> the
> >> > > > >>>> lifetime differently? We should clarify it in the KIP.
> >> > > > >>>>
> >> > > > >>>> Thanks!
> >> > > > >>>>
> >> > > > >>>> Lianet
> >> > > > >>>>
> >> > > > >>>> On Tue, Aug 27, 2024 at 2:27 AM TengYao Chi <
> >> kitingiao@gmail.com>
> >> > > > >> wrote:
> >> > > > >>>>
> >> > > > >>>>> Hi everyone,
> >> > > > >>>>>
> >> > > > >>>>> I have revised this KIP multiple times based on the feedback
> >> from
> >> > > our
> >> > > > >>>>> discussions.
> >> > > > >>>>> I would greatly appreciate it if you could review it when
> you
> >> > have
> >> > > > >> the
> >> > > > >>>>> time.
> >> > > > >>>>> If there are no further comments or suggestions, I plan to
> >> > proceed
> >> > > > >> with
> >> > > > >>>>> initiating a vote soon.
> >> > > > >>>>>
> >> > > > >>>>> Best regards,
> >> > > > >>>>> TengYao
> >> > > > >>>>>
> >> > > > >>>>> TengYao Chi <ki...@gmail.com> 於 2024年8月23日 週五 下午2:43寫道：
> >> > > > >>>>>
> >> > > > >>>>>> Hi Andrew,
> >> > > > >>>>>> Thank you for your previous feedback and insights.
> >> > > > >>>>>> Your contributions have added valuable perspectives to the
> >> > > > >>> discussions.
> >> > > > >>>>>> And we also benefit from the comparison of different
> >> solutions.
> >> > > > >>>>>> I’m also looking forward to seeing an initial version in
> >> > KIP-932,
> >> > > > >> as
> >> > > > >>> it
> >> > > > >>>>>> will provide a good reference for future implementations.
> >> > > > >>>>>>
> >> > > > >>>>>> Regarding your comment on AS2, I wanted to clarify that my
> >> > > > >>>> specification
> >> > > > >>>>>> references org.apache.kafka.common.Uuid.
> >> > > > >>>>>> I believe we’re referring to the same class, and it might
> >> just
> >> > be
> >> > > a
> >> > > > >>>> small
> >> > > > >>>>>> oversight due to the busy schedule.
> >> > > > >>>>>>
> >> > > > >>>>>> I want to express my gratitude once again for your many
> >> > insightful
> >> > > > >>>>>> comments, which have helped the discussion progress
> smoothly.
> >> > > > >>>>>>
> >> > > > >>>>>> Best regards,
> >> > > > >>>>>> TengYao
> >> > > > >>>>>>
> >> > > > >>>>>>
> >> > > > >>>>>> Andrew Schofield <an...@live.com> 於 2024年8月22日
> 週四
> >> > > > >>>> 下午11:28寫道：
> >> > > > >>>>>>
> >> > > > >>>>>>> Hi TengYao,
> >> > > > >>>>>>> I’ve been reading through the comments and I’m happy that
> >> the
> >> > > > >> lobby
> >> > > > >>>>>>> approach has not gained support.
> >> > > > >>>>>>>
> >> > > > >>>>>>> Assuming that this KIP is voted, I will be happy to change
> >> > > KIP-932
> >> > > > >>> so
> >> > > > >>>>>>> that it only supports client-side member ID generation.
> >> Because
> >> > > > >> that
> >> > > > >>>> KIP
> >> > > > >>>>>>> is still
> >> > > > >>>>>>> under development, I can do this in the first version of
> >> > > > >>>>>>> ShareGroupHeartbeat.
> >> > > > >>>>>>>
> >> > > > >>>>>>> AS2: For the encoding section, I suppose the specific
> >> encoding
> >> > > > >> which
> >> > > > >>>>>>> is used is what org.apache.kafka.utils.Uuid uses.
> >> > > > >>>>>>>
> >> > > > >>>>>>> Thanks,
> >> > > > >>>>>>> Andrew
> >> > > > >>>>>>>
> >> > > > >>>>>>>> On 14 Aug 2024, at 17:00, TengYao Chi <
> kitingiao@gmail.com
> >> >
> >> > > > >>> wrote:
> >> > > > >>>>>>>>
> >> > > > >>>>>>>> Hello Apoorv,
> >> > > > >>>>>>>> Thank you for your feedback.
> >> > > > >>>>>>>> Regarding the questions you raised, unfortunately, this
> KIP
> >> > > > >> cannot
> >> > > > >>>>>>>> guarantee the order of heartbeats. As with many classic
> >> > > > >>> distributed
> >> > > > >>>>>>> system
> >> > > > >>>>>>>> challenges, what we can do is make our best effort to
> >> ensure
> >> > > > >> that
> >> > > > >>>>> there
> >> > > > >>>>>>> are
> >> > > > >>>>>>>> no idle members or stale assignments under normal
> >> > circumstances.
> >> > > > >>>>>>>>
> >> > > > >>>>>>>> As for the lobby approach, I’m not a fan of it because it
> >> > > > >> requires
> >> > > > >>>>>>> adding a
> >> > > > >>>>>>>> mechanism to maintain client state within the
> >> ConsumerGroup,
> >> > > > >>> which,
> >> > > > >>>> in
> >> > > > >>>>>>> my
> >> > > > >>>>>>>> view, resembles something like a two-phase commit. This
> >> would
> >> > > > >>>>> introduce
> >> > > > >>>>>>>> more complexity than the proposal in this KIP, which is
> >> > > > >> something
> >> > > > >>> we
> >> > > > >>>>>>> want
> >> > > > >>>>>>>> to avoid. KIP-848 aims to simplify the existing protocol,
> >> and
> >> > > > >>> while
> >> > > > >>>>> the
> >> > > > >>>>>>>> lobby approach is a good one, I believe it is not the
> right
> >> > fit
> >> > > > >>> for
> >> > > > >>>>> this
> >> > > > >>>>>>>> particular situation.
> >> > > > >>>>>>>>
> >> > > > >>>>>>>> Best regards,
> >> > > > >>>>>>>> TengYao
> >> > > > >>>>>>>>
> >> > > > >>>>>>>> TengYao Chi <ki...@gmail.com> 於 2024年8月14日 週三
> >> 下午11:45寫道：
> >> > > > >>>>>>>>
> >> > > > >>>>>>>>> Hi David,
> >> > > > >>>>>>>>>
> >> > > > >>>>>>>>> I really appreciate your review and suggestions. As I am
> >> > still
> >> > > > >>>>> gaining
> >> > > > >>>>>>>>> experience in writing KIPs, your input has been
> incredibly
> >> > > > >>>> helpful. I
> >> > > > >>>>>>> am
> >> > > > >>>>>>>>> currently applying your suggestions to the KIP and will
> >> > > > >> complete
> >> > > > >>> it
> >> > > > >>>>> as
> >> > > > >>>>>>> soon
> >> > > > >>>>>>>>> as possible.
> >> > > > >>>>>>>>> Regarding the UUID part, I think we haven’t reached a
> >> > > > >> conclusion
> >> > > > >>>>>>> yet.(So
> >> > > > >>>>>>>>> far according to this thread)
> >> > > > >>>>>>>>> However, I will review the current implementation in the
> >> > Kafka
> >> > > > >>>> `Uuid`
> >> > > > >>>>>>>>> class and include a brief specification in the KIP.
> >> > > > >>>>>>>>>
> >> > > > >>>>>>>>> Once again, thank you so much for your help.
> >> > > > >>>>>>>>>
> >> > > > >>>>>>>>> Best regards,
> >> > > > >>>>>>>>> TengYao
> >> > > > >>>>>>>>>
> >> > > > >>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月14日 週三
> >> > 下午11:14寫道：
> >> > > > >>>>>>>>>
> >> > > > >>>>>>>>>> hi Apoorv
> >> > > > >>>>>>>>>>
> >> > > > >>>>>>>>>>> As the memberId is now known to the client, and client
> >> > might
> >> > > > >>> send
> >> > > > >>>>> the
> >> > > > >>>>>>>>>> leave
> >> > > > >>>>>>>>>> group heartbeat on shutdown prior to receiving the
> >> initial
> >> > > > >>>> heartbeat
> >> > > > >>>>>>>>>> response. If that's true then how do we guarantee that
> >> the 2
> >> > > > >>>>> requests
> >> > > > >>>>>>> to
> >> > > > >>>>>>>>>> join and leave will be processed in order, which could
> >> still
> >> > > > >>> leave
> >> > > > >>>>>>> stale
> >> > > > >>>>>>>>>> members or throw unknown member id exceptions?
> >> > > > >>>>>>>>>>
> >> > > > >>>>>>>>>> This is definitely a good question. the short answer:
> no
> >> > > > >>> guarantee
> >> > > > >>>>> but
> >> > > > >>>>>>>>>> best
> >> > > > >>>>>>>>>> efforts
> >> > > > >>>>>>>>>>
> >> > > > >>>>>>>>>> Please notice the root cause is "we have no enough time
> >> to
> >> > > > >> wait
> >> > > > >>>>>>> member id
> >> > > > >>>>>>>>>> (response) when closing consumer". Sadly, we can'
> >> guarantee
> >> > > > >> the
> >> > > > >>>>>>> request
> >> > > > >>>>>>>>>> order due to the same reason.
> >> > > > >>>>>>>>>>
> >> > > > >>>>>>>>>> However, in contrast to previous behavior, there is one
> >> big
> >> > > > >>>> benefit
> >> > > > >>>>>>> of new
> >> > > > >>>>>>>>>> approach - we can try STONITH because we know the
> member
> >> id
> >> > > > >>>>>>>>>>
> >> > > > >>>>>>>>>> Best,
> >> > > > >>>>>>>>>> Chia-Ping
> >> > > > >>>>>>>>>>
> >> > > > >>>>>>>>>>
> >> > > > >>>>>>>>>> Apoorv Mittal <ap...@gmail.com> 於 2024年8月14日
> 週三
> >> > > > >>>> 下午8:55寫道：
> >> > > > >>>>>>>>>>
> >> > > > >>>>>>>>>>> Hi TengYao,
> >> > > > >>>>>>>>>>> Thanks for the KIP. Continuing on the point which
> Andrew
> >> > > > >>>> mentioned
> >> > > > >>>>> as
> >> > > > >>>>>>>>>> AS1.
> >> > > > >>>>>>>>>>>
> >> > > > >>>>>>>>>>> As the memberId is now known to the client, and client
> >> > might
> >> > > > >>> send
> >> > > > >>>>> the
> >> > > > >>>>>>>>>> leave
> >> > > > >>>>>>>>>>> group heartbeat on shutdown prior to receiving the
> >> initial
> >> > > > >>>>> heartbeat
> >> > > > >>>>>>>>>>> response. If that's true then how do we guarantee that
> >> the
> >> > 2
> >> > > > >>>>>>> requests to
> >> > > > >>>>>>>>>>> join and leave will be processed in order, which could
> >> > still
> >> > > > >>>> leave
> >> > > > >>>>>>> stale
> >> > > > >>>>>>>>>>> members or throw unknown member id exceptions?
> >> > > > >>>>>>>>>>>
> >> > > > >>>>>>>>>>> Though the client side member id generation is helpful
> >> > which
> >> > > > >>> will
> >> > > > >>>>>>>>>> represent
> >> > > > >>>>>>>>>>> the same group perspective as from client and broker's
> >> end.
> >> > > > >>> But I
> >> > > > >>>>>>> think
> >> > > > >>>>>>>>>> the
> >> > > > >>>>>>>>>>> major concern we want to solve here is Stale Partition
> >> > > > >>>> Assignments
> >> > > > >>>>>>> which
> >> > > > >>>>>>>>>>> might still exist with the new approach. I am leaning
> >> > towards
> >> > > > >>> the
> >> > > > >>>>>>>>>>> suggestion mentioned by Andrew where partition
> >> assignment
> >> > > > >>>> triggers
> >> > > > >>>>> on
> >> > > > >>>>>>>>>>> subsequent heartbeat when client acknowledges the
> >> initial
> >> > > > >>>>> heartbeat,
> >> > > > >>>>>>>>>>> delayed partition assignment.
> >> > > > >>>>>>>>>>>
> >> > > > >>>>>>>>>>> Though on a separate note, I have a different
> question.
> >> > What
> >> > > > >>>>> happens
> >> > > > >>>>>>>>>> when
> >> > > > >>>>>>>>>>> there is an issue with the client which sends the
> >> initial
> >> > > > >>>> heartbeat
> >> > > > >>>>>>>>>> without
> >> > > > >>>>>>>>>>> memberId, then crashes and restarts? I think we must
> be
> >> > > > >>>>> re-triggering
> >> > > > >>>>>>>>>>> assignments and expiring members only after the
> >> heartbeat
> >> > > > >>> session
> >> > > > >>>>>>>>>> timeout?
> >> > > > >>>>>>>>>>> If that's true then shall delayed partition assignment
> >> can
> >> > > > >> help
> >> > > > >>>>>>> benefit
> >> > > > >>>>>>>>>> us
> >> > > > >>>>>>>>>>> from this situation as well?
> >> > > > >>>>>>>>>>>
> >> > > > >>>>>>>>>>> Regards,
> >> > > > >>>>>>>>>>> Apoorv Mittal
> >> > > > >>>>>>>>>>>
> >> > > > >>>>>>>>>>>
> >> > > > >>>>>>>>>>> On Wed, Aug 14, 2024 at 12:51 PM David Jacot
> >> > > > >>>>>>>>>> <dj...@confluent.io.invalid>
> >> > > > >>>>>>>>>>> wrote:
> >> > > > >>>>>>>>>>>
> >> > > > >>>>>>>>>>>> Hi Andrew,
> >> > > > >>>>>>>>>>>>
> >> > > > >>>>>>>>>>>> Personally, I don't like the lobby approach. It makes
> >> > things
> >> > > > >>>> more
> >> > > > >>>>>>>>>>>> complicated and it would require changing the records
> >> on
> >> > the
> >> > > > >>>>> server
> >> > > > >>>>>>>>>> too.
> >> > > > >>>>>>>>>>>> This is why I initially suggested the rejected
> >> alternative
> >> > > > >> #2
> >> > > > >>>>> which
> >> > > > >>>>>>> is
> >> > > > >>>>>>>>>>>> pretty close but also not perfect.
> >> > > > >>>>>>>>>>>>
> >> > > > >>>>>>>>>>>> I'd like to clarify one thing. The
> >> ConsumerGroupHeartbeat
> >> > > > >> API
> >> > > > >>>>>>> already
> >> > > > >>>>>>>>>>>> supports generating the member id on the client so we
> >> > don't
> >> > > > >>> need
> >> > > > >>>>> any
> >> > > > >>>>>>>>>>>> conditional logic on the client side. This is
> actually
> >> > what
> >> > > > >> we
> >> > > > >>>>>>> wanted
> >> > > > >>>>>>>>>> to
> >> > > > >>>>>>>>>>> do
> >> > > > >>>>>>>>>>>> in the first place but the idea got pushed back by
> >> Magnus
> >> > > > >> back
> >> > > > >>>>> then
> >> > > > >>>>>>>>>>> because
> >> > > > >>>>>>>>>>>> generating uuid from librdkafka required a new
> >> dependency.
> >> > > > >> It
> >> > > > >>>>> turns
> >> > > > >>>>>>>>>> out
> >> > > > >>>>>>>>>>>> that librdkafka has that dependency today. In
> >> retrospect,
> >> > we
> >> > > > >>>>> should
> >> > > > >>>>>>>>>> have
> >> > > > >>>>>>>>>>>> pushed back on this. Long story short, we can just do
> >> it.
> >> > > > >> The
> >> > > > >>>>>>>>>> proposal in
> >> > > > >>>>>>>>>>>> this KIP is to make the member id required in future
> >> > > > >> versions.
> >> > > > >>>> We
> >> > > > >>>>>>>>>> could
> >> > > > >>>>>>>>>>>> also decide not to do it and to keep supporting both
> >> > > > >>>> approaches. I
> >> > > > >>>>>>>>>> would
> >> > > > >>>>>>>>>>>> also be fine with this.
> >> > > > >>>>>>>>>>>>
> >> > > > >>>>>>>>>>>> Best,
> >> > > > >>>>>>>>>>>> David
> >> > > > >>>>>>>>>>>>
> >> > > > >>>>>>>>>>>> On Wed, Aug 14, 2024 at 12:30 PM Andrew Schofield <
> >> > > > >>>>>>>>>>>> andrew_schofield@live.com>
> >> > > > >>>>>>>>>>>> wrote:
> >> > > > >>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>> Hi TengYao,
> >> > > > >>>>>>>>>>>>> Thanks for your response. I’ll have just one more
> try
> >> to
> >> > > > >>>>> persuade.
> >> > > > >>>>>>>>>>>>> I feel that I will need to follow the approach with
> >> > KIP-932
> >> > > > >>>> when
> >> > > > >>>>>>>>>> we’ve
> >> > > > >>>>>>>>>>>>> made a decision, so I do have more than a passing
> >> > interest
> >> > > > >> in
> >> > > > >>>>> this.
> >> > > > >>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>> A group member in the lobby is in the group, but it
> >> does
> >> > > > >> not
> >> > > > >>>> have
> >> > > > >>>>>>>>>> any
> >> > > > >>>>>>>>>>>>> assignments. A member of a consumer group can have
> no
> >> > > > >>> assigned
> >> > > > >>>>>>>>>>>>> partitions (such as 5 CG members subscribed to a
> topic
> >> > > > >> with 4
> >> > > > >>>>>>>>>>>> partitions),
> >> > > > >>>>>>>>>>>>> so it’s a situation that consumer group members
> >> already
> >> > > > >>> expect.
> >> > > > >>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>> One of Kafka’s strengths is the way that we handle
> API
> >> > > > >>>>> versioning.
> >> > > > >>>>>>>>>>>>> But, there is a cost - the behaviour is different
> >> > depending
> >> > > > >>> on
> >> > > > >>>>> the
> >> > > > >>>>>>>>>> RPC
> >> > > > >>>>>>>>>>>>> version. KIP-848 is on the cusp of completion, but
> >> we’re
> >> > > > >>>> already
> >> > > > >>>>>>>>>> adding
> >> > > > >>>>>>>>>>>>> conditional logic for v0/v1 for
> >> ConsumerGroupHeartbeat.
> >> > > > >>> That’s
> >> > > > >>>> a
> >> > > > >>>>>>>>>> pity.
> >> > > > >>>>>>>>>>>>> Only a minor issue, but it’s unfortunate.
> >> > > > >>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>> Thanks,
> >> > > > >>>>>>>>>>>>> Andrew
> >> > > > >>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>> On 14 Aug 2024, at 08:47, TengYao Chi <
> >> > > > >> kitingiao@gmail.com>
> >> > > > >>>>>>>>>> wrote:
> >> > > > >>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>> Hello Andrew
> >> > > > >>>>>>>>>>>>>> Thank you for your thoughtful suggestions and
> getting
> >> > the
> >> > > > >>>>>>>>>> discussion
> >> > > > >>>>>>>>>>>>> going.
> >> > > > >>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>> To AS1:
> >> > > > >>>>>>>>>>>>>> In the current scenario where the server generates
> >> the
> >> > > > >> UUID,
> >> > > > >>>> if
> >> > > > >>>>>>>>>> the
> >> > > > >>>>>>>>>>>>> client
> >> > > > >>>>>>>>>>>>>> shuts down before receiving the memberId generated
> by
> >> > the
> >> > > > >> GC
> >> > > > >>>>>>>>>>>> (regardless
> >> > > > >>>>>>>>>>>>> of
> >> > > > >>>>>>>>>>>>>> whether it’s a graceful shutdown or not), the GC
> will
> >> > > > >> still
> >> > > > >>>> have
> >> > > > >>>>>>>>>> to
> >> > > > >>>>>>>>>>>> wait
> >> > > > >>>>>>>>>>>>>> for the heartbeat timeout because the client
> doesn’t
> >> > know
> >> > > > >>> its
> >> > > > >>>>>>>>>>> memberId.
> >> > > > >>>>>>>>>>>>>> This KIP indeed cannot completely resolve the
> >> > idempotency
> >> > > > >>>> issue,
> >> > > > >>>>>>>>>> but
> >> > > > >>>>>>>>>>> it
> >> > > > >>>>>>>>>>>>> can
> >> > > > >>>>>>>>>>>>>> better handle shutdown scenarios under normal
> >> > > > >> circumstances
> >> > > > >>>>>>>>>> because
> >> > > > >>>>>>>>>>> the
> >> > > > >>>>>>>>>>>>>> client always knows its memberId. Even if the
> client
> >> > shuts
> >> > > > >>>> down
> >> > > > >>>>>>>>>>>>> immediately
> >> > > > >>>>>>>>>>>>>> after the initial heartbeat, as long as it
> performs a
> >> > > > >>> graceful
> >> > > > >>>>>>>>>>> shutdown
> >> > > > >>>>>>>>>>>>> and
> >> > > > >>>>>>>>>>>>>> sends a leave heartbeat, the GC can manage the
> >> situation
> >> > > > >> and
> >> > > > >>>>>>>>>> remove
> >> > > > >>>>>>>>>>> the
> >> > > > >>>>>>>>>>>>>> member. Therefore, the goal of this KIP is to
> address
> >> > the
> >> > > > >>>> issue
> >> > > > >>>>>>>>>> where
> >> > > > >>>>>>>>>>>> the
> >> > > > >>>>>>>>>>>>>> GC has to wait for the heartbeat timeout due to the
> >> > client
> >> > > > >>>>> leaving
> >> > > > >>>>>>>>>>>>> without
> >> > > > >>>>>>>>>>>>>> knowing its memberId, which leads to reduced
> >> throughput
> >> > > > >> and
> >> > > > >>>>>>>>>> limited
> >> > > > >>>>>>>>>>>>>> scalability.
> >> > > > >>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>> The solution you suggest has also been proposed by
> >> > David.
> >> > > > >>> The
> >> > > > >>>>>>>>>> concern
> >> > > > >>>>>>>>>>>>> with
> >> > > > >>>>>>>>>>>>>> this approach is that it introduces additional
> >> > complexity
> >> > > > >>> for
> >> > > > >>>>>>>>>>>>>> compatibility, as the new server would not
> >> immediately
> >> > add
> >> > > > >>> the
> >> > > > >>>>>>>>>> member
> >> > > > >>>>>>>>>>>> to
> >> > > > >>>>>>>>>>>>>> the group, while the old server would. This
> requires
> >> > > > >> clients
> >> > > > >>>> to
> >> > > > >>>>>>>>>>>>>> differentiate whether their memberId has been added
> >> to
> >> > the
> >> > > > >>>> group
> >> > > > >>>>>>>>>> or
> >> > > > >>>>>>>>>>>> not,
> >> > > > >>>>>>>>>>>>>> which could result in unexpected logs.
> >> > > > >>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>> Best Regards,
> >> > > > >>>>>>>>>>>>>> TengYao
> >> > > > >>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>> Andrew Schofield <an...@live.com> 於
> >> > 2024年8月14日
> >> > > > >>> 週三
> >> > > > >>>>>>>>>>>> 上午12:29寫道：
> >> > > > >>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>> Hi TengYao,
> >> > > > >>>>>>>>>>>>>>> Thanks for the KIP. I wonder if there’s a
> different
> >> way
> >> > > > >> to
> >> > > > >>>>> close
> >> > > > >>>>>>>>>>> what
> >> > > > >>>>>>>>>>>>>>> is quite a small window.
> >> > > > >>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>> AS1: It is true that the initial heartbeat is not
> >> > > > >>> idempotent,
> >> > > > >>>>> but
> >> > > > >>>>>>>>>>> this
> >> > > > >>>>>>>>>>>>>>> remains
> >> > > > >>>>>>>>>>>>>>> true with this KIP. It’s just differently not
> >> > idempotent.
> >> > > > >>> If
> >> > > > >>>>> the
> >> > > > >>>>>>>>>>>> client
> >> > > > >>>>>>>>>>>>>>> makes its
> >> > > > >>>>>>>>>>>>>>> own member ID, sends a request and dies, the GC
> will
> >> > > > >> still
> >> > > > >>>> have
> >> > > > >>>>>>>>>>> added
> >> > > > >>>>>>>>>>>>>>> the member to the group and it will hang around
> >> until
> >> > the
> >> > > > >>>>> session
> >> > > > >>>>>>>>>>>>> expires.
> >> > > > >>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>> I wonder if the GC could still generate the member
> >> ID
> >> > in
> >> > > > >>>>>>>>>> response to
> >> > > > >>>>>>>>>>>> the
> >> > > > >>>>>>>>>>>>>>> first
> >> > > > >>>>>>>>>>>>>>> heartbeat, and put the member in a special PENDING
> >> > state
> >> > > > >>> with
> >> > > > >>>>> no
> >> > > > >>>>>>>>>>>>>>> assignments until the client sends the next
> >> heartbeat,
> >> > > > >> thus
> >> > > > >>>>>>>>>>> confirming
> >> > > > >>>>>>>>>>>>> it
> >> > > > >>>>>>>>>>>>>>> has received the member ID. This would not be a
> >> > protocol
> >> > > > >>>> change
> >> > > > >>>>>>>>>> at
> >> > > > >>>>>>>>>>>> all,
> >> > > > >>>>>>>>>>>>>>> just
> >> > > > >>>>>>>>>>>>>>> a change to the GC to keep a new member in the
> lobby
> >> > > > >> until
> >> > > > >>>> it’s
> >> > > > >>>>>>>>>>>>> comfirmed
> >> > > > >>>>>>>>>>>>>>> it knows its member ID.
> >> > > > >>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>> Thanks,
> >> > > > >>>>>>>>>>>>>>> Andrew
> >> > > > >>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>> On 13 Aug 2024, at 15:59, TengYao Chi <
> >> > > > >>> kitingiao@gmail.com>
> >> > > > >>>>>>>>>> wrote:
> >> > > > >>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>> Hi Chia-Ping,
> >> > > > >>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>> Thanks for review and suggestions.
> >> > > > >>>>>>>>>>>>>>>> I have updated the content of KIP accordingly.
> >> > > > >>>>>>>>>>>>>>>> Please take a look.
> >> > > > >>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>> Best regards,
> >> > > > >>>>>>>>>>>>>>>> TengYao
> >> > > > >>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> 2024年8月13日
> >> 週二
> >> > > > >>>>> 下午9:45寫道：
> >> > > > >>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>> hi TengYao
> >> > > > >>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>> thanks for this KIP.
> >> > > > >>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>> 1) could you please describe the before/after
> >> > behavior
> >> > > > >> in
> >> > > > >>>> the
> >> > > > >>>>>>>>>>>>> "Proposed
> >> > > > >>>>>>>>>>>>>>>>> Changes" section? IIRC, current RPC allows HB
> >> having
> >> > > > >>> member
> >> > > > >>>>> id
> >> > > > >>>>>>>>>>>>>>> generated by
> >> > > > >>>>>>>>>>>>>>>>> client, right? If HB has no member ID, server
> will
> >> > > > >>> generate
> >> > > > >>>>> one
> >> > > > >>>>>>>>>>> and
> >> > > > >>>>>>>>>>>>> then
> >> > > > >>>>>>>>>>>>>>>>> return. The new behavior will enforce HB "must"
> >> have
> >> > > > >>> member
> >> > > > >>>>> ID.
> >> > > > >>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>> 2) could you please write the version number
> >> > explicitly
> >> > > > >>> in
> >> > > > >>>>> the
> >> > > > >>>>>>>>>> KIP
> >> > > > >>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>> 3) how new client code handle the old HB? Does
> it
> >> > > > >> always
> >> > > > >>>>>>>>>> generate
> >> > > > >>>>>>>>>>>>> member
> >> > > > >>>>>>>>>>>>>>>>> ID on client-side even though that is not
> >> restricted?
> >> > > > >>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>> Best,
> >> > > > >>>>>>>>>>>>>>>>> Chia-Ping
> >> > > > >>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>> On 2024/08/13 06:20:42 TengYao Chi wrote:
> >> > > > >>>>>>>>>>>>>>>>>> Hello everyone,
> >> > > > >>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>> I would like to start a discussion thread on
> >> > KIP-1082,
> >> > > > >>>> which
> >> > > > >>>>>>>>>>>> proposes
> >> > > > >>>>>>>>>>>>>>>>>> enabling id generation for clients over the
> >> > > > >>>>>>>>>>> ConsumerGroupHeartbeat
> >> > > > >>>>>>>>>>>>> RPC.
> >> > > > >>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>> Here is the KIP Link: KIP-1082
> >> > > > >>>>>>>>>>>>>>>>>> <
> >> > > > >>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>
> >> > > > >>>>>>>>>>>
> >> > > > >>>>>>>>>>
> >> > > > >>>>>>>
> >> > > > >>>>>
> >> > > > >>>>
> >> > > > >>>
> >> > > > >>
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
> >> > > > >>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>> Please take a look and let me know what you
> >> think,
> >> > > > >> and I
> >> > > > >>>>> would
> >> > > > >>>>>>>>>>>>>>> appreciate
> >> > > > >>>>>>>>>>>>>>>>>> any suggestions and feedback.
> >> > > > >>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>> Best regards,
> >> > > > >>>>>>>>>>>>>>>>>> TengYao
> >> > > > >>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>
> >> > > > >>>>>>>>>>>
> >> > > > >>>>>>>>>>
> >> > > > >>>>>>>>>
> >> > > > >>>>>>>
> >> > > > >>>>>>>
> >> > > > >>>>>
> >> > > > >>>>
> >> > > > >>>
> >> > > > >>
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[TengYao Chi <ki...@gmail.com>]

Hello everyone,

Since this KIP has been fully discussed, I will initiate a vote for it next
Monday.
Thank you and have a nice weekend.

Best regards,
TengYao

TengYao Chi <ki...@gmail.com> 於 2024年9月5日 週四 下午2:19寫道：

> Hello everyone,
>
> KT2: It looks like everyone who has expressed an opinion supports the
> second option: “Document a recommendation for clients to use UUIDs as
> member IDs, without strictly enforcing it.”
> I have updated the KIP accordingly.
> Please take a look, and let me know if you have any thoughts or feedback.
>
> Thank you!
>
> Best regards,
> TengYao
>
> Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月30日 週五 下午9:56寫道：
>
>> hi TengYao
>>
>> KT2: +1 to second approach
>>
>> Best,
>> Chia-Ping
>>
>>
>> David Jacot <dj...@confluent.io.invalid> 於 2024年8月30日 週五 下午9:15寫道：
>>
>> > Hi TengYao,
>> >
>> > KT2: I don't think that we can realistically validate the uuid on the
>> > server. It is basically a string of chars. So I lean towards having a
>> good
>> > recommendation in the KIP and in the document of the field in the RPC's
>> > definition.
>> >
>> > Best,
>> > David
>> >
>> > On Fri, Aug 30, 2024 at 3:02 PM TengYao Chi <ki...@gmail.com>
>> wrote:
>> >
>> > > Hello Kirk !
>> > >
>> > > Thank you for your comments !
>> > >
>> > > KT1: Yes, you are correct. The issue is not unique to the initial
>> > > heartbeat; there can always be cases where the broker might lose
>> > connection
>> > > with a member.
>> > >
>> > > KT2: Currently, if the client doesn't have a member ID and the
>> > memberEpoch
>> > > equals 0, the coordinator will generate a UUID as the member ID for
>> the
>> > > client. However, at the RPC level, the member ID is sent as a literal
>> > > string, meaning there are no restrictions on the format at this level.
>> > > This also reminds me that we haven't reached a final conclusion on
>> how to
>> > > enforce the use of UUIDs.
>> > > From our previous discussions, I recall two possible approaches:
>> > > The first is to validate the UUID on the server side, and if it's not
>> > > valid, throw an exception to the client.
>> > > The second is to document a recommendation for clients to use UUIDs as
>> > > member IDs, without strictly enforcing it.
>> > > I think it's time to decide on the approach we want to take.
>> > >
>> > > KT3: Yes, "session" can be considered synonymous with "membership" in
>> > this
>> > > context.
>> > >
>> > > KT4: Thank you for pointing that out. I will update the wording to
>> > > specifically say this behavior is for consumers.
>> > >
>> > > Thanks again for your comments.
>> > >
>> > > Best regards,
>> > > TengYao
>> > >
>> > > Kirk True <ki...@kirktrue.pro> 於 2024年8月30日 週五 上午12:39寫道：
>> > >
>> > > > Hi TengYao!
>> > > >
>> > > > Sorry for being late to the discussion...
>> > > >
>> > > > After reading the thread and then the KIP, I had a few
>> > > questions/comments:
>> > > >
>> > > > KT1: In Motivation, it states: "This scenario can result in the
>> broker
>> > > > registering a new member for which it will never receive a proper
>> leave
>> > > > request.” Just to be clear, the broker will always have cases where
>> it
>> > > > might lose connection with a member. That’s not unique to the
>> initial
>> > > > heartbeat, right?
>> > > >
>> > > > KT2: There was a bit of back and forth about format of the member
>> ID.
>> > > From
>> > > > what I gathered in the thread, the member ID is still defined in the
>> > RPC
>> > > as
>> > > > a string and not a UUID, right? The KIP states that the “client must
>> > > > generate a UUID as the member ID” and that the “server will validate
>> > > that a
>> > > > valid UUID is provided.” Is that a change for the server, or is it
>> > > already
>> > > > enforced as a UUID?
>> > > >
>> > > > KT3: Lianet mentioned some confusion over the use of the word
>> > “session.”
>> > > > Isn’t “session” synonymous with “membership?”
>> > > >
>> > > > KT4: Under “Member ID Lifecycle,” it states: "The client should
>> reuse
>> > the
>> > > > same UUID as the member ID for all heartbeats and rejoin attempts to
>> > > > maintain continuity within the group.” Could we change the first
>> part
>> > of
>> > > > that to “The Consumer instance should…” We do have lifetimes that
>> > extend
>> > > > past the lifetime of a client instance (such as the transaction ID).
>> > > >
>> > > > Thanks,
>> > > > Kirk
>> > > >
>> > > > > On Aug 29, 2024, at 1:28 AM, TengYao Chi <ki...@gmail.com>
>> > wrote:
>> > > > >
>> > > > > Hi David,
>> > > > >
>> > > > > Thank you for pointing that out.
>> > > > > I have updated the content of the KIP based on Lianet's and your
>> > > > feedback.
>> > > > > Please take a look and let me know your thoughts.
>> > > > >
>> > > > > Best regards,
>> > > > > TengYao
>> > > > >
>> > > > > David Jacot <dj...@confluent.io.invalid> 於 2024年8月29日 週四
>> 下午3:20寫道：
>> > > > >
>> > > > >> Hi TengYao,
>> > > > >>
>> > > > >> Thanks for the update. I haven't fully read it yet but I will
>> soon.
>> > > > >>
>> > > > >> LM4: This is incorrect. The consumer must keep its member id
>> during
>> > > its
>> > > > >> entire lifetime (until the process stops or dies). The protocol
>> > > > stipulates
>> > > > >> that a member must rejoin with the same member id and the member
>> > epoch
>> > > > set
>> > > > >> to zero when an FENCED_MEMBER_EPOCH occurs. This allows the
>> member
>> > to
>> > > > >> resynchronize itself. We should not change this behavior. I think
>> > that
>> > > > we
>> > > > >> should see the client side generation id as an incarnation id of
>> the
>> > > > >> application. It is generated once and kept until it stops or
>> dies.
>> > > > >>
>> > > > >> Best,
>> > > > >> David
>> > > > >>
>> > > > >> On Thu, Aug 29, 2024 at 6:21 AM TengYao Chi <kitingiao@gmail.com
>> >
>> > > > wrote:
>> > > > >>
>> > > > >>> Hello Lianet !
>> > > > >>>
>> > > > >>> Thanks for the reviews and suggestions!
>> > > > >>>
>> > > > >>> LM1: Indeed, we plan to enforce client-side ID generation in the
>> > > > future,
>> > > > >>> and it is not an alternative. I will change the title
>> accordingly.
>> > > > >>>
>> > > > >>> LM2: Yes, that's the expectation. I will add that statement to
>> the
>> > > > public
>> > > > >>> interface section.
>> > > > >>>
>> > > > >>> LM3: Thank you for the high-level perspective review. I think
>> > you're
>> > > > >> right;
>> > > > >>> our intention isn't very clear since it was placed at the end of
>> > the
>> > > > >>> section. I will try to rephrase that section to make it more
>> > obvious.
>> > > > >>>
>> > > > >>> LM4: Regarding the definition of "session" in this KIP, I
>> believe
>> > it
>> > > > >> refers
>> > > > >>> to the period between the *first-time heartbeat* and when the
>> > > *consumer
>> > > > >>> leaves the group* (whether through a graceful shutdown or a
>> > heartbeat
>> > > > >>> timeout). The consumer should reuse its UUID if it has been
>> > generated
>> > > > >>> before. The only situation in which it will regenerate the UUID
>> is
>> > if
>> > > > the
>> > > > >>> coordinator finds that there is already a consumer with the same
>> > > UUID.
>> > > > >>> IIRC, the coordinator should compare the member epochs, and the
>> > > > >>> later-joined consumer should be fenced off by the coordinator
>> due
>> > to
>> > > > >> having
>> > > > >>> a lower member epoch. Once the consumer receives a
>> > > > `FENCED_MEMBER_EPOCH`
>> > > > >>> error, it will generate a new UUID and attempt to rejoin. I will
>> > > > clarify
>> > > > >>> this in the KIP.
>> > > > >>>
>> > > > >>> Thanks again for your reviews, I really appreciate it.
>> > > > >>>
>> > > > >>> Best regards,
>> > > > >>> TengYao
>> > > > >>>
>> > > > >>> Lianet M. <li...@gmail.com> 於 2024年8月28日 週三 下午7:12寫道：
>> > > > >>>
>> > > > >>>> Hello TengYao! Thanks for taking on this issue, we've been
>> going
>> > > > around
>> > > > >>> it
>> > > > >>>> for a while.
>> > > > >>>>
>> > > > >>>> LM1: About the title of the KIP: "Enable ID Generation for
>> Clients
>> > > > over
>> > > > >>> the
>> > > > >>>> ConsumerGroupHeartbeat RPC". I find it confusing because it
>> hints
>> > > that
>> > > > >>>> we're adding it as an alternative (which was discussed and
>> > > discarded,
>> > > > >> in
>> > > > >>>> favour of really enforcing it). It's also missing the core
>> change
>> > > imo,
>> > > > >>>> which is "where" the generation happens. So, maybe more to the
>> > point
>> > > > >> with
>> > > > >>>> something along the lines of "Client-side generated ID for
>> clients
>> > > > over
>> > > > >>>> ConsumerGroupHeartbeat RPC"?
>> > > > >>>>
>> > > > >>>> LM2: On the public interfaces section, the KIP states that "the
>> > > server
>> > > > >>> will
>> > > > >>>> reject the request", but we should agree on the specific error
>> > > type. I
>> > > > >>>> expect it should fail with an InvalidRequestException, is that
>> the
>> > > > >>>> intention? (This was also suggested in the discussion thread
>> > before
>> > > > but
>> > > > >>> is
>> > > > >>>> not in the KIP).
>> > > > >>>>
>> > > > >>>> LM3. Related to my previous point, I find that to be the true
>> > > > >>> public-facing
>> > > > >>>> change (member ID mandatory at the protocol level), but it's
>> only
>> > at
>> > > > >> the
>> > > > >>>> end of the Public interfaces changes, kind of lost among
>> details
>> > of
>> > > > how
>> > > > >>>> we're going to do it. Should we rephrase that section with the
>> > > actual
>> > > > >>>> change first, and the hows after (ex. Bumping the version is
>> not
>> > the
>> > > > >>>> public-facing change in this case, it's just the mechanism to
>> > > properly
>> > > > >>>> introduce our change)
>> > > > >>>>
>> > > > >>>> LM4. Regarding the lifetime of the UUID: the KIP states we will
>> > > > "Verify
>> > > > >>>> that the UUID remains consistent across all subsequent
>> heartbeats
>> > > > >> during
>> > > > >>>> the session". What is this "session" referring to here? I would
>> > > expect
>> > > > >>> that
>> > > > >>>> the UUID is associated to a consumer instance (generated for
>> the
>> > > > >> consumer
>> > > > >>>> the first time it needs to send a HB if it doesn't have the
>> UUID
>> > > yet.
>> > > > >>> From
>> > > > >>>> there on, every time it needs to send a "first HB" again, it
>> will
>> > > > reuse
>> > > > >>> its
>> > > > >>>> UUID, is that the intention? Note that we should consider that
>> the
>> > > > same
>> > > > >>>> consumer instance may have many "first heartbeats", meaning
>> > > heartbeats
>> > > > >> to
>> > > > >>>> join the group when it's not part of it (ex. consumer
>> unsubscribe
>> > +
>> > > > >>>> subscribe, fenced, stale). Is this the intention or are you
>> > > > considering
>> > > > >>> the
>> > > > >>>> lifetime differently? We should clarify it in the KIP.
>> > > > >>>>
>> > > > >>>> Thanks!
>> > > > >>>>
>> > > > >>>> Lianet
>> > > > >>>>
>> > > > >>>> On Tue, Aug 27, 2024 at 2:27 AM TengYao Chi <
>> kitingiao@gmail.com>
>> > > > >> wrote:
>> > > > >>>>
>> > > > >>>>> Hi everyone,
>> > > > >>>>>
>> > > > >>>>> I have revised this KIP multiple times based on the feedback
>> from
>> > > our
>> > > > >>>>> discussions.
>> > > > >>>>> I would greatly appreciate it if you could review it when you
>> > have
>> > > > >> the
>> > > > >>>>> time.
>> > > > >>>>> If there are no further comments or suggestions, I plan to
>> > proceed
>> > > > >> with
>> > > > >>>>> initiating a vote soon.
>> > > > >>>>>
>> > > > >>>>> Best regards,
>> > > > >>>>> TengYao
>> > > > >>>>>
>> > > > >>>>> TengYao Chi <ki...@gmail.com> 於 2024年8月23日 週五 下午2:43寫道：
>> > > > >>>>>
>> > > > >>>>>> Hi Andrew,
>> > > > >>>>>> Thank you for your previous feedback and insights.
>> > > > >>>>>> Your contributions have added valuable perspectives to the
>> > > > >>> discussions.
>> > > > >>>>>> And we also benefit from the comparison of different
>> solutions.
>> > > > >>>>>> I’m also looking forward to seeing an initial version in
>> > KIP-932,
>> > > > >> as
>> > > > >>> it
>> > > > >>>>>> will provide a good reference for future implementations.
>> > > > >>>>>>
>> > > > >>>>>> Regarding your comment on AS2, I wanted to clarify that my
>> > > > >>>> specification
>> > > > >>>>>> references org.apache.kafka.common.Uuid.
>> > > > >>>>>> I believe we’re referring to the same class, and it might
>> just
>> > be
>> > > a
>> > > > >>>> small
>> > > > >>>>>> oversight due to the busy schedule.
>> > > > >>>>>>
>> > > > >>>>>> I want to express my gratitude once again for your many
>> > insightful
>> > > > >>>>>> comments, which have helped the discussion progress smoothly.
>> > > > >>>>>>
>> > > > >>>>>> Best regards,
>> > > > >>>>>> TengYao
>> > > > >>>>>>
>> > > > >>>>>>
>> > > > >>>>>> Andrew Schofield <an...@live.com> 於 2024年8月22日 週四
>> > > > >>>> 下午11:28寫道：
>> > > > >>>>>>
>> > > > >>>>>>> Hi TengYao,
>> > > > >>>>>>> I’ve been reading through the comments and I’m happy that
>> the
>> > > > >> lobby
>> > > > >>>>>>> approach has not gained support.
>> > > > >>>>>>>
>> > > > >>>>>>> Assuming that this KIP is voted, I will be happy to change
>> > > KIP-932
>> > > > >>> so
>> > > > >>>>>>> that it only supports client-side member ID generation.
>> Because
>> > > > >> that
>> > > > >>>> KIP
>> > > > >>>>>>> is still
>> > > > >>>>>>> under development, I can do this in the first version of
>> > > > >>>>>>> ShareGroupHeartbeat.
>> > > > >>>>>>>
>> > > > >>>>>>> AS2: For the encoding section, I suppose the specific
>> encoding
>> > > > >> which
>> > > > >>>>>>> is used is what org.apache.kafka.utils.Uuid uses.
>> > > > >>>>>>>
>> > > > >>>>>>> Thanks,
>> > > > >>>>>>> Andrew
>> > > > >>>>>>>
>> > > > >>>>>>>> On 14 Aug 2024, at 17:00, TengYao Chi <kitingiao@gmail.com
>> >
>> > > > >>> wrote:
>> > > > >>>>>>>>
>> > > > >>>>>>>> Hello Apoorv,
>> > > > >>>>>>>> Thank you for your feedback.
>> > > > >>>>>>>> Regarding the questions you raised, unfortunately, this KIP
>> > > > >> cannot
>> > > > >>>>>>>> guarantee the order of heartbeats. As with many classic
>> > > > >>> distributed
>> > > > >>>>>>> system
>> > > > >>>>>>>> challenges, what we can do is make our best effort to
>> ensure
>> > > > >> that
>> > > > >>>>> there
>> > > > >>>>>>> are
>> > > > >>>>>>>> no idle members or stale assignments under normal
>> > circumstances.
>> > > > >>>>>>>>
>> > > > >>>>>>>> As for the lobby approach, I’m not a fan of it because it
>> > > > >> requires
>> > > > >>>>>>> adding a
>> > > > >>>>>>>> mechanism to maintain client state within the
>> ConsumerGroup,
>> > > > >>> which,
>> > > > >>>> in
>> > > > >>>>>>> my
>> > > > >>>>>>>> view, resembles something like a two-phase commit. This
>> would
>> > > > >>>>> introduce
>> > > > >>>>>>>> more complexity than the proposal in this KIP, which is
>> > > > >> something
>> > > > >>> we
>> > > > >>>>>>> want
>> > > > >>>>>>>> to avoid. KIP-848 aims to simplify the existing protocol,
>> and
>> > > > >>> while
>> > > > >>>>> the
>> > > > >>>>>>>> lobby approach is a good one, I believe it is not the right
>> > fit
>> > > > >>> for
>> > > > >>>>> this
>> > > > >>>>>>>> particular situation.
>> > > > >>>>>>>>
>> > > > >>>>>>>> Best regards,
>> > > > >>>>>>>> TengYao
>> > > > >>>>>>>>
>> > > > >>>>>>>> TengYao Chi <ki...@gmail.com> 於 2024年8月14日 週三
>> 下午11:45寫道：
>> > > > >>>>>>>>
>> > > > >>>>>>>>> Hi David,
>> > > > >>>>>>>>>
>> > > > >>>>>>>>> I really appreciate your review and suggestions. As I am
>> > still
>> > > > >>>>> gaining
>> > > > >>>>>>>>> experience in writing KIPs, your input has been incredibly
>> > > > >>>> helpful. I
>> > > > >>>>>>> am
>> > > > >>>>>>>>> currently applying your suggestions to the KIP and will
>> > > > >> complete
>> > > > >>> it
>> > > > >>>>> as
>> > > > >>>>>>> soon
>> > > > >>>>>>>>> as possible.
>> > > > >>>>>>>>> Regarding the UUID part, I think we haven’t reached a
>> > > > >> conclusion
>> > > > >>>>>>> yet.(So
>> > > > >>>>>>>>> far according to this thread)
>> > > > >>>>>>>>> However, I will review the current implementation in the
>> > Kafka
>> > > > >>>> `Uuid`
>> > > > >>>>>>>>> class and include a brief specification in the KIP.
>> > > > >>>>>>>>>
>> > > > >>>>>>>>> Once again, thank you so much for your help.
>> > > > >>>>>>>>>
>> > > > >>>>>>>>> Best regards,
>> > > > >>>>>>>>> TengYao
>> > > > >>>>>>>>>
>> > > > >>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月14日 週三
>> > 下午11:14寫道：
>> > > > >>>>>>>>>
>> > > > >>>>>>>>>> hi Apoorv
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>>> As the memberId is now known to the client, and client
>> > might
>> > > > >>> send
>> > > > >>>>> the
>> > > > >>>>>>>>>> leave
>> > > > >>>>>>>>>> group heartbeat on shutdown prior to receiving the
>> initial
>> > > > >>>> heartbeat
>> > > > >>>>>>>>>> response. If that's true then how do we guarantee that
>> the 2
>> > > > >>>>> requests
>> > > > >>>>>>> to
>> > > > >>>>>>>>>> join and leave will be processed in order, which could
>> still
>> > > > >>> leave
>> > > > >>>>>>> stale
>> > > > >>>>>>>>>> members or throw unknown member id exceptions?
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> This is definitely a good question. the short answer: no
>> > > > >>> guarantee
>> > > > >>>>> but
>> > > > >>>>>>>>>> best
>> > > > >>>>>>>>>> efforts
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> Please notice the root cause is "we have no enough time
>> to
>> > > > >> wait
>> > > > >>>>>>> member id
>> > > > >>>>>>>>>> (response) when closing consumer". Sadly, we can'
>> guarantee
>> > > > >> the
>> > > > >>>>>>> request
>> > > > >>>>>>>>>> order due to the same reason.
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> However, in contrast to previous behavior, there is one
>> big
>> > > > >>>> benefit
>> > > > >>>>>>> of new
>> > > > >>>>>>>>>> approach - we can try STONITH because we know the member
>> id
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> Best,
>> > > > >>>>>>>>>> Chia-Ping
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> Apoorv Mittal <ap...@gmail.com> 於 2024年8月14日 週三
>> > > > >>>> 下午8:55寫道：
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>>> Hi TengYao,
>> > > > >>>>>>>>>>> Thanks for the KIP. Continuing on the point which Andrew
>> > > > >>>> mentioned
>> > > > >>>>> as
>> > > > >>>>>>>>>> AS1.
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> As the memberId is now known to the client, and client
>> > might
>> > > > >>> send
>> > > > >>>>> the
>> > > > >>>>>>>>>> leave
>> > > > >>>>>>>>>>> group heartbeat on shutdown prior to receiving the
>> initial
>> > > > >>>>> heartbeat
>> > > > >>>>>>>>>>> response. If that's true then how do we guarantee that
>> the
>> > 2
>> > > > >>>>>>> requests to
>> > > > >>>>>>>>>>> join and leave will be processed in order, which could
>> > still
>> > > > >>>> leave
>> > > > >>>>>>> stale
>> > > > >>>>>>>>>>> members or throw unknown member id exceptions?
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> Though the client side member id generation is helpful
>> > which
>> > > > >>> will
>> > > > >>>>>>>>>> represent
>> > > > >>>>>>>>>>> the same group perspective as from client and broker's
>> end.
>> > > > >>> But I
>> > > > >>>>>>> think
>> > > > >>>>>>>>>> the
>> > > > >>>>>>>>>>> major concern we want to solve here is Stale Partition
>> > > > >>>> Assignments
>> > > > >>>>>>> which
>> > > > >>>>>>>>>>> might still exist with the new approach. I am leaning
>> > towards
>> > > > >>> the
>> > > > >>>>>>>>>>> suggestion mentioned by Andrew where partition
>> assignment
>> > > > >>>> triggers
>> > > > >>>>> on
>> > > > >>>>>>>>>>> subsequent heartbeat when client acknowledges the
>> initial
>> > > > >>>>> heartbeat,
>> > > > >>>>>>>>>>> delayed partition assignment.
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> Though on a separate note, I have a different question.
>> > What
>> > > > >>>>> happens
>> > > > >>>>>>>>>> when
>> > > > >>>>>>>>>>> there is an issue with the client which sends the
>> initial
>> > > > >>>> heartbeat
>> > > > >>>>>>>>>> without
>> > > > >>>>>>>>>>> memberId, then crashes and restarts? I think we must be
>> > > > >>>>> re-triggering
>> > > > >>>>>>>>>>> assignments and expiring members only after the
>> heartbeat
>> > > > >>> session
>> > > > >>>>>>>>>> timeout?
>> > > > >>>>>>>>>>> If that's true then shall delayed partition assignment
>> can
>> > > > >> help
>> > > > >>>>>>> benefit
>> > > > >>>>>>>>>> us
>> > > > >>>>>>>>>>> from this situation as well?
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> Regards,
>> > > > >>>>>>>>>>> Apoorv Mittal
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> On Wed, Aug 14, 2024 at 12:51 PM David Jacot
>> > > > >>>>>>>>>> <dj...@confluent.io.invalid>
>> > > > >>>>>>>>>>> wrote:
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>>> Hi Andrew,
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>> Personally, I don't like the lobby approach. It makes
>> > things
>> > > > >>>> more
>> > > > >>>>>>>>>>>> complicated and it would require changing the records
>> on
>> > the
>> > > > >>>>> server
>> > > > >>>>>>>>>> too.
>> > > > >>>>>>>>>>>> This is why I initially suggested the rejected
>> alternative
>> > > > >> #2
>> > > > >>>>> which
>> > > > >>>>>>> is
>> > > > >>>>>>>>>>>> pretty close but also not perfect.
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>> I'd like to clarify one thing. The
>> ConsumerGroupHeartbeat
>> > > > >> API
>> > > > >>>>>>> already
>> > > > >>>>>>>>>>>> supports generating the member id on the client so we
>> > don't
>> > > > >>> need
>> > > > >>>>> any
>> > > > >>>>>>>>>>>> conditional logic on the client side. This is actually
>> > what
>> > > > >> we
>> > > > >>>>>>> wanted
>> > > > >>>>>>>>>> to
>> > > > >>>>>>>>>>> do
>> > > > >>>>>>>>>>>> in the first place but the idea got pushed back by
>> Magnus
>> > > > >> back
>> > > > >>>>> then
>> > > > >>>>>>>>>>> because
>> > > > >>>>>>>>>>>> generating uuid from librdkafka required a new
>> dependency.
>> > > > >> It
>> > > > >>>>> turns
>> > > > >>>>>>>>>> out
>> > > > >>>>>>>>>>>> that librdkafka has that dependency today. In
>> retrospect,
>> > we
>> > > > >>>>> should
>> > > > >>>>>>>>>> have
>> > > > >>>>>>>>>>>> pushed back on this. Long story short, we can just do
>> it.
>> > > > >> The
>> > > > >>>>>>>>>> proposal in
>> > > > >>>>>>>>>>>> this KIP is to make the member id required in future
>> > > > >> versions.
>> > > > >>>> We
>> > > > >>>>>>>>>> could
>> > > > >>>>>>>>>>>> also decide not to do it and to keep supporting both
>> > > > >>>> approaches. I
>> > > > >>>>>>>>>> would
>> > > > >>>>>>>>>>>> also be fine with this.
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>> Best,
>> > > > >>>>>>>>>>>> David
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>> On Wed, Aug 14, 2024 at 12:30 PM Andrew Schofield <
>> > > > >>>>>>>>>>>> andrew_schofield@live.com>
>> > > > >>>>>>>>>>>> wrote:
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>>> Hi TengYao,
>> > > > >>>>>>>>>>>>> Thanks for your response. I’ll have just one more try
>> to
>> > > > >>>>> persuade.
>> > > > >>>>>>>>>>>>> I feel that I will need to follow the approach with
>> > KIP-932
>> > > > >>>> when
>> > > > >>>>>>>>>> we’ve
>> > > > >>>>>>>>>>>>> made a decision, so I do have more than a passing
>> > interest
>> > > > >> in
>> > > > >>>>> this.
>> > > > >>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>> A group member in the lobby is in the group, but it
>> does
>> > > > >> not
>> > > > >>>> have
>> > > > >>>>>>>>>> any
>> > > > >>>>>>>>>>>>> assignments. A member of a consumer group can have no
>> > > > >>> assigned
>> > > > >>>>>>>>>>>>> partitions (such as 5 CG members subscribed to a topic
>> > > > >> with 4
>> > > > >>>>>>>>>>>> partitions),
>> > > > >>>>>>>>>>>>> so it’s a situation that consumer group members
>> already
>> > > > >>> expect.
>> > > > >>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>> One of Kafka’s strengths is the way that we handle API
>> > > > >>>>> versioning.
>> > > > >>>>>>>>>>>>> But, there is a cost - the behaviour is different
>> > depending
>> > > > >>> on
>> > > > >>>>> the
>> > > > >>>>>>>>>> RPC
>> > > > >>>>>>>>>>>>> version. KIP-848 is on the cusp of completion, but
>> we’re
>> > > > >>>> already
>> > > > >>>>>>>>>> adding
>> > > > >>>>>>>>>>>>> conditional logic for v0/v1 for
>> ConsumerGroupHeartbeat.
>> > > > >>> That’s
>> > > > >>>> a
>> > > > >>>>>>>>>> pity.
>> > > > >>>>>>>>>>>>> Only a minor issue, but it’s unfortunate.
>> > > > >>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>> Thanks,
>> > > > >>>>>>>>>>>>> Andrew
>> > > > >>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>> On 14 Aug 2024, at 08:47, TengYao Chi <
>> > > > >> kitingiao@gmail.com>
>> > > > >>>>>>>>>> wrote:
>> > > > >>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>> Hello Andrew
>> > > > >>>>>>>>>>>>>> Thank you for your thoughtful suggestions and getting
>> > the
>> > > > >>>>>>>>>> discussion
>> > > > >>>>>>>>>>>>> going.
>> > > > >>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>> To AS1:
>> > > > >>>>>>>>>>>>>> In the current scenario where the server generates
>> the
>> > > > >> UUID,
>> > > > >>>> if
>> > > > >>>>>>>>>> the
>> > > > >>>>>>>>>>>>> client
>> > > > >>>>>>>>>>>>>> shuts down before receiving the memberId generated by
>> > the
>> > > > >> GC
>> > > > >>>>>>>>>>>> (regardless
>> > > > >>>>>>>>>>>>> of
>> > > > >>>>>>>>>>>>>> whether it’s a graceful shutdown or not), the GC will
>> > > > >> still
>> > > > >>>> have
>> > > > >>>>>>>>>> to
>> > > > >>>>>>>>>>>> wait
>> > > > >>>>>>>>>>>>>> for the heartbeat timeout because the client doesn’t
>> > know
>> > > > >>> its
>> > > > >>>>>>>>>>> memberId.
>> > > > >>>>>>>>>>>>>> This KIP indeed cannot completely resolve the
>> > idempotency
>> > > > >>>> issue,
>> > > > >>>>>>>>>> but
>> > > > >>>>>>>>>>> it
>> > > > >>>>>>>>>>>>> can
>> > > > >>>>>>>>>>>>>> better handle shutdown scenarios under normal
>> > > > >> circumstances
>> > > > >>>>>>>>>> because
>> > > > >>>>>>>>>>> the
>> > > > >>>>>>>>>>>>>> client always knows its memberId. Even if the client
>> > shuts
>> > > > >>>> down
>> > > > >>>>>>>>>>>>> immediately
>> > > > >>>>>>>>>>>>>> after the initial heartbeat, as long as it performs a
>> > > > >>> graceful
>> > > > >>>>>>>>>>> shutdown
>> > > > >>>>>>>>>>>>> and
>> > > > >>>>>>>>>>>>>> sends a leave heartbeat, the GC can manage the
>> situation
>> > > > >> and
>> > > > >>>>>>>>>> remove
>> > > > >>>>>>>>>>> the
>> > > > >>>>>>>>>>>>>> member. Therefore, the goal of this KIP is to address
>> > the
>> > > > >>>> issue
>> > > > >>>>>>>>>> where
>> > > > >>>>>>>>>>>> the
>> > > > >>>>>>>>>>>>>> GC has to wait for the heartbeat timeout due to the
>> > client
>> > > > >>>>> leaving
>> > > > >>>>>>>>>>>>> without
>> > > > >>>>>>>>>>>>>> knowing its memberId, which leads to reduced
>> throughput
>> > > > >> and
>> > > > >>>>>>>>>> limited
>> > > > >>>>>>>>>>>>>> scalability.
>> > > > >>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>> The solution you suggest has also been proposed by
>> > David.
>> > > > >>> The
>> > > > >>>>>>>>>> concern
>> > > > >>>>>>>>>>>>> with
>> > > > >>>>>>>>>>>>>> this approach is that it introduces additional
>> > complexity
>> > > > >>> for
>> > > > >>>>>>>>>>>>>> compatibility, as the new server would not
>> immediately
>> > add
>> > > > >>> the
>> > > > >>>>>>>>>> member
>> > > > >>>>>>>>>>>> to
>> > > > >>>>>>>>>>>>>> the group, while the old server would. This requires
>> > > > >> clients
>> > > > >>>> to
>> > > > >>>>>>>>>>>>>> differentiate whether their memberId has been added
>> to
>> > the
>> > > > >>>> group
>> > > > >>>>>>>>>> or
>> > > > >>>>>>>>>>>> not,
>> > > > >>>>>>>>>>>>>> which could result in unexpected logs.
>> > > > >>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>> Best Regards,
>> > > > >>>>>>>>>>>>>> TengYao
>> > > > >>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>> Andrew Schofield <an...@live.com> 於
>> > 2024年8月14日
>> > > > >>> 週三
>> > > > >>>>>>>>>>>> 上午12:29寫道：
>> > > > >>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>>> Hi TengYao,
>> > > > >>>>>>>>>>>>>>> Thanks for the KIP. I wonder if there’s a different
>> way
>> > > > >> to
>> > > > >>>>> close
>> > > > >>>>>>>>>>> what
>> > > > >>>>>>>>>>>>>>> is quite a small window.
>> > > > >>>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>>> AS1: It is true that the initial heartbeat is not
>> > > > >>> idempotent,
>> > > > >>>>> but
>> > > > >>>>>>>>>>> this
>> > > > >>>>>>>>>>>>>>> remains
>> > > > >>>>>>>>>>>>>>> true with this KIP. It’s just differently not
>> > idempotent.
>> > > > >>> If
>> > > > >>>>> the
>> > > > >>>>>>>>>>>> client
>> > > > >>>>>>>>>>>>>>> makes its
>> > > > >>>>>>>>>>>>>>> own member ID, sends a request and dies, the GC will
>> > > > >> still
>> > > > >>>> have
>> > > > >>>>>>>>>>> added
>> > > > >>>>>>>>>>>>>>> the member to the group and it will hang around
>> until
>> > the
>> > > > >>>>> session
>> > > > >>>>>>>>>>>>> expires.
>> > > > >>>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>>> I wonder if the GC could still generate the member
>> ID
>> > in
>> > > > >>>>>>>>>> response to
>> > > > >>>>>>>>>>>> the
>> > > > >>>>>>>>>>>>>>> first
>> > > > >>>>>>>>>>>>>>> heartbeat, and put the member in a special PENDING
>> > state
>> > > > >>> with
>> > > > >>>>> no
>> > > > >>>>>>>>>>>>>>> assignments until the client sends the next
>> heartbeat,
>> > > > >> thus
>> > > > >>>>>>>>>>> confirming
>> > > > >>>>>>>>>>>>> it
>> > > > >>>>>>>>>>>>>>> has received the member ID. This would not be a
>> > protocol
>> > > > >>>> change
>> > > > >>>>>>>>>> at
>> > > > >>>>>>>>>>>> all,
>> > > > >>>>>>>>>>>>>>> just
>> > > > >>>>>>>>>>>>>>> a change to the GC to keep a new member in the lobby
>> > > > >> until
>> > > > >>>> it’s
>> > > > >>>>>>>>>>>>> comfirmed
>> > > > >>>>>>>>>>>>>>> it knows its member ID.
>> > > > >>>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>>> Thanks,
>> > > > >>>>>>>>>>>>>>> Andrew
>> > > > >>>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>>>> On 13 Aug 2024, at 15:59, TengYao Chi <
>> > > > >>> kitingiao@gmail.com>
>> > > > >>>>>>>>>> wrote:
>> > > > >>>>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>>>> Hi Chia-Ping,
>> > > > >>>>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>>>> Thanks for review and suggestions.
>> > > > >>>>>>>>>>>>>>>> I have updated the content of KIP accordingly.
>> > > > >>>>>>>>>>>>>>>> Please take a look.
>> > > > >>>>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>>>> Best regards,
>> > > > >>>>>>>>>>>>>>>> TengYao
>> > > > >>>>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2024年8月13日
>> 週二
>> > > > >>>>> 下午9:45寫道：
>> > > > >>>>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>>>>> hi TengYao
>> > > > >>>>>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>>>>> thanks for this KIP.
>> > > > >>>>>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>>>>> 1) could you please describe the before/after
>> > behavior
>> > > > >> in
>> > > > >>>> the
>> > > > >>>>>>>>>>>>> "Proposed
>> > > > >>>>>>>>>>>>>>>>> Changes" section? IIRC, current RPC allows HB
>> having
>> > > > >>> member
>> > > > >>>>> id
>> > > > >>>>>>>>>>>>>>> generated by
>> > > > >>>>>>>>>>>>>>>>> client, right? If HB has no member ID, server will
>> > > > >>> generate
>> > > > >>>>> one
>> > > > >>>>>>>>>>> and
>> > > > >>>>>>>>>>>>> then
>> > > > >>>>>>>>>>>>>>>>> return. The new behavior will enforce HB "must"
>> have
>> > > > >>> member
>> > > > >>>>> ID.
>> > > > >>>>>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>>>>> 2) could you please write the version number
>> > explicitly
>> > > > >>> in
>> > > > >>>>> the
>> > > > >>>>>>>>>> KIP
>> > > > >>>>>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>>>>> 3) how new client code handle the old HB? Does it
>> > > > >> always
>> > > > >>>>>>>>>> generate
>> > > > >>>>>>>>>>>>> member
>> > > > >>>>>>>>>>>>>>>>> ID on client-side even though that is not
>> restricted?
>> > > > >>>>>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>>>>> Best,
>> > > > >>>>>>>>>>>>>>>>> Chia-Ping
>> > > > >>>>>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>>>>> On 2024/08/13 06:20:42 TengYao Chi wrote:
>> > > > >>>>>>>>>>>>>>>>>> Hello everyone,
>> > > > >>>>>>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>>>>>> I would like to start a discussion thread on
>> > KIP-1082,
>> > > > >>>> which
>> > > > >>>>>>>>>>>> proposes
>> > > > >>>>>>>>>>>>>>>>>> enabling id generation for clients over the
>> > > > >>>>>>>>>>> ConsumerGroupHeartbeat
>> > > > >>>>>>>>>>>>> RPC.
>> > > > >>>>>>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>>>>>> Here is the KIP Link: KIP-1082
>> > > > >>>>>>>>>>>>>>>>>> <
>> > > > >>>>>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>
>> > > > >>>>>>>
>> > > > >>>>>
>> > > > >>>>
>> > > > >>>
>> > > > >>
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
>> > > > >>>>>>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>>>>>> Please take a look and let me know what you
>> think,
>> > > > >> and I
>> > > > >>>>> would
>> > > > >>>>>>>>>>>>>>> appreciate
>> > > > >>>>>>>>>>>>>>>>>> any suggestions and feedback.
>> > > > >>>>>>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>>>>>> Best regards,
>> > > > >>>>>>>>>>>>>>>>>> TengYao
>> > > > >>>>>>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>>
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>
>> > > > >>>>>>>
>> > > > >>>>>>>
>> > > > >>>>>
>> > > > >>>>
>> > > > >>>
>> > > > >>
>> > > >
>> > > >
>> > >
>> >
>>
>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[TengYao Chi <ki...@gmail.com>]

Hello everyone,

KT2: It looks like everyone who has expressed an opinion supports the
second option: “Document a recommendation for clients to use UUIDs as
member IDs, without strictly enforcing it.”
I have updated the KIP accordingly.
Please take a look, and let me know if you have any thoughts or feedback.

Thank you!

Best regards,
TengYao

Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月30日 週五 下午9:56寫道：

> hi TengYao
>
> KT2: +1 to second approach
>
> Best,
> Chia-Ping
>
>
> David Jacot <dj...@confluent.io.invalid> 於 2024年8月30日 週五 下午9:15寫道：
>
> > Hi TengYao,
> >
> > KT2: I don't think that we can realistically validate the uuid on the
> > server. It is basically a string of chars. So I lean towards having a
> good
> > recommendation in the KIP and in the document of the field in the RPC's
> > definition.
> >
> > Best,
> > David
> >
> > On Fri, Aug 30, 2024 at 3:02 PM TengYao Chi <ki...@gmail.com> wrote:
> >
> > > Hello Kirk !
> > >
> > > Thank you for your comments !
> > >
> > > KT1: Yes, you are correct. The issue is not unique to the initial
> > > heartbeat; there can always be cases where the broker might lose
> > connection
> > > with a member.
> > >
> > > KT2: Currently, if the client doesn't have a member ID and the
> > memberEpoch
> > > equals 0, the coordinator will generate a UUID as the member ID for the
> > > client. However, at the RPC level, the member ID is sent as a literal
> > > string, meaning there are no restrictions on the format at this level.
> > > This also reminds me that we haven't reached a final conclusion on how
> to
> > > enforce the use of UUIDs.
> > > From our previous discussions, I recall two possible approaches:
> > > The first is to validate the UUID on the server side, and if it's not
> > > valid, throw an exception to the client.
> > > The second is to document a recommendation for clients to use UUIDs as
> > > member IDs, without strictly enforcing it.
> > > I think it's time to decide on the approach we want to take.
> > >
> > > KT3: Yes, "session" can be considered synonymous with "membership" in
> > this
> > > context.
> > >
> > > KT4: Thank you for pointing that out. I will update the wording to
> > > specifically say this behavior is for consumers.
> > >
> > > Thanks again for your comments.
> > >
> > > Best regards,
> > > TengYao
> > >
> > > Kirk True <ki...@kirktrue.pro> 於 2024年8月30日 週五 上午12:39寫道：
> > >
> > > > Hi TengYao!
> > > >
> > > > Sorry for being late to the discussion...
> > > >
> > > > After reading the thread and then the KIP, I had a few
> > > questions/comments:
> > > >
> > > > KT1: In Motivation, it states: "This scenario can result in the
> broker
> > > > registering a new member for which it will never receive a proper
> leave
> > > > request.” Just to be clear, the broker will always have cases where
> it
> > > > might lose connection with a member. That’s not unique to the initial
> > > > heartbeat, right?
> > > >
> > > > KT2: There was a bit of back and forth about format of the member ID.
> > > From
> > > > what I gathered in the thread, the member ID is still defined in the
> > RPC
> > > as
> > > > a string and not a UUID, right? The KIP states that the “client must
> > > > generate a UUID as the member ID” and that the “server will validate
> > > that a
> > > > valid UUID is provided.” Is that a change for the server, or is it
> > > already
> > > > enforced as a UUID?
> > > >
> > > > KT3: Lianet mentioned some confusion over the use of the word
> > “session.”
> > > > Isn’t “session” synonymous with “membership?”
> > > >
> > > > KT4: Under “Member ID Lifecycle,” it states: "The client should reuse
> > the
> > > > same UUID as the member ID for all heartbeats and rejoin attempts to
> > > > maintain continuity within the group.” Could we change the first part
> > of
> > > > that to “The Consumer instance should…” We do have lifetimes that
> > extend
> > > > past the lifetime of a client instance (such as the transaction ID).
> > > >
> > > > Thanks,
> > > > Kirk
> > > >
> > > > > On Aug 29, 2024, at 1:28 AM, TengYao Chi <ki...@gmail.com>
> > wrote:
> > > > >
> > > > > Hi David,
> > > > >
> > > > > Thank you for pointing that out.
> > > > > I have updated the content of the KIP based on Lianet's and your
> > > > feedback.
> > > > > Please take a look and let me know your thoughts.
> > > > >
> > > > > Best regards,
> > > > > TengYao
> > > > >
> > > > > David Jacot <dj...@confluent.io.invalid> 於 2024年8月29日 週四
> 下午3:20寫道：
> > > > >
> > > > >> Hi TengYao,
> > > > >>
> > > > >> Thanks for the update. I haven't fully read it yet but I will
> soon.
> > > > >>
> > > > >> LM4: This is incorrect. The consumer must keep its member id
> during
> > > its
> > > > >> entire lifetime (until the process stops or dies). The protocol
> > > > stipulates
> > > > >> that a member must rejoin with the same member id and the member
> > epoch
> > > > set
> > > > >> to zero when an FENCED_MEMBER_EPOCH occurs. This allows the member
> > to
> > > > >> resynchronize itself. We should not change this behavior. I think
> > that
> > > > we
> > > > >> should see the client side generation id as an incarnation id of
> the
> > > > >> application. It is generated once and kept until it stops or dies.
> > > > >>
> > > > >> Best,
> > > > >> David
> > > > >>
> > > > >> On Thu, Aug 29, 2024 at 6:21 AM TengYao Chi <ki...@gmail.com>
> > > > wrote:
> > > > >>
> > > > >>> Hello Lianet !
> > > > >>>
> > > > >>> Thanks for the reviews and suggestions!
> > > > >>>
> > > > >>> LM1: Indeed, we plan to enforce client-side ID generation in the
> > > > future,
> > > > >>> and it is not an alternative. I will change the title
> accordingly.
> > > > >>>
> > > > >>> LM2: Yes, that's the expectation. I will add that statement to
> the
> > > > public
> > > > >>> interface section.
> > > > >>>
> > > > >>> LM3: Thank you for the high-level perspective review. I think
> > you're
> > > > >> right;
> > > > >>> our intention isn't very clear since it was placed at the end of
> > the
> > > > >>> section. I will try to rephrase that section to make it more
> > obvious.
> > > > >>>
> > > > >>> LM4: Regarding the definition of "session" in this KIP, I believe
> > it
> > > > >> refers
> > > > >>> to the period between the *first-time heartbeat* and when the
> > > *consumer
> > > > >>> leaves the group* (whether through a graceful shutdown or a
> > heartbeat
> > > > >>> timeout). The consumer should reuse its UUID if it has been
> > generated
> > > > >>> before. The only situation in which it will regenerate the UUID
> is
> > if
> > > > the
> > > > >>> coordinator finds that there is already a consumer with the same
> > > UUID.
> > > > >>> IIRC, the coordinator should compare the member epochs, and the
> > > > >>> later-joined consumer should be fenced off by the coordinator due
> > to
> > > > >> having
> > > > >>> a lower member epoch. Once the consumer receives a
> > > > `FENCED_MEMBER_EPOCH`
> > > > >>> error, it will generate a new UUID and attempt to rejoin. I will
> > > > clarify
> > > > >>> this in the KIP.
> > > > >>>
> > > > >>> Thanks again for your reviews, I really appreciate it.
> > > > >>>
> > > > >>> Best regards,
> > > > >>> TengYao
> > > > >>>
> > > > >>> Lianet M. <li...@gmail.com> 於 2024年8月28日 週三 下午7:12寫道：
> > > > >>>
> > > > >>>> Hello TengYao! Thanks for taking on this issue, we've been going
> > > > around
> > > > >>> it
> > > > >>>> for a while.
> > > > >>>>
> > > > >>>> LM1: About the title of the KIP: "Enable ID Generation for
> Clients
> > > > over
> > > > >>> the
> > > > >>>> ConsumerGroupHeartbeat RPC". I find it confusing because it
> hints
> > > that
> > > > >>>> we're adding it as an alternative (which was discussed and
> > > discarded,
> > > > >> in
> > > > >>>> favour of really enforcing it). It's also missing the core
> change
> > > imo,
> > > > >>>> which is "where" the generation happens. So, maybe more to the
> > point
> > > > >> with
> > > > >>>> something along the lines of "Client-side generated ID for
> clients
> > > > over
> > > > >>>> ConsumerGroupHeartbeat RPC"?
> > > > >>>>
> > > > >>>> LM2: On the public interfaces section, the KIP states that "the
> > > server
> > > > >>> will
> > > > >>>> reject the request", but we should agree on the specific error
> > > type. I
> > > > >>>> expect it should fail with an InvalidRequestException, is that
> the
> > > > >>>> intention? (This was also suggested in the discussion thread
> > before
> > > > but
> > > > >>> is
> > > > >>>> not in the KIP).
> > > > >>>>
> > > > >>>> LM3. Related to my previous point, I find that to be the true
> > > > >>> public-facing
> > > > >>>> change (member ID mandatory at the protocol level), but it's
> only
> > at
> > > > >> the
> > > > >>>> end of the Public interfaces changes, kind of lost among details
> > of
> > > > how
> > > > >>>> we're going to do it. Should we rephrase that section with the
> > > actual
> > > > >>>> change first, and the hows after (ex. Bumping the version is not
> > the
> > > > >>>> public-facing change in this case, it's just the mechanism to
> > > properly
> > > > >>>> introduce our change)
> > > > >>>>
> > > > >>>> LM4. Regarding the lifetime of the UUID: the KIP states we will
> > > > "Verify
> > > > >>>> that the UUID remains consistent across all subsequent
> heartbeats
> > > > >> during
> > > > >>>> the session". What is this "session" referring to here? I would
> > > expect
> > > > >>> that
> > > > >>>> the UUID is associated to a consumer instance (generated for the
> > > > >> consumer
> > > > >>>> the first time it needs to send a HB if it doesn't have the UUID
> > > yet.
> > > > >>> From
> > > > >>>> there on, every time it needs to send a "first HB" again, it
> will
> > > > reuse
> > > > >>> its
> > > > >>>> UUID, is that the intention? Note that we should consider that
> the
> > > > same
> > > > >>>> consumer instance may have many "first heartbeats", meaning
> > > heartbeats
> > > > >> to
> > > > >>>> join the group when it's not part of it (ex. consumer
> unsubscribe
> > +
> > > > >>>> subscribe, fenced, stale). Is this the intention or are you
> > > > considering
> > > > >>> the
> > > > >>>> lifetime differently? We should clarify it in the KIP.
> > > > >>>>
> > > > >>>> Thanks!
> > > > >>>>
> > > > >>>> Lianet
> > > > >>>>
> > > > >>>> On Tue, Aug 27, 2024 at 2:27 AM TengYao Chi <
> kitingiao@gmail.com>
> > > > >> wrote:
> > > > >>>>
> > > > >>>>> Hi everyone,
> > > > >>>>>
> > > > >>>>> I have revised this KIP multiple times based on the feedback
> from
> > > our
> > > > >>>>> discussions.
> > > > >>>>> I would greatly appreciate it if you could review it when you
> > have
> > > > >> the
> > > > >>>>> time.
> > > > >>>>> If there are no further comments or suggestions, I plan to
> > proceed
> > > > >> with
> > > > >>>>> initiating a vote soon.
> > > > >>>>>
> > > > >>>>> Best regards,
> > > > >>>>> TengYao
> > > > >>>>>
> > > > >>>>> TengYao Chi <ki...@gmail.com> 於 2024年8月23日 週五 下午2:43寫道：
> > > > >>>>>
> > > > >>>>>> Hi Andrew,
> > > > >>>>>> Thank you for your previous feedback and insights.
> > > > >>>>>> Your contributions have added valuable perspectives to the
> > > > >>> discussions.
> > > > >>>>>> And we also benefit from the comparison of different
> solutions.
> > > > >>>>>> I’m also looking forward to seeing an initial version in
> > KIP-932,
> > > > >> as
> > > > >>> it
> > > > >>>>>> will provide a good reference for future implementations.
> > > > >>>>>>
> > > > >>>>>> Regarding your comment on AS2, I wanted to clarify that my
> > > > >>>> specification
> > > > >>>>>> references org.apache.kafka.common.Uuid.
> > > > >>>>>> I believe we’re referring to the same class, and it might just
> > be
> > > a
> > > > >>>> small
> > > > >>>>>> oversight due to the busy schedule.
> > > > >>>>>>
> > > > >>>>>> I want to express my gratitude once again for your many
> > insightful
> > > > >>>>>> comments, which have helped the discussion progress smoothly.
> > > > >>>>>>
> > > > >>>>>> Best regards,
> > > > >>>>>> TengYao
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> Andrew Schofield <an...@live.com> 於 2024年8月22日 週四
> > > > >>>> 下午11:28寫道：
> > > > >>>>>>
> > > > >>>>>>> Hi TengYao,
> > > > >>>>>>> I’ve been reading through the comments and I’m happy that the
> > > > >> lobby
> > > > >>>>>>> approach has not gained support.
> > > > >>>>>>>
> > > > >>>>>>> Assuming that this KIP is voted, I will be happy to change
> > > KIP-932
> > > > >>> so
> > > > >>>>>>> that it only supports client-side member ID generation.
> Because
> > > > >> that
> > > > >>>> KIP
> > > > >>>>>>> is still
> > > > >>>>>>> under development, I can do this in the first version of
> > > > >>>>>>> ShareGroupHeartbeat.
> > > > >>>>>>>
> > > > >>>>>>> AS2: For the encoding section, I suppose the specific
> encoding
> > > > >> which
> > > > >>>>>>> is used is what org.apache.kafka.utils.Uuid uses.
> > > > >>>>>>>
> > > > >>>>>>> Thanks,
> > > > >>>>>>> Andrew
> > > > >>>>>>>
> > > > >>>>>>>> On 14 Aug 2024, at 17:00, TengYao Chi <ki...@gmail.com>
> > > > >>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>> Hello Apoorv,
> > > > >>>>>>>> Thank you for your feedback.
> > > > >>>>>>>> Regarding the questions you raised, unfortunately, this KIP
> > > > >> cannot
> > > > >>>>>>>> guarantee the order of heartbeats. As with many classic
> > > > >>> distributed
> > > > >>>>>>> system
> > > > >>>>>>>> challenges, what we can do is make our best effort to ensure
> > > > >> that
> > > > >>>>> there
> > > > >>>>>>> are
> > > > >>>>>>>> no idle members or stale assignments under normal
> > circumstances.
> > > > >>>>>>>>
> > > > >>>>>>>> As for the lobby approach, I’m not a fan of it because it
> > > > >> requires
> > > > >>>>>>> adding a
> > > > >>>>>>>> mechanism to maintain client state within the ConsumerGroup,
> > > > >>> which,
> > > > >>>> in
> > > > >>>>>>> my
> > > > >>>>>>>> view, resembles something like a two-phase commit. This
> would
> > > > >>>>> introduce
> > > > >>>>>>>> more complexity than the proposal in this KIP, which is
> > > > >> something
> > > > >>> we
> > > > >>>>>>> want
> > > > >>>>>>>> to avoid. KIP-848 aims to simplify the existing protocol,
> and
> > > > >>> while
> > > > >>>>> the
> > > > >>>>>>>> lobby approach is a good one, I believe it is not the right
> > fit
> > > > >>> for
> > > > >>>>> this
> > > > >>>>>>>> particular situation.
> > > > >>>>>>>>
> > > > >>>>>>>> Best regards,
> > > > >>>>>>>> TengYao
> > > > >>>>>>>>
> > > > >>>>>>>> TengYao Chi <ki...@gmail.com> 於 2024年8月14日 週三
> 下午11:45寫道：
> > > > >>>>>>>>
> > > > >>>>>>>>> Hi David,
> > > > >>>>>>>>>
> > > > >>>>>>>>> I really appreciate your review and suggestions. As I am
> > still
> > > > >>>>> gaining
> > > > >>>>>>>>> experience in writing KIPs, your input has been incredibly
> > > > >>>> helpful. I
> > > > >>>>>>> am
> > > > >>>>>>>>> currently applying your suggestions to the KIP and will
> > > > >> complete
> > > > >>> it
> > > > >>>>> as
> > > > >>>>>>> soon
> > > > >>>>>>>>> as possible.
> > > > >>>>>>>>> Regarding the UUID part, I think we haven’t reached a
> > > > >> conclusion
> > > > >>>>>>> yet.(So
> > > > >>>>>>>>> far according to this thread)
> > > > >>>>>>>>> However, I will review the current implementation in the
> > Kafka
> > > > >>>> `Uuid`
> > > > >>>>>>>>> class and include a brief specification in the KIP.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Once again, thank you so much for your help.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Best regards,
> > > > >>>>>>>>> TengYao
> > > > >>>>>>>>>
> > > > >>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月14日 週三
> > 下午11:14寫道：
> > > > >>>>>>>>>
> > > > >>>>>>>>>> hi Apoorv
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> As the memberId is now known to the client, and client
> > might
> > > > >>> send
> > > > >>>>> the
> > > > >>>>>>>>>> leave
> > > > >>>>>>>>>> group heartbeat on shutdown prior to receiving the initial
> > > > >>>> heartbeat
> > > > >>>>>>>>>> response. If that's true then how do we guarantee that
> the 2
> > > > >>>>> requests
> > > > >>>>>>> to
> > > > >>>>>>>>>> join and leave will be processed in order, which could
> still
> > > > >>> leave
> > > > >>>>>>> stale
> > > > >>>>>>>>>> members or throw unknown member id exceptions?
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> This is definitely a good question. the short answer: no
> > > > >>> guarantee
> > > > >>>>> but
> > > > >>>>>>>>>> best
> > > > >>>>>>>>>> efforts
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Please notice the root cause is "we have no enough time to
> > > > >> wait
> > > > >>>>>>> member id
> > > > >>>>>>>>>> (response) when closing consumer". Sadly, we can'
> guarantee
> > > > >> the
> > > > >>>>>>> request
> > > > >>>>>>>>>> order due to the same reason.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> However, in contrast to previous behavior, there is one
> big
> > > > >>>> benefit
> > > > >>>>>>> of new
> > > > >>>>>>>>>> approach - we can try STONITH because we know the member
> id
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Best,
> > > > >>>>>>>>>> Chia-Ping
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Apoorv Mittal <ap...@gmail.com> 於 2024年8月14日 週三
> > > > >>>> 下午8:55寫道：
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> Hi TengYao,
> > > > >>>>>>>>>>> Thanks for the KIP. Continuing on the point which Andrew
> > > > >>>> mentioned
> > > > >>>>> as
> > > > >>>>>>>>>> AS1.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> As the memberId is now known to the client, and client
> > might
> > > > >>> send
> > > > >>>>> the
> > > > >>>>>>>>>> leave
> > > > >>>>>>>>>>> group heartbeat on shutdown prior to receiving the
> initial
> > > > >>>>> heartbeat
> > > > >>>>>>>>>>> response. If that's true then how do we guarantee that
> the
> > 2
> > > > >>>>>>> requests to
> > > > >>>>>>>>>>> join and leave will be processed in order, which could
> > still
> > > > >>>> leave
> > > > >>>>>>> stale
> > > > >>>>>>>>>>> members or throw unknown member id exceptions?
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Though the client side member id generation is helpful
> > which
> > > > >>> will
> > > > >>>>>>>>>> represent
> > > > >>>>>>>>>>> the same group perspective as from client and broker's
> end.
> > > > >>> But I
> > > > >>>>>>> think
> > > > >>>>>>>>>> the
> > > > >>>>>>>>>>> major concern we want to solve here is Stale Partition
> > > > >>>> Assignments
> > > > >>>>>>> which
> > > > >>>>>>>>>>> might still exist with the new approach. I am leaning
> > towards
> > > > >>> the
> > > > >>>>>>>>>>> suggestion mentioned by Andrew where partition assignment
> > > > >>>> triggers
> > > > >>>>> on
> > > > >>>>>>>>>>> subsequent heartbeat when client acknowledges the initial
> > > > >>>>> heartbeat,
> > > > >>>>>>>>>>> delayed partition assignment.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Though on a separate note, I have a different question.
> > What
> > > > >>>>> happens
> > > > >>>>>>>>>> when
> > > > >>>>>>>>>>> there is an issue with the client which sends the initial
> > > > >>>> heartbeat
> > > > >>>>>>>>>> without
> > > > >>>>>>>>>>> memberId, then crashes and restarts? I think we must be
> > > > >>>>> re-triggering
> > > > >>>>>>>>>>> assignments and expiring members only after the heartbeat
> > > > >>> session
> > > > >>>>>>>>>> timeout?
> > > > >>>>>>>>>>> If that's true then shall delayed partition assignment
> can
> > > > >> help
> > > > >>>>>>> benefit
> > > > >>>>>>>>>> us
> > > > >>>>>>>>>>> from this situation as well?
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Regards,
> > > > >>>>>>>>>>> Apoorv Mittal
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> On Wed, Aug 14, 2024 at 12:51 PM David Jacot
> > > > >>>>>>>>>> <dj...@confluent.io.invalid>
> > > > >>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> Hi Andrew,
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Personally, I don't like the lobby approach. It makes
> > things
> > > > >>>> more
> > > > >>>>>>>>>>>> complicated and it would require changing the records on
> > the
> > > > >>>>> server
> > > > >>>>>>>>>> too.
> > > > >>>>>>>>>>>> This is why I initially suggested the rejected
> alternative
> > > > >> #2
> > > > >>>>> which
> > > > >>>>>>> is
> > > > >>>>>>>>>>>> pretty close but also not perfect.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> I'd like to clarify one thing. The
> ConsumerGroupHeartbeat
> > > > >> API
> > > > >>>>>>> already
> > > > >>>>>>>>>>>> supports generating the member id on the client so we
> > don't
> > > > >>> need
> > > > >>>>> any
> > > > >>>>>>>>>>>> conditional logic on the client side. This is actually
> > what
> > > > >> we
> > > > >>>>>>> wanted
> > > > >>>>>>>>>> to
> > > > >>>>>>>>>>> do
> > > > >>>>>>>>>>>> in the first place but the idea got pushed back by
> Magnus
> > > > >> back
> > > > >>>>> then
> > > > >>>>>>>>>>> because
> > > > >>>>>>>>>>>> generating uuid from librdkafka required a new
> dependency.
> > > > >> It
> > > > >>>>> turns
> > > > >>>>>>>>>> out
> > > > >>>>>>>>>>>> that librdkafka has that dependency today. In
> retrospect,
> > we
> > > > >>>>> should
> > > > >>>>>>>>>> have
> > > > >>>>>>>>>>>> pushed back on this. Long story short, we can just do
> it.
> > > > >> The
> > > > >>>>>>>>>> proposal in
> > > > >>>>>>>>>>>> this KIP is to make the member id required in future
> > > > >> versions.
> > > > >>>> We
> > > > >>>>>>>>>> could
> > > > >>>>>>>>>>>> also decide not to do it and to keep supporting both
> > > > >>>> approaches. I
> > > > >>>>>>>>>> would
> > > > >>>>>>>>>>>> also be fine with this.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Best,
> > > > >>>>>>>>>>>> David
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> On Wed, Aug 14, 2024 at 12:30 PM Andrew Schofield <
> > > > >>>>>>>>>>>> andrew_schofield@live.com>
> > > > >>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Hi TengYao,
> > > > >>>>>>>>>>>>> Thanks for your response. I’ll have just one more try
> to
> > > > >>>>> persuade.
> > > > >>>>>>>>>>>>> I feel that I will need to follow the approach with
> > KIP-932
> > > > >>>> when
> > > > >>>>>>>>>> we’ve
> > > > >>>>>>>>>>>>> made a decision, so I do have more than a passing
> > interest
> > > > >> in
> > > > >>>>> this.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> A group member in the lobby is in the group, but it
> does
> > > > >> not
> > > > >>>> have
> > > > >>>>>>>>>> any
> > > > >>>>>>>>>>>>> assignments. A member of a consumer group can have no
> > > > >>> assigned
> > > > >>>>>>>>>>>>> partitions (such as 5 CG members subscribed to a topic
> > > > >> with 4
> > > > >>>>>>>>>>>> partitions),
> > > > >>>>>>>>>>>>> so it’s a situation that consumer group members already
> > > > >>> expect.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> One of Kafka’s strengths is the way that we handle API
> > > > >>>>> versioning.
> > > > >>>>>>>>>>>>> But, there is a cost - the behaviour is different
> > depending
> > > > >>> on
> > > > >>>>> the
> > > > >>>>>>>>>> RPC
> > > > >>>>>>>>>>>>> version. KIP-848 is on the cusp of completion, but
> we’re
> > > > >>>> already
> > > > >>>>>>>>>> adding
> > > > >>>>>>>>>>>>> conditional logic for v0/v1 for ConsumerGroupHeartbeat.
> > > > >>> That’s
> > > > >>>> a
> > > > >>>>>>>>>> pity.
> > > > >>>>>>>>>>>>> Only a minor issue, but it’s unfortunate.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>>>> Andrew
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> On 14 Aug 2024, at 08:47, TengYao Chi <
> > > > >> kitingiao@gmail.com>
> > > > >>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Hello Andrew
> > > > >>>>>>>>>>>>>> Thank you for your thoughtful suggestions and getting
> > the
> > > > >>>>>>>>>> discussion
> > > > >>>>>>>>>>>>> going.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> To AS1:
> > > > >>>>>>>>>>>>>> In the current scenario where the server generates the
> > > > >> UUID,
> > > > >>>> if
> > > > >>>>>>>>>> the
> > > > >>>>>>>>>>>>> client
> > > > >>>>>>>>>>>>>> shuts down before receiving the memberId generated by
> > the
> > > > >> GC
> > > > >>>>>>>>>>>> (regardless
> > > > >>>>>>>>>>>>> of
> > > > >>>>>>>>>>>>>> whether it’s a graceful shutdown or not), the GC will
> > > > >> still
> > > > >>>> have
> > > > >>>>>>>>>> to
> > > > >>>>>>>>>>>> wait
> > > > >>>>>>>>>>>>>> for the heartbeat timeout because the client doesn’t
> > know
> > > > >>> its
> > > > >>>>>>>>>>> memberId.
> > > > >>>>>>>>>>>>>> This KIP indeed cannot completely resolve the
> > idempotency
> > > > >>>> issue,
> > > > >>>>>>>>>> but
> > > > >>>>>>>>>>> it
> > > > >>>>>>>>>>>>> can
> > > > >>>>>>>>>>>>>> better handle shutdown scenarios under normal
> > > > >> circumstances
> > > > >>>>>>>>>> because
> > > > >>>>>>>>>>> the
> > > > >>>>>>>>>>>>>> client always knows its memberId. Even if the client
> > shuts
> > > > >>>> down
> > > > >>>>>>>>>>>>> immediately
> > > > >>>>>>>>>>>>>> after the initial heartbeat, as long as it performs a
> > > > >>> graceful
> > > > >>>>>>>>>>> shutdown
> > > > >>>>>>>>>>>>> and
> > > > >>>>>>>>>>>>>> sends a leave heartbeat, the GC can manage the
> situation
> > > > >> and
> > > > >>>>>>>>>> remove
> > > > >>>>>>>>>>> the
> > > > >>>>>>>>>>>>>> member. Therefore, the goal of this KIP is to address
> > the
> > > > >>>> issue
> > > > >>>>>>>>>> where
> > > > >>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>> GC has to wait for the heartbeat timeout due to the
> > client
> > > > >>>>> leaving
> > > > >>>>>>>>>>>>> without
> > > > >>>>>>>>>>>>>> knowing its memberId, which leads to reduced
> throughput
> > > > >> and
> > > > >>>>>>>>>> limited
> > > > >>>>>>>>>>>>>> scalability.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> The solution you suggest has also been proposed by
> > David.
> > > > >>> The
> > > > >>>>>>>>>> concern
> > > > >>>>>>>>>>>>> with
> > > > >>>>>>>>>>>>>> this approach is that it introduces additional
> > complexity
> > > > >>> for
> > > > >>>>>>>>>>>>>> compatibility, as the new server would not immediately
> > add
> > > > >>> the
> > > > >>>>>>>>>> member
> > > > >>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>> the group, while the old server would. This requires
> > > > >> clients
> > > > >>>> to
> > > > >>>>>>>>>>>>>> differentiate whether their memberId has been added to
> > the
> > > > >>>> group
> > > > >>>>>>>>>> or
> > > > >>>>>>>>>>>> not,
> > > > >>>>>>>>>>>>>> which could result in unexpected logs.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>>>>> TengYao
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Andrew Schofield <an...@live.com> 於
> > 2024年8月14日
> > > > >>> 週三
> > > > >>>>>>>>>>>> 上午12:29寫道：
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Hi TengYao,
> > > > >>>>>>>>>>>>>>> Thanks for the KIP. I wonder if there’s a different
> way
> > > > >> to
> > > > >>>>> close
> > > > >>>>>>>>>>> what
> > > > >>>>>>>>>>>>>>> is quite a small window.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> AS1: It is true that the initial heartbeat is not
> > > > >>> idempotent,
> > > > >>>>> but
> > > > >>>>>>>>>>> this
> > > > >>>>>>>>>>>>>>> remains
> > > > >>>>>>>>>>>>>>> true with this KIP. It’s just differently not
> > idempotent.
> > > > >>> If
> > > > >>>>> the
> > > > >>>>>>>>>>>> client
> > > > >>>>>>>>>>>>>>> makes its
> > > > >>>>>>>>>>>>>>> own member ID, sends a request and dies, the GC will
> > > > >> still
> > > > >>>> have
> > > > >>>>>>>>>>> added
> > > > >>>>>>>>>>>>>>> the member to the group and it will hang around until
> > the
> > > > >>>>> session
> > > > >>>>>>>>>>>>> expires.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> I wonder if the GC could still generate the member ID
> > in
> > > > >>>>>>>>>> response to
> > > > >>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>> first
> > > > >>>>>>>>>>>>>>> heartbeat, and put the member in a special PENDING
> > state
> > > > >>> with
> > > > >>>>> no
> > > > >>>>>>>>>>>>>>> assignments until the client sends the next
> heartbeat,
> > > > >> thus
> > > > >>>>>>>>>>> confirming
> > > > >>>>>>>>>>>>> it
> > > > >>>>>>>>>>>>>>> has received the member ID. This would not be a
> > protocol
> > > > >>>> change
> > > > >>>>>>>>>> at
> > > > >>>>>>>>>>>> all,
> > > > >>>>>>>>>>>>>>> just
> > > > >>>>>>>>>>>>>>> a change to the GC to keep a new member in the lobby
> > > > >> until
> > > > >>>> it’s
> > > > >>>>>>>>>>>>> comfirmed
> > > > >>>>>>>>>>>>>>> it knows its member ID.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>>>>>> Andrew
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> On 13 Aug 2024, at 15:59, TengYao Chi <
> > > > >>> kitingiao@gmail.com>
> > > > >>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Hi Chia-Ping,
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Thanks for review and suggestions.
> > > > >>>>>>>>>>>>>>>> I have updated the content of KIP accordingly.
> > > > >>>>>>>>>>>>>>>> Please take a look.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Best regards,
> > > > >>>>>>>>>>>>>>>> TengYao
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2024年8月13日
> 週二
> > > > >>>>> 下午9:45寫道：
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> hi TengYao
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> thanks for this KIP.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> 1) could you please describe the before/after
> > behavior
> > > > >> in
> > > > >>>> the
> > > > >>>>>>>>>>>>> "Proposed
> > > > >>>>>>>>>>>>>>>>> Changes" section? IIRC, current RPC allows HB
> having
> > > > >>> member
> > > > >>>>> id
> > > > >>>>>>>>>>>>>>> generated by
> > > > >>>>>>>>>>>>>>>>> client, right? If HB has no member ID, server will
> > > > >>> generate
> > > > >>>>> one
> > > > >>>>>>>>>>> and
> > > > >>>>>>>>>>>>> then
> > > > >>>>>>>>>>>>>>>>> return. The new behavior will enforce HB "must"
> have
> > > > >>> member
> > > > >>>>> ID.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> 2) could you please write the version number
> > explicitly
> > > > >>> in
> > > > >>>>> the
> > > > >>>>>>>>>> KIP
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> 3) how new client code handle the old HB? Does it
> > > > >> always
> > > > >>>>>>>>>> generate
> > > > >>>>>>>>>>>>> member
> > > > >>>>>>>>>>>>>>>>> ID on client-side even though that is not
> restricted?
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Best,
> > > > >>>>>>>>>>>>>>>>> Chia-Ping
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> On 2024/08/13 06:20:42 TengYao Chi wrote:
> > > > >>>>>>>>>>>>>>>>>> Hello everyone,
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> I would like to start a discussion thread on
> > KIP-1082,
> > > > >>>> which
> > > > >>>>>>>>>>>> proposes
> > > > >>>>>>>>>>>>>>>>>> enabling id generation for clients over the
> > > > >>>>>>>>>>> ConsumerGroupHeartbeat
> > > > >>>>>>>>>>>>> RPC.
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Here is the KIP Link: KIP-1082
> > > > >>>>>>>>>>>>>>>>>> <
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Please take a look and let me know what you think,
> > > > >> and I
> > > > >>>>> would
> > > > >>>>>>>>>>>>>>> appreciate
> > > > >>>>>>>>>>>>>>>>>> any suggestions and feedback.
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Best regards,
> > > > >>>>>>>>>>>>>>>>>> TengYao
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[Chia-Ping Tsai <ch...@gmail.com>]

hi TengYao

KT2: +1 to second approach

Best,
Chia-Ping


David Jacot <dj...@confluent.io.invalid> 於 2024年8月30日 週五 下午9:15寫道：

> Hi TengYao,
>
> KT2: I don't think that we can realistically validate the uuid on the
> server. It is basically a string of chars. So I lean towards having a good
> recommendation in the KIP and in the document of the field in the RPC's
> definition.
>
> Best,
> David
>
> On Fri, Aug 30, 2024 at 3:02 PM TengYao Chi <ki...@gmail.com> wrote:
>
> > Hello Kirk !
> >
> > Thank you for your comments !
> >
> > KT1: Yes, you are correct. The issue is not unique to the initial
> > heartbeat; there can always be cases where the broker might lose
> connection
> > with a member.
> >
> > KT2: Currently, if the client doesn't have a member ID and the
> memberEpoch
> > equals 0, the coordinator will generate a UUID as the member ID for the
> > client. However, at the RPC level, the member ID is sent as a literal
> > string, meaning there are no restrictions on the format at this level.
> > This also reminds me that we haven't reached a final conclusion on how to
> > enforce the use of UUIDs.
> > From our previous discussions, I recall two possible approaches:
> > The first is to validate the UUID on the server side, and if it's not
> > valid, throw an exception to the client.
> > The second is to document a recommendation for clients to use UUIDs as
> > member IDs, without strictly enforcing it.
> > I think it's time to decide on the approach we want to take.
> >
> > KT3: Yes, "session" can be considered synonymous with "membership" in
> this
> > context.
> >
> > KT4: Thank you for pointing that out. I will update the wording to
> > specifically say this behavior is for consumers.
> >
> > Thanks again for your comments.
> >
> > Best regards,
> > TengYao
> >
> > Kirk True <ki...@kirktrue.pro> 於 2024年8月30日 週五 上午12:39寫道：
> >
> > > Hi TengYao!
> > >
> > > Sorry for being late to the discussion...
> > >
> > > After reading the thread and then the KIP, I had a few
> > questions/comments:
> > >
> > > KT1: In Motivation, it states: "This scenario can result in the broker
> > > registering a new member for which it will never receive a proper leave
> > > request.” Just to be clear, the broker will always have cases where it
> > > might lose connection with a member. That’s not unique to the initial
> > > heartbeat, right?
> > >
> > > KT2: There was a bit of back and forth about format of the member ID.
> > From
> > > what I gathered in the thread, the member ID is still defined in the
> RPC
> > as
> > > a string and not a UUID, right? The KIP states that the “client must
> > > generate a UUID as the member ID” and that the “server will validate
> > that a
> > > valid UUID is provided.” Is that a change for the server, or is it
> > already
> > > enforced as a UUID?
> > >
> > > KT3: Lianet mentioned some confusion over the use of the word
> “session.”
> > > Isn’t “session” synonymous with “membership?”
> > >
> > > KT4: Under “Member ID Lifecycle,” it states: "The client should reuse
> the
> > > same UUID as the member ID for all heartbeats and rejoin attempts to
> > > maintain continuity within the group.” Could we change the first part
> of
> > > that to “The Consumer instance should…” We do have lifetimes that
> extend
> > > past the lifetime of a client instance (such as the transaction ID).
> > >
> > > Thanks,
> > > Kirk
> > >
> > > > On Aug 29, 2024, at 1:28 AM, TengYao Chi <ki...@gmail.com>
> wrote:
> > > >
> > > > Hi David,
> > > >
> > > > Thank you for pointing that out.
> > > > I have updated the content of the KIP based on Lianet's and your
> > > feedback.
> > > > Please take a look and let me know your thoughts.
> > > >
> > > > Best regards,
> > > > TengYao
> > > >
> > > > David Jacot <dj...@confluent.io.invalid> 於 2024年8月29日 週四 下午3:20寫道：
> > > >
> > > >> Hi TengYao,
> > > >>
> > > >> Thanks for the update. I haven't fully read it yet but I will soon.
> > > >>
> > > >> LM4: This is incorrect. The consumer must keep its member id during
> > its
> > > >> entire lifetime (until the process stops or dies). The protocol
> > > stipulates
> > > >> that a member must rejoin with the same member id and the member
> epoch
> > > set
> > > >> to zero when an FENCED_MEMBER_EPOCH occurs. This allows the member
> to
> > > >> resynchronize itself. We should not change this behavior. I think
> that
> > > we
> > > >> should see the client side generation id as an incarnation id of the
> > > >> application. It is generated once and kept until it stops or dies.
> > > >>
> > > >> Best,
> > > >> David
> > > >>
> > > >> On Thu, Aug 29, 2024 at 6:21 AM TengYao Chi <ki...@gmail.com>
> > > wrote:
> > > >>
> > > >>> Hello Lianet !
> > > >>>
> > > >>> Thanks for the reviews and suggestions!
> > > >>>
> > > >>> LM1: Indeed, we plan to enforce client-side ID generation in the
> > > future,
> > > >>> and it is not an alternative. I will change the title accordingly.
> > > >>>
> > > >>> LM2: Yes, that's the expectation. I will add that statement to the
> > > public
> > > >>> interface section.
> > > >>>
> > > >>> LM3: Thank you for the high-level perspective review. I think
> you're
> > > >> right;
> > > >>> our intention isn't very clear since it was placed at the end of
> the
> > > >>> section. I will try to rephrase that section to make it more
> obvious.
> > > >>>
> > > >>> LM4: Regarding the definition of "session" in this KIP, I believe
> it
> > > >> refers
> > > >>> to the period between the *first-time heartbeat* and when the
> > *consumer
> > > >>> leaves the group* (whether through a graceful shutdown or a
> heartbeat
> > > >>> timeout). The consumer should reuse its UUID if it has been
> generated
> > > >>> before. The only situation in which it will regenerate the UUID is
> if
> > > the
> > > >>> coordinator finds that there is already a consumer with the same
> > UUID.
> > > >>> IIRC, the coordinator should compare the member epochs, and the
> > > >>> later-joined consumer should be fenced off by the coordinator due
> to
> > > >> having
> > > >>> a lower member epoch. Once the consumer receives a
> > > `FENCED_MEMBER_EPOCH`
> > > >>> error, it will generate a new UUID and attempt to rejoin. I will
> > > clarify
> > > >>> this in the KIP.
> > > >>>
> > > >>> Thanks again for your reviews, I really appreciate it.
> > > >>>
> > > >>> Best regards,
> > > >>> TengYao
> > > >>>
> > > >>> Lianet M. <li...@gmail.com> 於 2024年8月28日 週三 下午7:12寫道：
> > > >>>
> > > >>>> Hello TengYao! Thanks for taking on this issue, we've been going
> > > around
> > > >>> it
> > > >>>> for a while.
> > > >>>>
> > > >>>> LM1: About the title of the KIP: "Enable ID Generation for Clients
> > > over
> > > >>> the
> > > >>>> ConsumerGroupHeartbeat RPC". I find it confusing because it hints
> > that
> > > >>>> we're adding it as an alternative (which was discussed and
> > discarded,
> > > >> in
> > > >>>> favour of really enforcing it). It's also missing the core change
> > imo,
> > > >>>> which is "where" the generation happens. So, maybe more to the
> point
> > > >> with
> > > >>>> something along the lines of "Client-side generated ID for clients
> > > over
> > > >>>> ConsumerGroupHeartbeat RPC"?
> > > >>>>
> > > >>>> LM2: On the public interfaces section, the KIP states that "the
> > server
> > > >>> will
> > > >>>> reject the request", but we should agree on the specific error
> > type. I
> > > >>>> expect it should fail with an InvalidRequestException, is that the
> > > >>>> intention? (This was also suggested in the discussion thread
> before
> > > but
> > > >>> is
> > > >>>> not in the KIP).
> > > >>>>
> > > >>>> LM3. Related to my previous point, I find that to be the true
> > > >>> public-facing
> > > >>>> change (member ID mandatory at the protocol level), but it's only
> at
> > > >> the
> > > >>>> end of the Public interfaces changes, kind of lost among details
> of
> > > how
> > > >>>> we're going to do it. Should we rephrase that section with the
> > actual
> > > >>>> change first, and the hows after (ex. Bumping the version is not
> the
> > > >>>> public-facing change in this case, it's just the mechanism to
> > properly
> > > >>>> introduce our change)
> > > >>>>
> > > >>>> LM4. Regarding the lifetime of the UUID: the KIP states we will
> > > "Verify
> > > >>>> that the UUID remains consistent across all subsequent heartbeats
> > > >> during
> > > >>>> the session". What is this "session" referring to here? I would
> > expect
> > > >>> that
> > > >>>> the UUID is associated to a consumer instance (generated for the
> > > >> consumer
> > > >>>> the first time it needs to send a HB if it doesn't have the UUID
> > yet.
> > > >>> From
> > > >>>> there on, every time it needs to send a "first HB" again, it will
> > > reuse
> > > >>> its
> > > >>>> UUID, is that the intention? Note that we should consider that the
> > > same
> > > >>>> consumer instance may have many "first heartbeats", meaning
> > heartbeats
> > > >> to
> > > >>>> join the group when it's not part of it (ex. consumer unsubscribe
> +
> > > >>>> subscribe, fenced, stale). Is this the intention or are you
> > > considering
> > > >>> the
> > > >>>> lifetime differently? We should clarify it in the KIP.
> > > >>>>
> > > >>>> Thanks!
> > > >>>>
> > > >>>> Lianet
> > > >>>>
> > > >>>> On Tue, Aug 27, 2024 at 2:27 AM TengYao Chi <ki...@gmail.com>
> > > >> wrote:
> > > >>>>
> > > >>>>> Hi everyone,
> > > >>>>>
> > > >>>>> I have revised this KIP multiple times based on the feedback from
> > our
> > > >>>>> discussions.
> > > >>>>> I would greatly appreciate it if you could review it when you
> have
> > > >> the
> > > >>>>> time.
> > > >>>>> If there are no further comments or suggestions, I plan to
> proceed
> > > >> with
> > > >>>>> initiating a vote soon.
> > > >>>>>
> > > >>>>> Best regards,
> > > >>>>> TengYao
> > > >>>>>
> > > >>>>> TengYao Chi <ki...@gmail.com> 於 2024年8月23日 週五 下午2:43寫道：
> > > >>>>>
> > > >>>>>> Hi Andrew,
> > > >>>>>> Thank you for your previous feedback and insights.
> > > >>>>>> Your contributions have added valuable perspectives to the
> > > >>> discussions.
> > > >>>>>> And we also benefit from the comparison of different solutions.
> > > >>>>>> I’m also looking forward to seeing an initial version in
> KIP-932,
> > > >> as
> > > >>> it
> > > >>>>>> will provide a good reference for future implementations.
> > > >>>>>>
> > > >>>>>> Regarding your comment on AS2, I wanted to clarify that my
> > > >>>> specification
> > > >>>>>> references org.apache.kafka.common.Uuid.
> > > >>>>>> I believe we’re referring to the same class, and it might just
> be
> > a
> > > >>>> small
> > > >>>>>> oversight due to the busy schedule.
> > > >>>>>>
> > > >>>>>> I want to express my gratitude once again for your many
> insightful
> > > >>>>>> comments, which have helped the discussion progress smoothly.
> > > >>>>>>
> > > >>>>>> Best regards,
> > > >>>>>> TengYao
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> Andrew Schofield <an...@live.com> 於 2024年8月22日 週四
> > > >>>> 下午11:28寫道：
> > > >>>>>>
> > > >>>>>>> Hi TengYao,
> > > >>>>>>> I’ve been reading through the comments and I’m happy that the
> > > >> lobby
> > > >>>>>>> approach has not gained support.
> > > >>>>>>>
> > > >>>>>>> Assuming that this KIP is voted, I will be happy to change
> > KIP-932
> > > >>> so
> > > >>>>>>> that it only supports client-side member ID generation. Because
> > > >> that
> > > >>>> KIP
> > > >>>>>>> is still
> > > >>>>>>> under development, I can do this in the first version of
> > > >>>>>>> ShareGroupHeartbeat.
> > > >>>>>>>
> > > >>>>>>> AS2: For the encoding section, I suppose the specific encoding
> > > >> which
> > > >>>>>>> is used is what org.apache.kafka.utils.Uuid uses.
> > > >>>>>>>
> > > >>>>>>> Thanks,
> > > >>>>>>> Andrew
> > > >>>>>>>
> > > >>>>>>>> On 14 Aug 2024, at 17:00, TengYao Chi <ki...@gmail.com>
> > > >>> wrote:
> > > >>>>>>>>
> > > >>>>>>>> Hello Apoorv,
> > > >>>>>>>> Thank you for your feedback.
> > > >>>>>>>> Regarding the questions you raised, unfortunately, this KIP
> > > >> cannot
> > > >>>>>>>> guarantee the order of heartbeats. As with many classic
> > > >>> distributed
> > > >>>>>>> system
> > > >>>>>>>> challenges, what we can do is make our best effort to ensure
> > > >> that
> > > >>>>> there
> > > >>>>>>> are
> > > >>>>>>>> no idle members or stale assignments under normal
> circumstances.
> > > >>>>>>>>
> > > >>>>>>>> As for the lobby approach, I’m not a fan of it because it
> > > >> requires
> > > >>>>>>> adding a
> > > >>>>>>>> mechanism to maintain client state within the ConsumerGroup,
> > > >>> which,
> > > >>>> in
> > > >>>>>>> my
> > > >>>>>>>> view, resembles something like a two-phase commit. This would
> > > >>>>> introduce
> > > >>>>>>>> more complexity than the proposal in this KIP, which is
> > > >> something
> > > >>> we
> > > >>>>>>> want
> > > >>>>>>>> to avoid. KIP-848 aims to simplify the existing protocol, and
> > > >>> while
> > > >>>>> the
> > > >>>>>>>> lobby approach is a good one, I believe it is not the right
> fit
> > > >>> for
> > > >>>>> this
> > > >>>>>>>> particular situation.
> > > >>>>>>>>
> > > >>>>>>>> Best regards,
> > > >>>>>>>> TengYao
> > > >>>>>>>>
> > > >>>>>>>> TengYao Chi <ki...@gmail.com> 於 2024年8月14日 週三 下午11:45寫道：
> > > >>>>>>>>
> > > >>>>>>>>> Hi David,
> > > >>>>>>>>>
> > > >>>>>>>>> I really appreciate your review and suggestions. As I am
> still
> > > >>>>> gaining
> > > >>>>>>>>> experience in writing KIPs, your input has been incredibly
> > > >>>> helpful. I
> > > >>>>>>> am
> > > >>>>>>>>> currently applying your suggestions to the KIP and will
> > > >> complete
> > > >>> it
> > > >>>>> as
> > > >>>>>>> soon
> > > >>>>>>>>> as possible.
> > > >>>>>>>>> Regarding the UUID part, I think we haven’t reached a
> > > >> conclusion
> > > >>>>>>> yet.(So
> > > >>>>>>>>> far according to this thread)
> > > >>>>>>>>> However, I will review the current implementation in the
> Kafka
> > > >>>> `Uuid`
> > > >>>>>>>>> class and include a brief specification in the KIP.
> > > >>>>>>>>>
> > > >>>>>>>>> Once again, thank you so much for your help.
> > > >>>>>>>>>
> > > >>>>>>>>> Best regards,
> > > >>>>>>>>> TengYao
> > > >>>>>>>>>
> > > >>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月14日 週三
> 下午11:14寫道：
> > > >>>>>>>>>
> > > >>>>>>>>>> hi Apoorv
> > > >>>>>>>>>>
> > > >>>>>>>>>>> As the memberId is now known to the client, and client
> might
> > > >>> send
> > > >>>>> the
> > > >>>>>>>>>> leave
> > > >>>>>>>>>> group heartbeat on shutdown prior to receiving the initial
> > > >>>> heartbeat
> > > >>>>>>>>>> response. If that's true then how do we guarantee that the 2
> > > >>>>> requests
> > > >>>>>>> to
> > > >>>>>>>>>> join and leave will be processed in order, which could still
> > > >>> leave
> > > >>>>>>> stale
> > > >>>>>>>>>> members or throw unknown member id exceptions?
> > > >>>>>>>>>>
> > > >>>>>>>>>> This is definitely a good question. the short answer: no
> > > >>> guarantee
> > > >>>>> but
> > > >>>>>>>>>> best
> > > >>>>>>>>>> efforts
> > > >>>>>>>>>>
> > > >>>>>>>>>> Please notice the root cause is "we have no enough time to
> > > >> wait
> > > >>>>>>> member id
> > > >>>>>>>>>> (response) when closing consumer". Sadly, we can' guarantee
> > > >> the
> > > >>>>>>> request
> > > >>>>>>>>>> order due to the same reason.
> > > >>>>>>>>>>
> > > >>>>>>>>>> However, in contrast to previous behavior, there is one big
> > > >>>> benefit
> > > >>>>>>> of new
> > > >>>>>>>>>> approach - we can try STONITH because we know the member id
> > > >>>>>>>>>>
> > > >>>>>>>>>> Best,
> > > >>>>>>>>>> Chia-Ping
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> Apoorv Mittal <ap...@gmail.com> 於 2024年8月14日 週三
> > > >>>> 下午8:55寫道：
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Hi TengYao,
> > > >>>>>>>>>>> Thanks for the KIP. Continuing on the point which Andrew
> > > >>>> mentioned
> > > >>>>> as
> > > >>>>>>>>>> AS1.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> As the memberId is now known to the client, and client
> might
> > > >>> send
> > > >>>>> the
> > > >>>>>>>>>> leave
> > > >>>>>>>>>>> group heartbeat on shutdown prior to receiving the initial
> > > >>>>> heartbeat
> > > >>>>>>>>>>> response. If that's true then how do we guarantee that the
> 2
> > > >>>>>>> requests to
> > > >>>>>>>>>>> join and leave will be processed in order, which could
> still
> > > >>>> leave
> > > >>>>>>> stale
> > > >>>>>>>>>>> members or throw unknown member id exceptions?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Though the client side member id generation is helpful
> which
> > > >>> will
> > > >>>>>>>>>> represent
> > > >>>>>>>>>>> the same group perspective as from client and broker's end.
> > > >>> But I
> > > >>>>>>> think
> > > >>>>>>>>>> the
> > > >>>>>>>>>>> major concern we want to solve here is Stale Partition
> > > >>>> Assignments
> > > >>>>>>> which
> > > >>>>>>>>>>> might still exist with the new approach. I am leaning
> towards
> > > >>> the
> > > >>>>>>>>>>> suggestion mentioned by Andrew where partition assignment
> > > >>>> triggers
> > > >>>>> on
> > > >>>>>>>>>>> subsequent heartbeat when client acknowledges the initial
> > > >>>>> heartbeat,
> > > >>>>>>>>>>> delayed partition assignment.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Though on a separate note, I have a different question.
> What
> > > >>>>> happens
> > > >>>>>>>>>> when
> > > >>>>>>>>>>> there is an issue with the client which sends the initial
> > > >>>> heartbeat
> > > >>>>>>>>>> without
> > > >>>>>>>>>>> memberId, then crashes and restarts? I think we must be
> > > >>>>> re-triggering
> > > >>>>>>>>>>> assignments and expiring members only after the heartbeat
> > > >>> session
> > > >>>>>>>>>> timeout?
> > > >>>>>>>>>>> If that's true then shall delayed partition assignment can
> > > >> help
> > > >>>>>>> benefit
> > > >>>>>>>>>> us
> > > >>>>>>>>>>> from this situation as well?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Regards,
> > > >>>>>>>>>>> Apoorv Mittal
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Wed, Aug 14, 2024 at 12:51 PM David Jacot
> > > >>>>>>>>>> <dj...@confluent.io.invalid>
> > > >>>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Hi Andrew,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Personally, I don't like the lobby approach. It makes
> things
> > > >>>> more
> > > >>>>>>>>>>>> complicated and it would require changing the records on
> the
> > > >>>>> server
> > > >>>>>>>>>> too.
> > > >>>>>>>>>>>> This is why I initially suggested the rejected alternative
> > > >> #2
> > > >>>>> which
> > > >>>>>>> is
> > > >>>>>>>>>>>> pretty close but also not perfect.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I'd like to clarify one thing. The ConsumerGroupHeartbeat
> > > >> API
> > > >>>>>>> already
> > > >>>>>>>>>>>> supports generating the member id on the client so we
> don't
> > > >>> need
> > > >>>>> any
> > > >>>>>>>>>>>> conditional logic on the client side. This is actually
> what
> > > >> we
> > > >>>>>>> wanted
> > > >>>>>>>>>> to
> > > >>>>>>>>>>> do
> > > >>>>>>>>>>>> in the first place but the idea got pushed back by Magnus
> > > >> back
> > > >>>>> then
> > > >>>>>>>>>>> because
> > > >>>>>>>>>>>> generating uuid from librdkafka required a new dependency.
> > > >> It
> > > >>>>> turns
> > > >>>>>>>>>> out
> > > >>>>>>>>>>>> that librdkafka has that dependency today. In retrospect,
> we
> > > >>>>> should
> > > >>>>>>>>>> have
> > > >>>>>>>>>>>> pushed back on this. Long story short, we can just do it.
> > > >> The
> > > >>>>>>>>>> proposal in
> > > >>>>>>>>>>>> this KIP is to make the member id required in future
> > > >> versions.
> > > >>>> We
> > > >>>>>>>>>> could
> > > >>>>>>>>>>>> also decide not to do it and to keep supporting both
> > > >>>> approaches. I
> > > >>>>>>>>>> would
> > > >>>>>>>>>>>> also be fine with this.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Best,
> > > >>>>>>>>>>>> David
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Wed, Aug 14, 2024 at 12:30 PM Andrew Schofield <
> > > >>>>>>>>>>>> andrew_schofield@live.com>
> > > >>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> Hi TengYao,
> > > >>>>>>>>>>>>> Thanks for your response. I’ll have just one more try to
> > > >>>>> persuade.
> > > >>>>>>>>>>>>> I feel that I will need to follow the approach with
> KIP-932
> > > >>>> when
> > > >>>>>>>>>> we’ve
> > > >>>>>>>>>>>>> made a decision, so I do have more than a passing
> interest
> > > >> in
> > > >>>>> this.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> A group member in the lobby is in the group, but it does
> > > >> not
> > > >>>> have
> > > >>>>>>>>>> any
> > > >>>>>>>>>>>>> assignments. A member of a consumer group can have no
> > > >>> assigned
> > > >>>>>>>>>>>>> partitions (such as 5 CG members subscribed to a topic
> > > >> with 4
> > > >>>>>>>>>>>> partitions),
> > > >>>>>>>>>>>>> so it’s a situation that consumer group members already
> > > >>> expect.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> One of Kafka’s strengths is the way that we handle API
> > > >>>>> versioning.
> > > >>>>>>>>>>>>> But, there is a cost - the behaviour is different
> depending
> > > >>> on
> > > >>>>> the
> > > >>>>>>>>>> RPC
> > > >>>>>>>>>>>>> version. KIP-848 is on the cusp of completion, but we’re
> > > >>>> already
> > > >>>>>>>>>> adding
> > > >>>>>>>>>>>>> conditional logic for v0/v1 for ConsumerGroupHeartbeat.
> > > >>> That’s
> > > >>>> a
> > > >>>>>>>>>> pity.
> > > >>>>>>>>>>>>> Only a minor issue, but it’s unfortunate.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>> Andrew
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On 14 Aug 2024, at 08:47, TengYao Chi <
> > > >> kitingiao@gmail.com>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Hello Andrew
> > > >>>>>>>>>>>>>> Thank you for your thoughtful suggestions and getting
> the
> > > >>>>>>>>>> discussion
> > > >>>>>>>>>>>>> going.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> To AS1:
> > > >>>>>>>>>>>>>> In the current scenario where the server generates the
> > > >> UUID,
> > > >>>> if
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>> client
> > > >>>>>>>>>>>>>> shuts down before receiving the memberId generated by
> the
> > > >> GC
> > > >>>>>>>>>>>> (regardless
> > > >>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>> whether it’s a graceful shutdown or not), the GC will
> > > >> still
> > > >>>> have
> > > >>>>>>>>>> to
> > > >>>>>>>>>>>> wait
> > > >>>>>>>>>>>>>> for the heartbeat timeout because the client doesn’t
> know
> > > >>> its
> > > >>>>>>>>>>> memberId.
> > > >>>>>>>>>>>>>> This KIP indeed cannot completely resolve the
> idempotency
> > > >>>> issue,
> > > >>>>>>>>>> but
> > > >>>>>>>>>>> it
> > > >>>>>>>>>>>>> can
> > > >>>>>>>>>>>>>> better handle shutdown scenarios under normal
> > > >> circumstances
> > > >>>>>>>>>> because
> > > >>>>>>>>>>> the
> > > >>>>>>>>>>>>>> client always knows its memberId. Even if the client
> shuts
> > > >>>> down
> > > >>>>>>>>>>>>> immediately
> > > >>>>>>>>>>>>>> after the initial heartbeat, as long as it performs a
> > > >>> graceful
> > > >>>>>>>>>>> shutdown
> > > >>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>> sends a leave heartbeat, the GC can manage the situation
> > > >> and
> > > >>>>>>>>>> remove
> > > >>>>>>>>>>> the
> > > >>>>>>>>>>>>>> member. Therefore, the goal of this KIP is to address
> the
> > > >>>> issue
> > > >>>>>>>>>> where
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>> GC has to wait for the heartbeat timeout due to the
> client
> > > >>>>> leaving
> > > >>>>>>>>>>>>> without
> > > >>>>>>>>>>>>>> knowing its memberId, which leads to reduced throughput
> > > >> and
> > > >>>>>>>>>> limited
> > > >>>>>>>>>>>>>> scalability.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> The solution you suggest has also been proposed by
> David.
> > > >>> The
> > > >>>>>>>>>> concern
> > > >>>>>>>>>>>>> with
> > > >>>>>>>>>>>>>> this approach is that it introduces additional
> complexity
> > > >>> for
> > > >>>>>>>>>>>>>> compatibility, as the new server would not immediately
> add
> > > >>> the
> > > >>>>>>>>>> member
> > > >>>>>>>>>>>> to
> > > >>>>>>>>>>>>>> the group, while the old server would. This requires
> > > >> clients
> > > >>>> to
> > > >>>>>>>>>>>>>> differentiate whether their memberId has been added to
> the
> > > >>>> group
> > > >>>>>>>>>> or
> > > >>>>>>>>>>>> not,
> > > >>>>>>>>>>>>>> which could result in unexpected logs.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>> TengYao
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Andrew Schofield <an...@live.com> 於
> 2024年8月14日
> > > >>> 週三
> > > >>>>>>>>>>>> 上午12:29寫道：
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Hi TengYao,
> > > >>>>>>>>>>>>>>> Thanks for the KIP. I wonder if there’s a different way
> > > >> to
> > > >>>>> close
> > > >>>>>>>>>>> what
> > > >>>>>>>>>>>>>>> is quite a small window.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> AS1: It is true that the initial heartbeat is not
> > > >>> idempotent,
> > > >>>>> but
> > > >>>>>>>>>>> this
> > > >>>>>>>>>>>>>>> remains
> > > >>>>>>>>>>>>>>> true with this KIP. It’s just differently not
> idempotent.
> > > >>> If
> > > >>>>> the
> > > >>>>>>>>>>>> client
> > > >>>>>>>>>>>>>>> makes its
> > > >>>>>>>>>>>>>>> own member ID, sends a request and dies, the GC will
> > > >> still
> > > >>>> have
> > > >>>>>>>>>>> added
> > > >>>>>>>>>>>>>>> the member to the group and it will hang around until
> the
> > > >>>>> session
> > > >>>>>>>>>>>>> expires.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> I wonder if the GC could still generate the member ID
> in
> > > >>>>>>>>>> response to
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>> first
> > > >>>>>>>>>>>>>>> heartbeat, and put the member in a special PENDING
> state
> > > >>> with
> > > >>>>> no
> > > >>>>>>>>>>>>>>> assignments until the client sends the next heartbeat,
> > > >> thus
> > > >>>>>>>>>>> confirming
> > > >>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>> has received the member ID. This would not be a
> protocol
> > > >>>> change
> > > >>>>>>>>>> at
> > > >>>>>>>>>>>> all,
> > > >>>>>>>>>>>>>>> just
> > > >>>>>>>>>>>>>>> a change to the GC to keep a new member in the lobby
> > > >> until
> > > >>>> it’s
> > > >>>>>>>>>>>>> comfirmed
> > > >>>>>>>>>>>>>>> it knows its member ID.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>> Andrew
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On 13 Aug 2024, at 15:59, TengYao Chi <
> > > >>> kitingiao@gmail.com>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Hi Chia-Ping,
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Thanks for review and suggestions.
> > > >>>>>>>>>>>>>>>> I have updated the content of KIP accordingly.
> > > >>>>>>>>>>>>>>>> Please take a look.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Best regards,
> > > >>>>>>>>>>>>>>>> TengYao
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2024年8月13日 週二
> > > >>>>> 下午9:45寫道：
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> hi TengYao
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> thanks for this KIP.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> 1) could you please describe the before/after
> behavior
> > > >> in
> > > >>>> the
> > > >>>>>>>>>>>>> "Proposed
> > > >>>>>>>>>>>>>>>>> Changes" section? IIRC, current RPC allows HB having
> > > >>> member
> > > >>>>> id
> > > >>>>>>>>>>>>>>> generated by
> > > >>>>>>>>>>>>>>>>> client, right? If HB has no member ID, server will
> > > >>> generate
> > > >>>>> one
> > > >>>>>>>>>>> and
> > > >>>>>>>>>>>>> then
> > > >>>>>>>>>>>>>>>>> return. The new behavior will enforce HB "must" have
> > > >>> member
> > > >>>>> ID.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> 2) could you please write the version number
> explicitly
> > > >>> in
> > > >>>>> the
> > > >>>>>>>>>> KIP
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> 3) how new client code handle the old HB? Does it
> > > >> always
> > > >>>>>>>>>> generate
> > > >>>>>>>>>>>>> member
> > > >>>>>>>>>>>>>>>>> ID on client-side even though that is not restricted?
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>> Chia-Ping
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> On 2024/08/13 06:20:42 TengYao Chi wrote:
> > > >>>>>>>>>>>>>>>>>> Hello everyone,
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> I would like to start a discussion thread on
> KIP-1082,
> > > >>>> which
> > > >>>>>>>>>>>> proposes
> > > >>>>>>>>>>>>>>>>>> enabling id generation for clients over the
> > > >>>>>>>>>>> ConsumerGroupHeartbeat
> > > >>>>>>>>>>>>> RPC.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Here is the KIP Link: KIP-1082
> > > >>>>>>>>>>>>>>>>>> <
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Please take a look and let me know what you think,
> > > >> and I
> > > >>>>> would
> > > >>>>>>>>>>>>>>> appreciate
> > > >>>>>>>>>>>>>>>>>> any suggestions and feedback.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Best regards,
> > > >>>>>>>>>>>>>>>>>> TengYao
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> > >
> >
>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[David Jacot <dj...@confluent.io.INVALID>]

Hi TengYao,

KT2: I don't think that we can realistically validate the uuid on the
server. It is basically a string of chars. So I lean towards having a good
recommendation in the KIP and in the document of the field in the RPC's
definition.

Best,
David

On Fri, Aug 30, 2024 at 3:02 PM TengYao Chi <ki...@gmail.com> wrote:

> Hello Kirk !
>
> Thank you for your comments !
>
> KT1: Yes, you are correct. The issue is not unique to the initial
> heartbeat; there can always be cases where the broker might lose connection
> with a member.
>
> KT2: Currently, if the client doesn't have a member ID and the memberEpoch
> equals 0, the coordinator will generate a UUID as the member ID for the
> client. However, at the RPC level, the member ID is sent as a literal
> string, meaning there are no restrictions on the format at this level.
> This also reminds me that we haven't reached a final conclusion on how to
> enforce the use of UUIDs.
> From our previous discussions, I recall two possible approaches:
> The first is to validate the UUID on the server side, and if it's not
> valid, throw an exception to the client.
> The second is to document a recommendation for clients to use UUIDs as
> member IDs, without strictly enforcing it.
> I think it's time to decide on the approach we want to take.
>
> KT3: Yes, "session" can be considered synonymous with "membership" in this
> context.
>
> KT4: Thank you for pointing that out. I will update the wording to
> specifically say this behavior is for consumers.
>
> Thanks again for your comments.
>
> Best regards,
> TengYao
>
> Kirk True <ki...@kirktrue.pro> 於 2024年8月30日 週五 上午12:39寫道：
>
> > Hi TengYao!
> >
> > Sorry for being late to the discussion...
> >
> > After reading the thread and then the KIP, I had a few
> questions/comments:
> >
> > KT1: In Motivation, it states: "This scenario can result in the broker
> > registering a new member for which it will never receive a proper leave
> > request.” Just to be clear, the broker will always have cases where it
> > might lose connection with a member. That’s not unique to the initial
> > heartbeat, right?
> >
> > KT2: There was a bit of back and forth about format of the member ID.
> From
> > what I gathered in the thread, the member ID is still defined in the RPC
> as
> > a string and not a UUID, right? The KIP states that the “client must
> > generate a UUID as the member ID” and that the “server will validate
> that a
> > valid UUID is provided.” Is that a change for the server, or is it
> already
> > enforced as a UUID?
> >
> > KT3: Lianet mentioned some confusion over the use of the word “session.”
> > Isn’t “session” synonymous with “membership?”
> >
> > KT4: Under “Member ID Lifecycle,” it states: "The client should reuse the
> > same UUID as the member ID for all heartbeats and rejoin attempts to
> > maintain continuity within the group.” Could we change the first part of
> > that to “The Consumer instance should…” We do have lifetimes that extend
> > past the lifetime of a client instance (such as the transaction ID).
> >
> > Thanks,
> > Kirk
> >
> > > On Aug 29, 2024, at 1:28 AM, TengYao Chi <ki...@gmail.com> wrote:
> > >
> > > Hi David,
> > >
> > > Thank you for pointing that out.
> > > I have updated the content of the KIP based on Lianet's and your
> > feedback.
> > > Please take a look and let me know your thoughts.
> > >
> > > Best regards,
> > > TengYao
> > >
> > > David Jacot <dj...@confluent.io.invalid> 於 2024年8月29日 週四 下午3:20寫道：
> > >
> > >> Hi TengYao,
> > >>
> > >> Thanks for the update. I haven't fully read it yet but I will soon.
> > >>
> > >> LM4: This is incorrect. The consumer must keep its member id during
> its
> > >> entire lifetime (until the process stops or dies). The protocol
> > stipulates
> > >> that a member must rejoin with the same member id and the member epoch
> > set
> > >> to zero when an FENCED_MEMBER_EPOCH occurs. This allows the member to
> > >> resynchronize itself. We should not change this behavior. I think that
> > we
> > >> should see the client side generation id as an incarnation id of the
> > >> application. It is generated once and kept until it stops or dies.
> > >>
> > >> Best,
> > >> David
> > >>
> > >> On Thu, Aug 29, 2024 at 6:21 AM TengYao Chi <ki...@gmail.com>
> > wrote:
> > >>
> > >>> Hello Lianet !
> > >>>
> > >>> Thanks for the reviews and suggestions!
> > >>>
> > >>> LM1: Indeed, we plan to enforce client-side ID generation in the
> > future,
> > >>> and it is not an alternative. I will change the title accordingly.
> > >>>
> > >>> LM2: Yes, that's the expectation. I will add that statement to the
> > public
> > >>> interface section.
> > >>>
> > >>> LM3: Thank you for the high-level perspective review. I think you're
> > >> right;
> > >>> our intention isn't very clear since it was placed at the end of the
> > >>> section. I will try to rephrase that section to make it more obvious.
> > >>>
> > >>> LM4: Regarding the definition of "session" in this KIP, I believe it
> > >> refers
> > >>> to the period between the *first-time heartbeat* and when the
> *consumer
> > >>> leaves the group* (whether through a graceful shutdown or a heartbeat
> > >>> timeout). The consumer should reuse its UUID if it has been generated
> > >>> before. The only situation in which it will regenerate the UUID is if
> > the
> > >>> coordinator finds that there is already a consumer with the same
> UUID.
> > >>> IIRC, the coordinator should compare the member epochs, and the
> > >>> later-joined consumer should be fenced off by the coordinator due to
> > >> having
> > >>> a lower member epoch. Once the consumer receives a
> > `FENCED_MEMBER_EPOCH`
> > >>> error, it will generate a new UUID and attempt to rejoin. I will
> > clarify
> > >>> this in the KIP.
> > >>>
> > >>> Thanks again for your reviews, I really appreciate it.
> > >>>
> > >>> Best regards,
> > >>> TengYao
> > >>>
> > >>> Lianet M. <li...@gmail.com> 於 2024年8月28日 週三 下午7:12寫道：
> > >>>
> > >>>> Hello TengYao! Thanks for taking on this issue, we've been going
> > around
> > >>> it
> > >>>> for a while.
> > >>>>
> > >>>> LM1: About the title of the KIP: "Enable ID Generation for Clients
> > over
> > >>> the
> > >>>> ConsumerGroupHeartbeat RPC". I find it confusing because it hints
> that
> > >>>> we're adding it as an alternative (which was discussed and
> discarded,
> > >> in
> > >>>> favour of really enforcing it). It's also missing the core change
> imo,
> > >>>> which is "where" the generation happens. So, maybe more to the point
> > >> with
> > >>>> something along the lines of "Client-side generated ID for clients
> > over
> > >>>> ConsumerGroupHeartbeat RPC"?
> > >>>>
> > >>>> LM2: On the public interfaces section, the KIP states that "the
> server
> > >>> will
> > >>>> reject the request", but we should agree on the specific error
> type. I
> > >>>> expect it should fail with an InvalidRequestException, is that the
> > >>>> intention? (This was also suggested in the discussion thread before
> > but
> > >>> is
> > >>>> not in the KIP).
> > >>>>
> > >>>> LM3. Related to my previous point, I find that to be the true
> > >>> public-facing
> > >>>> change (member ID mandatory at the protocol level), but it's only at
> > >> the
> > >>>> end of the Public interfaces changes, kind of lost among details of
> > how
> > >>>> we're going to do it. Should we rephrase that section with the
> actual
> > >>>> change first, and the hows after (ex. Bumping the version is not the
> > >>>> public-facing change in this case, it's just the mechanism to
> properly
> > >>>> introduce our change)
> > >>>>
> > >>>> LM4. Regarding the lifetime of the UUID: the KIP states we will
> > "Verify
> > >>>> that the UUID remains consistent across all subsequent heartbeats
> > >> during
> > >>>> the session". What is this "session" referring to here? I would
> expect
> > >>> that
> > >>>> the UUID is associated to a consumer instance (generated for the
> > >> consumer
> > >>>> the first time it needs to send a HB if it doesn't have the UUID
> yet.
> > >>> From
> > >>>> there on, every time it needs to send a "first HB" again, it will
> > reuse
> > >>> its
> > >>>> UUID, is that the intention? Note that we should consider that the
> > same
> > >>>> consumer instance may have many "first heartbeats", meaning
> heartbeats
> > >> to
> > >>>> join the group when it's not part of it (ex. consumer unsubscribe +
> > >>>> subscribe, fenced, stale). Is this the intention or are you
> > considering
> > >>> the
> > >>>> lifetime differently? We should clarify it in the KIP.
> > >>>>
> > >>>> Thanks!
> > >>>>
> > >>>> Lianet
> > >>>>
> > >>>> On Tue, Aug 27, 2024 at 2:27 AM TengYao Chi <ki...@gmail.com>
> > >> wrote:
> > >>>>
> > >>>>> Hi everyone,
> > >>>>>
> > >>>>> I have revised this KIP multiple times based on the feedback from
> our
> > >>>>> discussions.
> > >>>>> I would greatly appreciate it if you could review it when you have
> > >> the
> > >>>>> time.
> > >>>>> If there are no further comments or suggestions, I plan to proceed
> > >> with
> > >>>>> initiating a vote soon.
> > >>>>>
> > >>>>> Best regards,
> > >>>>> TengYao
> > >>>>>
> > >>>>> TengYao Chi <ki...@gmail.com> 於 2024年8月23日 週五 下午2:43寫道：
> > >>>>>
> > >>>>>> Hi Andrew,
> > >>>>>> Thank you for your previous feedback and insights.
> > >>>>>> Your contributions have added valuable perspectives to the
> > >>> discussions.
> > >>>>>> And we also benefit from the comparison of different solutions.
> > >>>>>> I’m also looking forward to seeing an initial version in KIP-932,
> > >> as
> > >>> it
> > >>>>>> will provide a good reference for future implementations.
> > >>>>>>
> > >>>>>> Regarding your comment on AS2, I wanted to clarify that my
> > >>>> specification
> > >>>>>> references org.apache.kafka.common.Uuid.
> > >>>>>> I believe we’re referring to the same class, and it might just be
> a
> > >>>> small
> > >>>>>> oversight due to the busy schedule.
> > >>>>>>
> > >>>>>> I want to express my gratitude once again for your many insightful
> > >>>>>> comments, which have helped the discussion progress smoothly.
> > >>>>>>
> > >>>>>> Best regards,
> > >>>>>> TengYao
> > >>>>>>
> > >>>>>>
> > >>>>>> Andrew Schofield <an...@live.com> 於 2024年8月22日 週四
> > >>>> 下午11:28寫道：
> > >>>>>>
> > >>>>>>> Hi TengYao,
> > >>>>>>> I’ve been reading through the comments and I’m happy that the
> > >> lobby
> > >>>>>>> approach has not gained support.
> > >>>>>>>
> > >>>>>>> Assuming that this KIP is voted, I will be happy to change
> KIP-932
> > >>> so
> > >>>>>>> that it only supports client-side member ID generation. Because
> > >> that
> > >>>> KIP
> > >>>>>>> is still
> > >>>>>>> under development, I can do this in the first version of
> > >>>>>>> ShareGroupHeartbeat.
> > >>>>>>>
> > >>>>>>> AS2: For the encoding section, I suppose the specific encoding
> > >> which
> > >>>>>>> is used is what org.apache.kafka.utils.Uuid uses.
> > >>>>>>>
> > >>>>>>> Thanks,
> > >>>>>>> Andrew
> > >>>>>>>
> > >>>>>>>> On 14 Aug 2024, at 17:00, TengYao Chi <ki...@gmail.com>
> > >>> wrote:
> > >>>>>>>>
> > >>>>>>>> Hello Apoorv,
> > >>>>>>>> Thank you for your feedback.
> > >>>>>>>> Regarding the questions you raised, unfortunately, this KIP
> > >> cannot
> > >>>>>>>> guarantee the order of heartbeats. As with many classic
> > >>> distributed
> > >>>>>>> system
> > >>>>>>>> challenges, what we can do is make our best effort to ensure
> > >> that
> > >>>>> there
> > >>>>>>> are
> > >>>>>>>> no idle members or stale assignments under normal circumstances.
> > >>>>>>>>
> > >>>>>>>> As for the lobby approach, I’m not a fan of it because it
> > >> requires
> > >>>>>>> adding a
> > >>>>>>>> mechanism to maintain client state within the ConsumerGroup,
> > >>> which,
> > >>>> in
> > >>>>>>> my
> > >>>>>>>> view, resembles something like a two-phase commit. This would
> > >>>>> introduce
> > >>>>>>>> more complexity than the proposal in this KIP, which is
> > >> something
> > >>> we
> > >>>>>>> want
> > >>>>>>>> to avoid. KIP-848 aims to simplify the existing protocol, and
> > >>> while
> > >>>>> the
> > >>>>>>>> lobby approach is a good one, I believe it is not the right fit
> > >>> for
> > >>>>> this
> > >>>>>>>> particular situation.
> > >>>>>>>>
> > >>>>>>>> Best regards,
> > >>>>>>>> TengYao
> > >>>>>>>>
> > >>>>>>>> TengYao Chi <ki...@gmail.com> 於 2024年8月14日 週三 下午11:45寫道：
> > >>>>>>>>
> > >>>>>>>>> Hi David,
> > >>>>>>>>>
> > >>>>>>>>> I really appreciate your review and suggestions. As I am still
> > >>>>> gaining
> > >>>>>>>>> experience in writing KIPs, your input has been incredibly
> > >>>> helpful. I
> > >>>>>>> am
> > >>>>>>>>> currently applying your suggestions to the KIP and will
> > >> complete
> > >>> it
> > >>>>> as
> > >>>>>>> soon
> > >>>>>>>>> as possible.
> > >>>>>>>>> Regarding the UUID part, I think we haven’t reached a
> > >> conclusion
> > >>>>>>> yet.(So
> > >>>>>>>>> far according to this thread)
> > >>>>>>>>> However, I will review the current implementation in the Kafka
> > >>>> `Uuid`
> > >>>>>>>>> class and include a brief specification in the KIP.
> > >>>>>>>>>
> > >>>>>>>>> Once again, thank you so much for your help.
> > >>>>>>>>>
> > >>>>>>>>> Best regards,
> > >>>>>>>>> TengYao
> > >>>>>>>>>
> > >>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月14日 週三 下午11:14寫道：
> > >>>>>>>>>
> > >>>>>>>>>> hi Apoorv
> > >>>>>>>>>>
> > >>>>>>>>>>> As the memberId is now known to the client, and client might
> > >>> send
> > >>>>> the
> > >>>>>>>>>> leave
> > >>>>>>>>>> group heartbeat on shutdown prior to receiving the initial
> > >>>> heartbeat
> > >>>>>>>>>> response. If that's true then how do we guarantee that the 2
> > >>>>> requests
> > >>>>>>> to
> > >>>>>>>>>> join and leave will be processed in order, which could still
> > >>> leave
> > >>>>>>> stale
> > >>>>>>>>>> members or throw unknown member id exceptions?
> > >>>>>>>>>>
> > >>>>>>>>>> This is definitely a good question. the short answer: no
> > >>> guarantee
> > >>>>> but
> > >>>>>>>>>> best
> > >>>>>>>>>> efforts
> > >>>>>>>>>>
> > >>>>>>>>>> Please notice the root cause is "we have no enough time to
> > >> wait
> > >>>>>>> member id
> > >>>>>>>>>> (response) when closing consumer". Sadly, we can' guarantee
> > >> the
> > >>>>>>> request
> > >>>>>>>>>> order due to the same reason.
> > >>>>>>>>>>
> > >>>>>>>>>> However, in contrast to previous behavior, there is one big
> > >>>> benefit
> > >>>>>>> of new
> > >>>>>>>>>> approach - we can try STONITH because we know the member id
> > >>>>>>>>>>
> > >>>>>>>>>> Best,
> > >>>>>>>>>> Chia-Ping
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Apoorv Mittal <ap...@gmail.com> 於 2024年8月14日 週三
> > >>>> 下午8:55寫道：
> > >>>>>>>>>>
> > >>>>>>>>>>> Hi TengYao,
> > >>>>>>>>>>> Thanks for the KIP. Continuing on the point which Andrew
> > >>>> mentioned
> > >>>>> as
> > >>>>>>>>>> AS1.
> > >>>>>>>>>>>
> > >>>>>>>>>>> As the memberId is now known to the client, and client might
> > >>> send
> > >>>>> the
> > >>>>>>>>>> leave
> > >>>>>>>>>>> group heartbeat on shutdown prior to receiving the initial
> > >>>>> heartbeat
> > >>>>>>>>>>> response. If that's true then how do we guarantee that the 2
> > >>>>>>> requests to
> > >>>>>>>>>>> join and leave will be processed in order, which could still
> > >>>> leave
> > >>>>>>> stale
> > >>>>>>>>>>> members or throw unknown member id exceptions?
> > >>>>>>>>>>>
> > >>>>>>>>>>> Though the client side member id generation is helpful which
> > >>> will
> > >>>>>>>>>> represent
> > >>>>>>>>>>> the same group perspective as from client and broker's end.
> > >>> But I
> > >>>>>>> think
> > >>>>>>>>>> the
> > >>>>>>>>>>> major concern we want to solve here is Stale Partition
> > >>>> Assignments
> > >>>>>>> which
> > >>>>>>>>>>> might still exist with the new approach. I am leaning towards
> > >>> the
> > >>>>>>>>>>> suggestion mentioned by Andrew where partition assignment
> > >>>> triggers
> > >>>>> on
> > >>>>>>>>>>> subsequent heartbeat when client acknowledges the initial
> > >>>>> heartbeat,
> > >>>>>>>>>>> delayed partition assignment.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Though on a separate note, I have a different question. What
> > >>>>> happens
> > >>>>>>>>>> when
> > >>>>>>>>>>> there is an issue with the client which sends the initial
> > >>>> heartbeat
> > >>>>>>>>>> without
> > >>>>>>>>>>> memberId, then crashes and restarts? I think we must be
> > >>>>> re-triggering
> > >>>>>>>>>>> assignments and expiring members only after the heartbeat
> > >>> session
> > >>>>>>>>>> timeout?
> > >>>>>>>>>>> If that's true then shall delayed partition assignment can
> > >> help
> > >>>>>>> benefit
> > >>>>>>>>>> us
> > >>>>>>>>>>> from this situation as well?
> > >>>>>>>>>>>
> > >>>>>>>>>>> Regards,
> > >>>>>>>>>>> Apoorv Mittal
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Wed, Aug 14, 2024 at 12:51 PM David Jacot
> > >>>>>>>>>> <dj...@confluent.io.invalid>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hi Andrew,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Personally, I don't like the lobby approach. It makes things
> > >>>> more
> > >>>>>>>>>>>> complicated and it would require changing the records on the
> > >>>>> server
> > >>>>>>>>>> too.
> > >>>>>>>>>>>> This is why I initially suggested the rejected alternative
> > >> #2
> > >>>>> which
> > >>>>>>> is
> > >>>>>>>>>>>> pretty close but also not perfect.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I'd like to clarify one thing. The ConsumerGroupHeartbeat
> > >> API
> > >>>>>>> already
> > >>>>>>>>>>>> supports generating the member id on the client so we don't
> > >>> need
> > >>>>> any
> > >>>>>>>>>>>> conditional logic on the client side. This is actually what
> > >> we
> > >>>>>>> wanted
> > >>>>>>>>>> to
> > >>>>>>>>>>> do
> > >>>>>>>>>>>> in the first place but the idea got pushed back by Magnus
> > >> back
> > >>>>> then
> > >>>>>>>>>>> because
> > >>>>>>>>>>>> generating uuid from librdkafka required a new dependency.
> > >> It
> > >>>>> turns
> > >>>>>>>>>> out
> > >>>>>>>>>>>> that librdkafka has that dependency today. In retrospect, we
> > >>>>> should
> > >>>>>>>>>> have
> > >>>>>>>>>>>> pushed back on this. Long story short, we can just do it.
> > >> The
> > >>>>>>>>>> proposal in
> > >>>>>>>>>>>> this KIP is to make the member id required in future
> > >> versions.
> > >>>> We
> > >>>>>>>>>> could
> > >>>>>>>>>>>> also decide not to do it and to keep supporting both
> > >>>> approaches. I
> > >>>>>>>>>> would
> > >>>>>>>>>>>> also be fine with this.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Best,
> > >>>>>>>>>>>> David
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Wed, Aug 14, 2024 at 12:30 PM Andrew Schofield <
> > >>>>>>>>>>>> andrew_schofield@live.com>
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi TengYao,
> > >>>>>>>>>>>>> Thanks for your response. I’ll have just one more try to
> > >>>>> persuade.
> > >>>>>>>>>>>>> I feel that I will need to follow the approach with KIP-932
> > >>>> when
> > >>>>>>>>>> we’ve
> > >>>>>>>>>>>>> made a decision, so I do have more than a passing interest
> > >> in
> > >>>>> this.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> A group member in the lobby is in the group, but it does
> > >> not
> > >>>> have
> > >>>>>>>>>> any
> > >>>>>>>>>>>>> assignments. A member of a consumer group can have no
> > >>> assigned
> > >>>>>>>>>>>>> partitions (such as 5 CG members subscribed to a topic
> > >> with 4
> > >>>>>>>>>>>> partitions),
> > >>>>>>>>>>>>> so it’s a situation that consumer group members already
> > >>> expect.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> One of Kafka’s strengths is the way that we handle API
> > >>>>> versioning.
> > >>>>>>>>>>>>> But, there is a cost - the behaviour is different depending
> > >>> on
> > >>>>> the
> > >>>>>>>>>> RPC
> > >>>>>>>>>>>>> version. KIP-848 is on the cusp of completion, but we’re
> > >>>> already
> > >>>>>>>>>> adding
> > >>>>>>>>>>>>> conditional logic for v0/v1 for ConsumerGroupHeartbeat.
> > >>> That’s
> > >>>> a
> > >>>>>>>>>> pity.
> > >>>>>>>>>>>>> Only a minor issue, but it’s unfortunate.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>> Andrew
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On 14 Aug 2024, at 08:47, TengYao Chi <
> > >> kitingiao@gmail.com>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hello Andrew
> > >>>>>>>>>>>>>> Thank you for your thoughtful suggestions and getting the
> > >>>>>>>>>> discussion
> > >>>>>>>>>>>>> going.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> To AS1:
> > >>>>>>>>>>>>>> In the current scenario where the server generates the
> > >> UUID,
> > >>>> if
> > >>>>>>>>>> the
> > >>>>>>>>>>>>> client
> > >>>>>>>>>>>>>> shuts down before receiving the memberId generated by the
> > >> GC
> > >>>>>>>>>>>> (regardless
> > >>>>>>>>>>>>> of
> > >>>>>>>>>>>>>> whether it’s a graceful shutdown or not), the GC will
> > >> still
> > >>>> have
> > >>>>>>>>>> to
> > >>>>>>>>>>>> wait
> > >>>>>>>>>>>>>> for the heartbeat timeout because the client doesn’t know
> > >>> its
> > >>>>>>>>>>> memberId.
> > >>>>>>>>>>>>>> This KIP indeed cannot completely resolve the idempotency
> > >>>> issue,
> > >>>>>>>>>> but
> > >>>>>>>>>>> it
> > >>>>>>>>>>>>> can
> > >>>>>>>>>>>>>> better handle shutdown scenarios under normal
> > >> circumstances
> > >>>>>>>>>> because
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>> client always knows its memberId. Even if the client shuts
> > >>>> down
> > >>>>>>>>>>>>> immediately
> > >>>>>>>>>>>>>> after the initial heartbeat, as long as it performs a
> > >>> graceful
> > >>>>>>>>>>> shutdown
> > >>>>>>>>>>>>> and
> > >>>>>>>>>>>>>> sends a leave heartbeat, the GC can manage the situation
> > >> and
> > >>>>>>>>>> remove
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>> member. Therefore, the goal of this KIP is to address the
> > >>>> issue
> > >>>>>>>>>> where
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>> GC has to wait for the heartbeat timeout due to the client
> > >>>>> leaving
> > >>>>>>>>>>>>> without
> > >>>>>>>>>>>>>> knowing its memberId, which leads to reduced throughput
> > >> and
> > >>>>>>>>>> limited
> > >>>>>>>>>>>>>> scalability.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> The solution you suggest has also been proposed by David.
> > >>> The
> > >>>>>>>>>> concern
> > >>>>>>>>>>>>> with
> > >>>>>>>>>>>>>> this approach is that it introduces additional complexity
> > >>> for
> > >>>>>>>>>>>>>> compatibility, as the new server would not immediately add
> > >>> the
> > >>>>>>>>>> member
> > >>>>>>>>>>>> to
> > >>>>>>>>>>>>>> the group, while the old server would. This requires
> > >> clients
> > >>>> to
> > >>>>>>>>>>>>>> differentiate whether their memberId has been added to the
> > >>>> group
> > >>>>>>>>>> or
> > >>>>>>>>>>>> not,
> > >>>>>>>>>>>>>> which could result in unexpected logs.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>> TengYao
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Andrew Schofield <an...@live.com> 於 2024年8月14日
> > >>> 週三
> > >>>>>>>>>>>> 上午12:29寫道：
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Hi TengYao,
> > >>>>>>>>>>>>>>> Thanks for the KIP. I wonder if there’s a different way
> > >> to
> > >>>>> close
> > >>>>>>>>>>> what
> > >>>>>>>>>>>>>>> is quite a small window.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> AS1: It is true that the initial heartbeat is not
> > >>> idempotent,
> > >>>>> but
> > >>>>>>>>>>> this
> > >>>>>>>>>>>>>>> remains
> > >>>>>>>>>>>>>>> true with this KIP. It’s just differently not idempotent.
> > >>> If
> > >>>>> the
> > >>>>>>>>>>>> client
> > >>>>>>>>>>>>>>> makes its
> > >>>>>>>>>>>>>>> own member ID, sends a request and dies, the GC will
> > >> still
> > >>>> have
> > >>>>>>>>>>> added
> > >>>>>>>>>>>>>>> the member to the group and it will hang around until the
> > >>>>> session
> > >>>>>>>>>>>>> expires.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> I wonder if the GC could still generate the member ID in
> > >>>>>>>>>> response to
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>>> first
> > >>>>>>>>>>>>>>> heartbeat, and put the member in a special PENDING state
> > >>> with
> > >>>>> no
> > >>>>>>>>>>>>>>> assignments until the client sends the next heartbeat,
> > >> thus
> > >>>>>>>>>>> confirming
> > >>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>> has received the member ID. This would not be a protocol
> > >>>> change
> > >>>>>>>>>> at
> > >>>>>>>>>>>> all,
> > >>>>>>>>>>>>>>> just
> > >>>>>>>>>>>>>>> a change to the GC to keep a new member in the lobby
> > >> until
> > >>>> it’s
> > >>>>>>>>>>>>> comfirmed
> > >>>>>>>>>>>>>>> it knows its member ID.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>> Andrew
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On 13 Aug 2024, at 15:59, TengYao Chi <
> > >>> kitingiao@gmail.com>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Hi Chia-Ping,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thanks for review and suggestions.
> > >>>>>>>>>>>>>>>> I have updated the content of KIP accordingly.
> > >>>>>>>>>>>>>>>> Please take a look.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>>> TengYao
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2024年8月13日 週二
> > >>>>> 下午9:45寫道：
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> hi TengYao
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> thanks for this KIP.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> 1) could you please describe the before/after behavior
> > >> in
> > >>>> the
> > >>>>>>>>>>>>> "Proposed
> > >>>>>>>>>>>>>>>>> Changes" section? IIRC, current RPC allows HB having
> > >>> member
> > >>>>> id
> > >>>>>>>>>>>>>>> generated by
> > >>>>>>>>>>>>>>>>> client, right? If HB has no member ID, server will
> > >>> generate
> > >>>>> one
> > >>>>>>>>>>> and
> > >>>>>>>>>>>>> then
> > >>>>>>>>>>>>>>>>> return. The new behavior will enforce HB "must" have
> > >>> member
> > >>>>> ID.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> 2) could you please write the version number explicitly
> > >>> in
> > >>>>> the
> > >>>>>>>>>> KIP
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> 3) how new client code handle the old HB? Does it
> > >> always
> > >>>>>>>>>> generate
> > >>>>>>>>>>>>> member
> > >>>>>>>>>>>>>>>>> ID on client-side even though that is not restricted?
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On 2024/08/13 06:20:42 TengYao Chi wrote:
> > >>>>>>>>>>>>>>>>>> Hello everyone,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> I would like to start a discussion thread on KIP-1082,
> > >>>> which
> > >>>>>>>>>>>> proposes
> > >>>>>>>>>>>>>>>>>> enabling id generation for clients over the
> > >>>>>>>>>>> ConsumerGroupHeartbeat
> > >>>>>>>>>>>>> RPC.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Here is the KIP Link: KIP-1082
> > >>>>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Please take a look and let me know what you think,
> > >> and I
> > >>>>> would
> > >>>>>>>>>>>>>>> appreciate
> > >>>>>>>>>>>>>>>>>> any suggestions and feedback.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>>>>> TengYao
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> >
>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[TengYao Chi <ki...@gmail.com>]

Hello Kirk !

Thank you for your comments !

KT1: Yes, you are correct. The issue is not unique to the initial
heartbeat; there can always be cases where the broker might lose connection
with a member.

KT2: Currently, if the client doesn't have a member ID and the memberEpoch
equals 0, the coordinator will generate a UUID as the member ID for the
client. However, at the RPC level, the member ID is sent as a literal
string, meaning there are no restrictions on the format at this level.
This also reminds me that we haven't reached a final conclusion on how to
enforce the use of UUIDs.
From our previous discussions, I recall two possible approaches:
The first is to validate the UUID on the server side, and if it's not
valid, throw an exception to the client.
The second is to document a recommendation for clients to use UUIDs as
member IDs, without strictly enforcing it.
I think it's time to decide on the approach we want to take.

KT3: Yes, "session" can be considered synonymous with "membership" in this
context.

KT4: Thank you for pointing that out. I will update the wording to
specifically say this behavior is for consumers.

Thanks again for your comments.

Best regards,
TengYao

Kirk True <ki...@kirktrue.pro> 於 2024年8月30日 週五 上午12:39寫道：

> Hi TengYao!
>
> Sorry for being late to the discussion...
>
> After reading the thread and then the KIP, I had a few questions/comments:
>
> KT1: In Motivation, it states: "This scenario can result in the broker
> registering a new member for which it will never receive a proper leave
> request.” Just to be clear, the broker will always have cases where it
> might lose connection with a member. That’s not unique to the initial
> heartbeat, right?
>
> KT2: There was a bit of back and forth about format of the member ID. From
> what I gathered in the thread, the member ID is still defined in the RPC as
> a string and not a UUID, right? The KIP states that the “client must
> generate a UUID as the member ID” and that the “server will validate that a
> valid UUID is provided.” Is that a change for the server, or is it already
> enforced as a UUID?
>
> KT3: Lianet mentioned some confusion over the use of the word “session.”
> Isn’t “session” synonymous with “membership?”
>
> KT4: Under “Member ID Lifecycle,” it states: "The client should reuse the
> same UUID as the member ID for all heartbeats and rejoin attempts to
> maintain continuity within the group.” Could we change the first part of
> that to “The Consumer instance should…” We do have lifetimes that extend
> past the lifetime of a client instance (such as the transaction ID).
>
> Thanks,
> Kirk
>
> > On Aug 29, 2024, at 1:28 AM, TengYao Chi <ki...@gmail.com> wrote:
> >
> > Hi David,
> >
> > Thank you for pointing that out.
> > I have updated the content of the KIP based on Lianet's and your
> feedback.
> > Please take a look and let me know your thoughts.
> >
> > Best regards,
> > TengYao
> >
> > David Jacot <dj...@confluent.io.invalid> 於 2024年8月29日 週四 下午3:20寫道：
> >
> >> Hi TengYao,
> >>
> >> Thanks for the update. I haven't fully read it yet but I will soon.
> >>
> >> LM4: This is incorrect. The consumer must keep its member id during its
> >> entire lifetime (until the process stops or dies). The protocol
> stipulates
> >> that a member must rejoin with the same member id and the member epoch
> set
> >> to zero when an FENCED_MEMBER_EPOCH occurs. This allows the member to
> >> resynchronize itself. We should not change this behavior. I think that
> we
> >> should see the client side generation id as an incarnation id of the
> >> application. It is generated once and kept until it stops or dies.
> >>
> >> Best,
> >> David
> >>
> >> On Thu, Aug 29, 2024 at 6:21 AM TengYao Chi <ki...@gmail.com>
> wrote:
> >>
> >>> Hello Lianet !
> >>>
> >>> Thanks for the reviews and suggestions!
> >>>
> >>> LM1: Indeed, we plan to enforce client-side ID generation in the
> future,
> >>> and it is not an alternative. I will change the title accordingly.
> >>>
> >>> LM2: Yes, that's the expectation. I will add that statement to the
> public
> >>> interface section.
> >>>
> >>> LM3: Thank you for the high-level perspective review. I think you're
> >> right;
> >>> our intention isn't very clear since it was placed at the end of the
> >>> section. I will try to rephrase that section to make it more obvious.
> >>>
> >>> LM4: Regarding the definition of "session" in this KIP, I believe it
> >> refers
> >>> to the period between the *first-time heartbeat* and when the *consumer
> >>> leaves the group* (whether through a graceful shutdown or a heartbeat
> >>> timeout). The consumer should reuse its UUID if it has been generated
> >>> before. The only situation in which it will regenerate the UUID is if
> the
> >>> coordinator finds that there is already a consumer with the same UUID.
> >>> IIRC, the coordinator should compare the member epochs, and the
> >>> later-joined consumer should be fenced off by the coordinator due to
> >> having
> >>> a lower member epoch. Once the consumer receives a
> `FENCED_MEMBER_EPOCH`
> >>> error, it will generate a new UUID and attempt to rejoin. I will
> clarify
> >>> this in the KIP.
> >>>
> >>> Thanks again for your reviews, I really appreciate it.
> >>>
> >>> Best regards,
> >>> TengYao
> >>>
> >>> Lianet M. <li...@gmail.com> 於 2024年8月28日 週三 下午7:12寫道：
> >>>
> >>>> Hello TengYao! Thanks for taking on this issue, we've been going
> around
> >>> it
> >>>> for a while.
> >>>>
> >>>> LM1: About the title of the KIP: "Enable ID Generation for Clients
> over
> >>> the
> >>>> ConsumerGroupHeartbeat RPC". I find it confusing because it hints that
> >>>> we're adding it as an alternative (which was discussed and discarded,
> >> in
> >>>> favour of really enforcing it). It's also missing the core change imo,
> >>>> which is "where" the generation happens. So, maybe more to the point
> >> with
> >>>> something along the lines of "Client-side generated ID for clients
> over
> >>>> ConsumerGroupHeartbeat RPC"?
> >>>>
> >>>> LM2: On the public interfaces section, the KIP states that "the server
> >>> will
> >>>> reject the request", but we should agree on the specific error type. I
> >>>> expect it should fail with an InvalidRequestException, is that the
> >>>> intention? (This was also suggested in the discussion thread before
> but
> >>> is
> >>>> not in the KIP).
> >>>>
> >>>> LM3. Related to my previous point, I find that to be the true
> >>> public-facing
> >>>> change (member ID mandatory at the protocol level), but it's only at
> >> the
> >>>> end of the Public interfaces changes, kind of lost among details of
> how
> >>>> we're going to do it. Should we rephrase that section with the actual
> >>>> change first, and the hows after (ex. Bumping the version is not the
> >>>> public-facing change in this case, it's just the mechanism to properly
> >>>> introduce our change)
> >>>>
> >>>> LM4. Regarding the lifetime of the UUID: the KIP states we will
> "Verify
> >>>> that the UUID remains consistent across all subsequent heartbeats
> >> during
> >>>> the session". What is this "session" referring to here? I would expect
> >>> that
> >>>> the UUID is associated to a consumer instance (generated for the
> >> consumer
> >>>> the first time it needs to send a HB if it doesn't have the UUID yet.
> >>> From
> >>>> there on, every time it needs to send a "first HB" again, it will
> reuse
> >>> its
> >>>> UUID, is that the intention? Note that we should consider that the
> same
> >>>> consumer instance may have many "first heartbeats", meaning heartbeats
> >> to
> >>>> join the group when it's not part of it (ex. consumer unsubscribe +
> >>>> subscribe, fenced, stale). Is this the intention or are you
> considering
> >>> the
> >>>> lifetime differently? We should clarify it in the KIP.
> >>>>
> >>>> Thanks!
> >>>>
> >>>> Lianet
> >>>>
> >>>> On Tue, Aug 27, 2024 at 2:27 AM TengYao Chi <ki...@gmail.com>
> >> wrote:
> >>>>
> >>>>> Hi everyone,
> >>>>>
> >>>>> I have revised this KIP multiple times based on the feedback from our
> >>>>> discussions.
> >>>>> I would greatly appreciate it if you could review it when you have
> >> the
> >>>>> time.
> >>>>> If there are no further comments or suggestions, I plan to proceed
> >> with
> >>>>> initiating a vote soon.
> >>>>>
> >>>>> Best regards,
> >>>>> TengYao
> >>>>>
> >>>>> TengYao Chi <ki...@gmail.com> 於 2024年8月23日 週五 下午2:43寫道：
> >>>>>
> >>>>>> Hi Andrew,
> >>>>>> Thank you for your previous feedback and insights.
> >>>>>> Your contributions have added valuable perspectives to the
> >>> discussions.
> >>>>>> And we also benefit from the comparison of different solutions.
> >>>>>> I’m also looking forward to seeing an initial version in KIP-932,
> >> as
> >>> it
> >>>>>> will provide a good reference for future implementations.
> >>>>>>
> >>>>>> Regarding your comment on AS2, I wanted to clarify that my
> >>>> specification
> >>>>>> references org.apache.kafka.common.Uuid.
> >>>>>> I believe we’re referring to the same class, and it might just be a
> >>>> small
> >>>>>> oversight due to the busy schedule.
> >>>>>>
> >>>>>> I want to express my gratitude once again for your many insightful
> >>>>>> comments, which have helped the discussion progress smoothly.
> >>>>>>
> >>>>>> Best regards,
> >>>>>> TengYao
> >>>>>>
> >>>>>>
> >>>>>> Andrew Schofield <an...@live.com> 於 2024年8月22日 週四
> >>>> 下午11:28寫道：
> >>>>>>
> >>>>>>> Hi TengYao,
> >>>>>>> I’ve been reading through the comments and I’m happy that the
> >> lobby
> >>>>>>> approach has not gained support.
> >>>>>>>
> >>>>>>> Assuming that this KIP is voted, I will be happy to change KIP-932
> >>> so
> >>>>>>> that it only supports client-side member ID generation. Because
> >> that
> >>>> KIP
> >>>>>>> is still
> >>>>>>> under development, I can do this in the first version of
> >>>>>>> ShareGroupHeartbeat.
> >>>>>>>
> >>>>>>> AS2: For the encoding section, I suppose the specific encoding
> >> which
> >>>>>>> is used is what org.apache.kafka.utils.Uuid uses.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Andrew
> >>>>>>>
> >>>>>>>> On 14 Aug 2024, at 17:00, TengYao Chi <ki...@gmail.com>
> >>> wrote:
> >>>>>>>>
> >>>>>>>> Hello Apoorv,
> >>>>>>>> Thank you for your feedback.
> >>>>>>>> Regarding the questions you raised, unfortunately, this KIP
> >> cannot
> >>>>>>>> guarantee the order of heartbeats. As with many classic
> >>> distributed
> >>>>>>> system
> >>>>>>>> challenges, what we can do is make our best effort to ensure
> >> that
> >>>>> there
> >>>>>>> are
> >>>>>>>> no idle members or stale assignments under normal circumstances.
> >>>>>>>>
> >>>>>>>> As for the lobby approach, I’m not a fan of it because it
> >> requires
> >>>>>>> adding a
> >>>>>>>> mechanism to maintain client state within the ConsumerGroup,
> >>> which,
> >>>> in
> >>>>>>> my
> >>>>>>>> view, resembles something like a two-phase commit. This would
> >>>>> introduce
> >>>>>>>> more complexity than the proposal in this KIP, which is
> >> something
> >>> we
> >>>>>>> want
> >>>>>>>> to avoid. KIP-848 aims to simplify the existing protocol, and
> >>> while
> >>>>> the
> >>>>>>>> lobby approach is a good one, I believe it is not the right fit
> >>> for
> >>>>> this
> >>>>>>>> particular situation.
> >>>>>>>>
> >>>>>>>> Best regards,
> >>>>>>>> TengYao
> >>>>>>>>
> >>>>>>>> TengYao Chi <ki...@gmail.com> 於 2024年8月14日 週三 下午11:45寫道：
> >>>>>>>>
> >>>>>>>>> Hi David,
> >>>>>>>>>
> >>>>>>>>> I really appreciate your review and suggestions. As I am still
> >>>>> gaining
> >>>>>>>>> experience in writing KIPs, your input has been incredibly
> >>>> helpful. I
> >>>>>>> am
> >>>>>>>>> currently applying your suggestions to the KIP and will
> >> complete
> >>> it
> >>>>> as
> >>>>>>> soon
> >>>>>>>>> as possible.
> >>>>>>>>> Regarding the UUID part, I think we haven’t reached a
> >> conclusion
> >>>>>>> yet.(So
> >>>>>>>>> far according to this thread)
> >>>>>>>>> However, I will review the current implementation in the Kafka
> >>>> `Uuid`
> >>>>>>>>> class and include a brief specification in the KIP.
> >>>>>>>>>
> >>>>>>>>> Once again, thank you so much for your help.
> >>>>>>>>>
> >>>>>>>>> Best regards,
> >>>>>>>>> TengYao
> >>>>>>>>>
> >>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月14日 週三 下午11:14寫道：
> >>>>>>>>>
> >>>>>>>>>> hi Apoorv
> >>>>>>>>>>
> >>>>>>>>>>> As the memberId is now known to the client, and client might
> >>> send
> >>>>> the
> >>>>>>>>>> leave
> >>>>>>>>>> group heartbeat on shutdown prior to receiving the initial
> >>>> heartbeat
> >>>>>>>>>> response. If that's true then how do we guarantee that the 2
> >>>>> requests
> >>>>>>> to
> >>>>>>>>>> join and leave will be processed in order, which could still
> >>> leave
> >>>>>>> stale
> >>>>>>>>>> members or throw unknown member id exceptions?
> >>>>>>>>>>
> >>>>>>>>>> This is definitely a good question. the short answer: no
> >>> guarantee
> >>>>> but
> >>>>>>>>>> best
> >>>>>>>>>> efforts
> >>>>>>>>>>
> >>>>>>>>>> Please notice the root cause is "we have no enough time to
> >> wait
> >>>>>>> member id
> >>>>>>>>>> (response) when closing consumer". Sadly, we can' guarantee
> >> the
> >>>>>>> request
> >>>>>>>>>> order due to the same reason.
> >>>>>>>>>>
> >>>>>>>>>> However, in contrast to previous behavior, there is one big
> >>>> benefit
> >>>>>>> of new
> >>>>>>>>>> approach - we can try STONITH because we know the member id
> >>>>>>>>>>
> >>>>>>>>>> Best,
> >>>>>>>>>> Chia-Ping
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Apoorv Mittal <ap...@gmail.com> 於 2024年8月14日 週三
> >>>> 下午8:55寫道：
> >>>>>>>>>>
> >>>>>>>>>>> Hi TengYao,
> >>>>>>>>>>> Thanks for the KIP. Continuing on the point which Andrew
> >>>> mentioned
> >>>>> as
> >>>>>>>>>> AS1.
> >>>>>>>>>>>
> >>>>>>>>>>> As the memberId is now known to the client, and client might
> >>> send
> >>>>> the
> >>>>>>>>>> leave
> >>>>>>>>>>> group heartbeat on shutdown prior to receiving the initial
> >>>>> heartbeat
> >>>>>>>>>>> response. If that's true then how do we guarantee that the 2
> >>>>>>> requests to
> >>>>>>>>>>> join and leave will be processed in order, which could still
> >>>> leave
> >>>>>>> stale
> >>>>>>>>>>> members or throw unknown member id exceptions?
> >>>>>>>>>>>
> >>>>>>>>>>> Though the client side member id generation is helpful which
> >>> will
> >>>>>>>>>> represent
> >>>>>>>>>>> the same group perspective as from client and broker's end.
> >>> But I
> >>>>>>> think
> >>>>>>>>>> the
> >>>>>>>>>>> major concern we want to solve here is Stale Partition
> >>>> Assignments
> >>>>>>> which
> >>>>>>>>>>> might still exist with the new approach. I am leaning towards
> >>> the
> >>>>>>>>>>> suggestion mentioned by Andrew where partition assignment
> >>>> triggers
> >>>>> on
> >>>>>>>>>>> subsequent heartbeat when client acknowledges the initial
> >>>>> heartbeat,
> >>>>>>>>>>> delayed partition assignment.
> >>>>>>>>>>>
> >>>>>>>>>>> Though on a separate note, I have a different question. What
> >>>>> happens
> >>>>>>>>>> when
> >>>>>>>>>>> there is an issue with the client which sends the initial
> >>>> heartbeat
> >>>>>>>>>> without
> >>>>>>>>>>> memberId, then crashes and restarts? I think we must be
> >>>>> re-triggering
> >>>>>>>>>>> assignments and expiring members only after the heartbeat
> >>> session
> >>>>>>>>>> timeout?
> >>>>>>>>>>> If that's true then shall delayed partition assignment can
> >> help
> >>>>>>> benefit
> >>>>>>>>>> us
> >>>>>>>>>>> from this situation as well?
> >>>>>>>>>>>
> >>>>>>>>>>> Regards,
> >>>>>>>>>>> Apoorv Mittal
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Aug 14, 2024 at 12:51 PM David Jacot
> >>>>>>>>>> <dj...@confluent.io.invalid>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi Andrew,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Personally, I don't like the lobby approach. It makes things
> >>>> more
> >>>>>>>>>>>> complicated and it would require changing the records on the
> >>>>> server
> >>>>>>>>>> too.
> >>>>>>>>>>>> This is why I initially suggested the rejected alternative
> >> #2
> >>>>> which
> >>>>>>> is
> >>>>>>>>>>>> pretty close but also not perfect.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I'd like to clarify one thing. The ConsumerGroupHeartbeat
> >> API
> >>>>>>> already
> >>>>>>>>>>>> supports generating the member id on the client so we don't
> >>> need
> >>>>> any
> >>>>>>>>>>>> conditional logic on the client side. This is actually what
> >> we
> >>>>>>> wanted
> >>>>>>>>>> to
> >>>>>>>>>>> do
> >>>>>>>>>>>> in the first place but the idea got pushed back by Magnus
> >> back
> >>>>> then
> >>>>>>>>>>> because
> >>>>>>>>>>>> generating uuid from librdkafka required a new dependency.
> >> It
> >>>>> turns
> >>>>>>>>>> out
> >>>>>>>>>>>> that librdkafka has that dependency today. In retrospect, we
> >>>>> should
> >>>>>>>>>> have
> >>>>>>>>>>>> pushed back on this. Long story short, we can just do it.
> >> The
> >>>>>>>>>> proposal in
> >>>>>>>>>>>> this KIP is to make the member id required in future
> >> versions.
> >>>> We
> >>>>>>>>>> could
> >>>>>>>>>>>> also decide not to do it and to keep supporting both
> >>>> approaches. I
> >>>>>>>>>> would
> >>>>>>>>>>>> also be fine with this.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best,
> >>>>>>>>>>>> David
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Wed, Aug 14, 2024 at 12:30 PM Andrew Schofield <
> >>>>>>>>>>>> andrew_schofield@live.com>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi TengYao,
> >>>>>>>>>>>>> Thanks for your response. I’ll have just one more try to
> >>>>> persuade.
> >>>>>>>>>>>>> I feel that I will need to follow the approach with KIP-932
> >>>> when
> >>>>>>>>>> we’ve
> >>>>>>>>>>>>> made a decision, so I do have more than a passing interest
> >> in
> >>>>> this.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> A group member in the lobby is in the group, but it does
> >> not
> >>>> have
> >>>>>>>>>> any
> >>>>>>>>>>>>> assignments. A member of a consumer group can have no
> >>> assigned
> >>>>>>>>>>>>> partitions (such as 5 CG members subscribed to a topic
> >> with 4
> >>>>>>>>>>>> partitions),
> >>>>>>>>>>>>> so it’s a situation that consumer group members already
> >>> expect.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> One of Kafka’s strengths is the way that we handle API
> >>>>> versioning.
> >>>>>>>>>>>>> But, there is a cost - the behaviour is different depending
> >>> on
> >>>>> the
> >>>>>>>>>> RPC
> >>>>>>>>>>>>> version. KIP-848 is on the cusp of completion, but we’re
> >>>> already
> >>>>>>>>>> adding
> >>>>>>>>>>>>> conditional logic for v0/v1 for ConsumerGroupHeartbeat.
> >>> That’s
> >>>> a
> >>>>>>>>>> pity.
> >>>>>>>>>>>>> Only a minor issue, but it’s unfortunate.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>> Andrew
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 14 Aug 2024, at 08:47, TengYao Chi <
> >> kitingiao@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hello Andrew
> >>>>>>>>>>>>>> Thank you for your thoughtful suggestions and getting the
> >>>>>>>>>> discussion
> >>>>>>>>>>>>> going.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> To AS1:
> >>>>>>>>>>>>>> In the current scenario where the server generates the
> >> UUID,
> >>>> if
> >>>>>>>>>> the
> >>>>>>>>>>>>> client
> >>>>>>>>>>>>>> shuts down before receiving the memberId generated by the
> >> GC
> >>>>>>>>>>>> (regardless
> >>>>>>>>>>>>> of
> >>>>>>>>>>>>>> whether it’s a graceful shutdown or not), the GC will
> >> still
> >>>> have
> >>>>>>>>>> to
> >>>>>>>>>>>> wait
> >>>>>>>>>>>>>> for the heartbeat timeout because the client doesn’t know
> >>> its
> >>>>>>>>>>> memberId.
> >>>>>>>>>>>>>> This KIP indeed cannot completely resolve the idempotency
> >>>> issue,
> >>>>>>>>>> but
> >>>>>>>>>>> it
> >>>>>>>>>>>>> can
> >>>>>>>>>>>>>> better handle shutdown scenarios under normal
> >> circumstances
> >>>>>>>>>> because
> >>>>>>>>>>> the
> >>>>>>>>>>>>>> client always knows its memberId. Even if the client shuts
> >>>> down
> >>>>>>>>>>>>> immediately
> >>>>>>>>>>>>>> after the initial heartbeat, as long as it performs a
> >>> graceful
> >>>>>>>>>>> shutdown
> >>>>>>>>>>>>> and
> >>>>>>>>>>>>>> sends a leave heartbeat, the GC can manage the situation
> >> and
> >>>>>>>>>> remove
> >>>>>>>>>>> the
> >>>>>>>>>>>>>> member. Therefore, the goal of this KIP is to address the
> >>>> issue
> >>>>>>>>>> where
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>> GC has to wait for the heartbeat timeout due to the client
> >>>>> leaving
> >>>>>>>>>>>>> without
> >>>>>>>>>>>>>> knowing its memberId, which leads to reduced throughput
> >> and
> >>>>>>>>>> limited
> >>>>>>>>>>>>>> scalability.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> The solution you suggest has also been proposed by David.
> >>> The
> >>>>>>>>>> concern
> >>>>>>>>>>>>> with
> >>>>>>>>>>>>>> this approach is that it introduces additional complexity
> >>> for
> >>>>>>>>>>>>>> compatibility, as the new server would not immediately add
> >>> the
> >>>>>>>>>> member
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>> the group, while the old server would. This requires
> >> clients
> >>>> to
> >>>>>>>>>>>>>> differentiate whether their memberId has been added to the
> >>>> group
> >>>>>>>>>> or
> >>>>>>>>>>>> not,
> >>>>>>>>>>>>>> which could result in unexpected logs.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>> TengYao
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Andrew Schofield <an...@live.com> 於 2024年8月14日
> >>> 週三
> >>>>>>>>>>>> 上午12:29寫道：
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi TengYao,
> >>>>>>>>>>>>>>> Thanks for the KIP. I wonder if there’s a different way
> >> to
> >>>>> close
> >>>>>>>>>>> what
> >>>>>>>>>>>>>>> is quite a small window.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> AS1: It is true that the initial heartbeat is not
> >>> idempotent,
> >>>>> but
> >>>>>>>>>>> this
> >>>>>>>>>>>>>>> remains
> >>>>>>>>>>>>>>> true with this KIP. It’s just differently not idempotent.
> >>> If
> >>>>> the
> >>>>>>>>>>>> client
> >>>>>>>>>>>>>>> makes its
> >>>>>>>>>>>>>>> own member ID, sends a request and dies, the GC will
> >> still
> >>>> have
> >>>>>>>>>>> added
> >>>>>>>>>>>>>>> the member to the group and it will hang around until the
> >>>>> session
> >>>>>>>>>>>>> expires.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I wonder if the GC could still generate the member ID in
> >>>>>>>>>> response to
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>> first
> >>>>>>>>>>>>>>> heartbeat, and put the member in a special PENDING state
> >>> with
> >>>>> no
> >>>>>>>>>>>>>>> assignments until the client sends the next heartbeat,
> >> thus
> >>>>>>>>>>> confirming
> >>>>>>>>>>>>> it
> >>>>>>>>>>>>>>> has received the member ID. This would not be a protocol
> >>>> change
> >>>>>>>>>> at
> >>>>>>>>>>>> all,
> >>>>>>>>>>>>>>> just
> >>>>>>>>>>>>>>> a change to the GC to keep a new member in the lobby
> >> until
> >>>> it’s
> >>>>>>>>>>>>> comfirmed
> >>>>>>>>>>>>>>> it knows its member ID.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>> Andrew
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On 13 Aug 2024, at 15:59, TengYao Chi <
> >>> kitingiao@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi Chia-Ping,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks for review and suggestions.
> >>>>>>>>>>>>>>>> I have updated the content of KIP accordingly.
> >>>>>>>>>>>>>>>> Please take a look.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>> TengYao
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2024年8月13日 週二
> >>>>> 下午9:45寫道：
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> hi TengYao
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> thanks for this KIP.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> 1) could you please describe the before/after behavior
> >> in
> >>>> the
> >>>>>>>>>>>>> "Proposed
> >>>>>>>>>>>>>>>>> Changes" section? IIRC, current RPC allows HB having
> >>> member
> >>>>> id
> >>>>>>>>>>>>>>> generated by
> >>>>>>>>>>>>>>>>> client, right? If HB has no member ID, server will
> >>> generate
> >>>>> one
> >>>>>>>>>>> and
> >>>>>>>>>>>>> then
> >>>>>>>>>>>>>>>>> return. The new behavior will enforce HB "must" have
> >>> member
> >>>>> ID.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> 2) could you please write the version number explicitly
> >>> in
> >>>>> the
> >>>>>>>>>> KIP
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> 3) how new client code handle the old HB? Does it
> >> always
> >>>>>>>>>> generate
> >>>>>>>>>>>>> member
> >>>>>>>>>>>>>>>>> ID on client-side even though that is not restricted?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On 2024/08/13 06:20:42 TengYao Chi wrote:
> >>>>>>>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I would like to start a discussion thread on KIP-1082,
> >>>> which
> >>>>>>>>>>>> proposes
> >>>>>>>>>>>>>>>>>> enabling id generation for clients over the
> >>>>>>>>>>> ConsumerGroupHeartbeat
> >>>>>>>>>>>>> RPC.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Here is the KIP Link: KIP-1082
> >>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Please take a look and let me know what you think,
> >> and I
> >>>>> would
> >>>>>>>>>>>>>>> appreciate
> >>>>>>>>>>>>>>>>>> any suggestions and feedback.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>> TengYao
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[Kirk True <ki...@kirktrue.pro>]

Hi TengYao!

Sorry for being late to the discussion...

After reading the thread and then the KIP, I had a few questions/comments:

KT1: In Motivation, it states: "This scenario can result in the broker registering a new member for which it will never receive a proper leave request.” Just to be clear, the broker will always have cases where it might lose connection with a member. That’s not unique to the initial heartbeat, right?

KT2: There was a bit of back and forth about format of the member ID. From what I gathered in the thread, the member ID is still defined in the RPC as a string and not a UUID, right? The KIP states that the “client must generate a UUID as the member ID” and that the “server will validate that a valid UUID is provided.” Is that a change for the server, or is it already enforced as a UUID?

KT3: Lianet mentioned some confusion over the use of the word “session.” Isn’t “session” synonymous with “membership?”

KT4: Under “Member ID Lifecycle,” it states: "The client should reuse the same UUID as the member ID for all heartbeats and rejoin attempts to maintain continuity within the group.” Could we change the first part of that to “The Consumer instance should…” We do have lifetimes that extend past the lifetime of a client instance (such as the transaction ID).

Thanks,
Kirk

> On Aug 29, 2024, at 1:28 AM, TengYao Chi <ki...@gmail.com> wrote:
> 
> Hi David,
> 
> Thank you for pointing that out.
> I have updated the content of the KIP based on Lianet's and your feedback.
> Please take a look and let me know your thoughts.
> 
> Best regards,
> TengYao
> 
> David Jacot <dj...@confluent.io.invalid> 於 2024年8月29日 週四 下午3:20寫道：
> 
>> Hi TengYao,
>> 
>> Thanks for the update. I haven't fully read it yet but I will soon.
>> 
>> LM4: This is incorrect. The consumer must keep its member id during its
>> entire lifetime (until the process stops or dies). The protocol stipulates
>> that a member must rejoin with the same member id and the member epoch set
>> to zero when an FENCED_MEMBER_EPOCH occurs. This allows the member to
>> resynchronize itself. We should not change this behavior. I think that we
>> should see the client side generation id as an incarnation id of the
>> application. It is generated once and kept until it stops or dies.
>> 
>> Best,
>> David
>> 
>> On Thu, Aug 29, 2024 at 6:21 AM TengYao Chi <ki...@gmail.com> wrote:
>> 
>>> Hello Lianet !
>>> 
>>> Thanks for the reviews and suggestions!
>>> 
>>> LM1: Indeed, we plan to enforce client-side ID generation in the future,
>>> and it is not an alternative. I will change the title accordingly.
>>> 
>>> LM2: Yes, that's the expectation. I will add that statement to the public
>>> interface section.
>>> 
>>> LM3: Thank you for the high-level perspective review. I think you're
>> right;
>>> our intention isn't very clear since it was placed at the end of the
>>> section. I will try to rephrase that section to make it more obvious.
>>> 
>>> LM4: Regarding the definition of "session" in this KIP, I believe it
>> refers
>>> to the period between the *first-time heartbeat* and when the *consumer
>>> leaves the group* (whether through a graceful shutdown or a heartbeat
>>> timeout). The consumer should reuse its UUID if it has been generated
>>> before. The only situation in which it will regenerate the UUID is if the
>>> coordinator finds that there is already a consumer with the same UUID.
>>> IIRC, the coordinator should compare the member epochs, and the
>>> later-joined consumer should be fenced off by the coordinator due to
>> having
>>> a lower member epoch. Once the consumer receives a `FENCED_MEMBER_EPOCH`
>>> error, it will generate a new UUID and attempt to rejoin. I will clarify
>>> this in the KIP.
>>> 
>>> Thanks again for your reviews, I really appreciate it.
>>> 
>>> Best regards,
>>> TengYao
>>> 
>>> Lianet M. <li...@gmail.com> 於 2024年8月28日 週三 下午7:12寫道：
>>> 
>>>> Hello TengYao! Thanks for taking on this issue, we've been going around
>>> it
>>>> for a while.
>>>> 
>>>> LM1: About the title of the KIP: "Enable ID Generation for Clients over
>>> the
>>>> ConsumerGroupHeartbeat RPC". I find it confusing because it hints that
>>>> we're adding it as an alternative (which was discussed and discarded,
>> in
>>>> favour of really enforcing it). It's also missing the core change imo,
>>>> which is "where" the generation happens. So, maybe more to the point
>> with
>>>> something along the lines of "Client-side generated ID for clients over
>>>> ConsumerGroupHeartbeat RPC"?
>>>> 
>>>> LM2: On the public interfaces section, the KIP states that "the server
>>> will
>>>> reject the request", but we should agree on the specific error type. I
>>>> expect it should fail with an InvalidRequestException, is that the
>>>> intention? (This was also suggested in the discussion thread before but
>>> is
>>>> not in the KIP).
>>>> 
>>>> LM3. Related to my previous point, I find that to be the true
>>> public-facing
>>>> change (member ID mandatory at the protocol level), but it's only at
>> the
>>>> end of the Public interfaces changes, kind of lost among details of how
>>>> we're going to do it. Should we rephrase that section with the actual
>>>> change first, and the hows after (ex. Bumping the version is not the
>>>> public-facing change in this case, it's just the mechanism to properly
>>>> introduce our change)
>>>> 
>>>> LM4. Regarding the lifetime of the UUID: the KIP states we will "Verify
>>>> that the UUID remains consistent across all subsequent heartbeats
>> during
>>>> the session". What is this "session" referring to here? I would expect
>>> that
>>>> the UUID is associated to a consumer instance (generated for the
>> consumer
>>>> the first time it needs to send a HB if it doesn't have the UUID yet.
>>> From
>>>> there on, every time it needs to send a "first HB" again, it will reuse
>>> its
>>>> UUID, is that the intention? Note that we should consider that the same
>>>> consumer instance may have many "first heartbeats", meaning heartbeats
>> to
>>>> join the group when it's not part of it (ex. consumer unsubscribe +
>>>> subscribe, fenced, stale). Is this the intention or are you considering
>>> the
>>>> lifetime differently? We should clarify it in the KIP.
>>>> 
>>>> Thanks!
>>>> 
>>>> Lianet
>>>> 
>>>> On Tue, Aug 27, 2024 at 2:27 AM TengYao Chi <ki...@gmail.com>
>> wrote:
>>>> 
>>>>> Hi everyone,
>>>>> 
>>>>> I have revised this KIP multiple times based on the feedback from our
>>>>> discussions.
>>>>> I would greatly appreciate it if you could review it when you have
>> the
>>>>> time.
>>>>> If there are no further comments or suggestions, I plan to proceed
>> with
>>>>> initiating a vote soon.
>>>>> 
>>>>> Best regards,
>>>>> TengYao
>>>>> 
>>>>> TengYao Chi <ki...@gmail.com> 於 2024年8月23日 週五 下午2:43寫道：
>>>>> 
>>>>>> Hi Andrew,
>>>>>> Thank you for your previous feedback and insights.
>>>>>> Your contributions have added valuable perspectives to the
>>> discussions.
>>>>>> And we also benefit from the comparison of different solutions.
>>>>>> I’m also looking forward to seeing an initial version in KIP-932,
>> as
>>> it
>>>>>> will provide a good reference for future implementations.
>>>>>> 
>>>>>> Regarding your comment on AS2, I wanted to clarify that my
>>>> specification
>>>>>> references org.apache.kafka.common.Uuid.
>>>>>> I believe we’re referring to the same class, and it might just be a
>>>> small
>>>>>> oversight due to the busy schedule.
>>>>>> 
>>>>>> I want to express my gratitude once again for your many insightful
>>>>>> comments, which have helped the discussion progress smoothly.
>>>>>> 
>>>>>> Best regards,
>>>>>> TengYao
>>>>>> 
>>>>>> 
>>>>>> Andrew Schofield <an...@live.com> 於 2024年8月22日 週四
>>>> 下午11:28寫道：
>>>>>> 
>>>>>>> Hi TengYao,
>>>>>>> I’ve been reading through the comments and I’m happy that the
>> lobby
>>>>>>> approach has not gained support.
>>>>>>> 
>>>>>>> Assuming that this KIP is voted, I will be happy to change KIP-932
>>> so
>>>>>>> that it only supports client-side member ID generation. Because
>> that
>>>> KIP
>>>>>>> is still
>>>>>>> under development, I can do this in the first version of
>>>>>>> ShareGroupHeartbeat.
>>>>>>> 
>>>>>>> AS2: For the encoding section, I suppose the specific encoding
>> which
>>>>>>> is used is what org.apache.kafka.utils.Uuid uses.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Andrew
>>>>>>> 
>>>>>>>> On 14 Aug 2024, at 17:00, TengYao Chi <ki...@gmail.com>
>>> wrote:
>>>>>>>> 
>>>>>>>> Hello Apoorv,
>>>>>>>> Thank you for your feedback.
>>>>>>>> Regarding the questions you raised, unfortunately, this KIP
>> cannot
>>>>>>>> guarantee the order of heartbeats. As with many classic
>>> distributed
>>>>>>> system
>>>>>>>> challenges, what we can do is make our best effort to ensure
>> that
>>>>> there
>>>>>>> are
>>>>>>>> no idle members or stale assignments under normal circumstances.
>>>>>>>> 
>>>>>>>> As for the lobby approach, I’m not a fan of it because it
>> requires
>>>>>>> adding a
>>>>>>>> mechanism to maintain client state within the ConsumerGroup,
>>> which,
>>>> in
>>>>>>> my
>>>>>>>> view, resembles something like a two-phase commit. This would
>>>>> introduce
>>>>>>>> more complexity than the proposal in this KIP, which is
>> something
>>> we
>>>>>>> want
>>>>>>>> to avoid. KIP-848 aims to simplify the existing protocol, and
>>> while
>>>>> the
>>>>>>>> lobby approach is a good one, I believe it is not the right fit
>>> for
>>>>> this
>>>>>>>> particular situation.
>>>>>>>> 
>>>>>>>> Best regards,
>>>>>>>> TengYao
>>>>>>>> 
>>>>>>>> TengYao Chi <ki...@gmail.com> 於 2024年8月14日 週三 下午11:45寫道：
>>>>>>>> 
>>>>>>>>> Hi David,
>>>>>>>>> 
>>>>>>>>> I really appreciate your review and suggestions. As I am still
>>>>> gaining
>>>>>>>>> experience in writing KIPs, your input has been incredibly
>>>> helpful. I
>>>>>>> am
>>>>>>>>> currently applying your suggestions to the KIP and will
>> complete
>>> it
>>>>> as
>>>>>>> soon
>>>>>>>>> as possible.
>>>>>>>>> Regarding the UUID part, I think we haven’t reached a
>> conclusion
>>>>>>> yet.(So
>>>>>>>>> far according to this thread)
>>>>>>>>> However, I will review the current implementation in the Kafka
>>>> `Uuid`
>>>>>>>>> class and include a brief specification in the KIP.
>>>>>>>>> 
>>>>>>>>> Once again, thank you so much for your help.
>>>>>>>>> 
>>>>>>>>> Best regards,
>>>>>>>>> TengYao
>>>>>>>>> 
>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月14日 週三 下午11:14寫道：
>>>>>>>>> 
>>>>>>>>>> hi Apoorv
>>>>>>>>>> 
>>>>>>>>>>> As the memberId is now known to the client, and client might
>>> send
>>>>> the
>>>>>>>>>> leave
>>>>>>>>>> group heartbeat on shutdown prior to receiving the initial
>>>> heartbeat
>>>>>>>>>> response. If that's true then how do we guarantee that the 2
>>>>> requests
>>>>>>> to
>>>>>>>>>> join and leave will be processed in order, which could still
>>> leave
>>>>>>> stale
>>>>>>>>>> members or throw unknown member id exceptions?
>>>>>>>>>> 
>>>>>>>>>> This is definitely a good question. the short answer: no
>>> guarantee
>>>>> but
>>>>>>>>>> best
>>>>>>>>>> efforts
>>>>>>>>>> 
>>>>>>>>>> Please notice the root cause is "we have no enough time to
>> wait
>>>>>>> member id
>>>>>>>>>> (response) when closing consumer". Sadly, we can' guarantee
>> the
>>>>>>> request
>>>>>>>>>> order due to the same reason.
>>>>>>>>>> 
>>>>>>>>>> However, in contrast to previous behavior, there is one big
>>>> benefit
>>>>>>> of new
>>>>>>>>>> approach - we can try STONITH because we know the member id
>>>>>>>>>> 
>>>>>>>>>> Best,
>>>>>>>>>> Chia-Ping
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Apoorv Mittal <ap...@gmail.com> 於 2024年8月14日 週三
>>>> 下午8:55寫道：
>>>>>>>>>> 
>>>>>>>>>>> Hi TengYao,
>>>>>>>>>>> Thanks for the KIP. Continuing on the point which Andrew
>>>> mentioned
>>>>> as
>>>>>>>>>> AS1.
>>>>>>>>>>> 
>>>>>>>>>>> As the memberId is now known to the client, and client might
>>> send
>>>>> the
>>>>>>>>>> leave
>>>>>>>>>>> group heartbeat on shutdown prior to receiving the initial
>>>>> heartbeat
>>>>>>>>>>> response. If that's true then how do we guarantee that the 2
>>>>>>> requests to
>>>>>>>>>>> join and leave will be processed in order, which could still
>>>> leave
>>>>>>> stale
>>>>>>>>>>> members or throw unknown member id exceptions?
>>>>>>>>>>> 
>>>>>>>>>>> Though the client side member id generation is helpful which
>>> will
>>>>>>>>>> represent
>>>>>>>>>>> the same group perspective as from client and broker's end.
>>> But I
>>>>>>> think
>>>>>>>>>> the
>>>>>>>>>>> major concern we want to solve here is Stale Partition
>>>> Assignments
>>>>>>> which
>>>>>>>>>>> might still exist with the new approach. I am leaning towards
>>> the
>>>>>>>>>>> suggestion mentioned by Andrew where partition assignment
>>>> triggers
>>>>> on
>>>>>>>>>>> subsequent heartbeat when client acknowledges the initial
>>>>> heartbeat,
>>>>>>>>>>> delayed partition assignment.
>>>>>>>>>>> 
>>>>>>>>>>> Though on a separate note, I have a different question. What
>>>>> happens
>>>>>>>>>> when
>>>>>>>>>>> there is an issue with the client which sends the initial
>>>> heartbeat
>>>>>>>>>> without
>>>>>>>>>>> memberId, then crashes and restarts? I think we must be
>>>>> re-triggering
>>>>>>>>>>> assignments and expiring members only after the heartbeat
>>> session
>>>>>>>>>> timeout?
>>>>>>>>>>> If that's true then shall delayed partition assignment can
>> help
>>>>>>> benefit
>>>>>>>>>> us
>>>>>>>>>>> from this situation as well?
>>>>>>>>>>> 
>>>>>>>>>>> Regards,
>>>>>>>>>>> Apoorv Mittal
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Wed, Aug 14, 2024 at 12:51 PM David Jacot
>>>>>>>>>> <dj...@confluent.io.invalid>
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hi Andrew,
>>>>>>>>>>>> 
>>>>>>>>>>>> Personally, I don't like the lobby approach. It makes things
>>>> more
>>>>>>>>>>>> complicated and it would require changing the records on the
>>>>> server
>>>>>>>>>> too.
>>>>>>>>>>>> This is why I initially suggested the rejected alternative
>> #2
>>>>> which
>>>>>>> is
>>>>>>>>>>>> pretty close but also not perfect.
>>>>>>>>>>>> 
>>>>>>>>>>>> I'd like to clarify one thing. The ConsumerGroupHeartbeat
>> API
>>>>>>> already
>>>>>>>>>>>> supports generating the member id on the client so we don't
>>> need
>>>>> any
>>>>>>>>>>>> conditional logic on the client side. This is actually what
>> we
>>>>>>> wanted
>>>>>>>>>> to
>>>>>>>>>>> do
>>>>>>>>>>>> in the first place but the idea got pushed back by Magnus
>> back
>>>>> then
>>>>>>>>>>> because
>>>>>>>>>>>> generating uuid from librdkafka required a new dependency.
>> It
>>>>> turns
>>>>>>>>>> out
>>>>>>>>>>>> that librdkafka has that dependency today. In retrospect, we
>>>>> should
>>>>>>>>>> have
>>>>>>>>>>>> pushed back on this. Long story short, we can just do it.
>> The
>>>>>>>>>> proposal in
>>>>>>>>>>>> this KIP is to make the member id required in future
>> versions.
>>>> We
>>>>>>>>>> could
>>>>>>>>>>>> also decide not to do it and to keep supporting both
>>>> approaches. I
>>>>>>>>>> would
>>>>>>>>>>>> also be fine with this.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best,
>>>>>>>>>>>> David
>>>>>>>>>>>> 
>>>>>>>>>>>> On Wed, Aug 14, 2024 at 12:30 PM Andrew Schofield <
>>>>>>>>>>>> andrew_schofield@live.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi TengYao,
>>>>>>>>>>>>> Thanks for your response. I’ll have just one more try to
>>>>> persuade.
>>>>>>>>>>>>> I feel that I will need to follow the approach with KIP-932
>>>> when
>>>>>>>>>> we’ve
>>>>>>>>>>>>> made a decision, so I do have more than a passing interest
>> in
>>>>> this.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> A group member in the lobby is in the group, but it does
>> not
>>>> have
>>>>>>>>>> any
>>>>>>>>>>>>> assignments. A member of a consumer group can have no
>>> assigned
>>>>>>>>>>>>> partitions (such as 5 CG members subscribed to a topic
>> with 4
>>>>>>>>>>>> partitions),
>>>>>>>>>>>>> so it’s a situation that consumer group members already
>>> expect.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> One of Kafka’s strengths is the way that we handle API
>>>>> versioning.
>>>>>>>>>>>>> But, there is a cost - the behaviour is different depending
>>> on
>>>>> the
>>>>>>>>>> RPC
>>>>>>>>>>>>> version. KIP-848 is on the cusp of completion, but we’re
>>>> already
>>>>>>>>>> adding
>>>>>>>>>>>>> conditional logic for v0/v1 for ConsumerGroupHeartbeat.
>>> That’s
>>>> a
>>>>>>>>>> pity.
>>>>>>>>>>>>> Only a minor issue, but it’s unfortunate.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Andrew
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 14 Aug 2024, at 08:47, TengYao Chi <
>> kitingiao@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hello Andrew
>>>>>>>>>>>>>> Thank you for your thoughtful suggestions and getting the
>>>>>>>>>> discussion
>>>>>>>>>>>>> going.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> To AS1:
>>>>>>>>>>>>>> In the current scenario where the server generates the
>> UUID,
>>>> if
>>>>>>>>>> the
>>>>>>>>>>>>> client
>>>>>>>>>>>>>> shuts down before receiving the memberId generated by the
>> GC
>>>>>>>>>>>> (regardless
>>>>>>>>>>>>> of
>>>>>>>>>>>>>> whether it’s a graceful shutdown or not), the GC will
>> still
>>>> have
>>>>>>>>>> to
>>>>>>>>>>>> wait
>>>>>>>>>>>>>> for the heartbeat timeout because the client doesn’t know
>>> its
>>>>>>>>>>> memberId.
>>>>>>>>>>>>>> This KIP indeed cannot completely resolve the idempotency
>>>> issue,
>>>>>>>>>> but
>>>>>>>>>>> it
>>>>>>>>>>>>> can
>>>>>>>>>>>>>> better handle shutdown scenarios under normal
>> circumstances
>>>>>>>>>> because
>>>>>>>>>>> the
>>>>>>>>>>>>>> client always knows its memberId. Even if the client shuts
>>>> down
>>>>>>>>>>>>> immediately
>>>>>>>>>>>>>> after the initial heartbeat, as long as it performs a
>>> graceful
>>>>>>>>>>> shutdown
>>>>>>>>>>>>> and
>>>>>>>>>>>>>> sends a leave heartbeat, the GC can manage the situation
>> and
>>>>>>>>>> remove
>>>>>>>>>>> the
>>>>>>>>>>>>>> member. Therefore, the goal of this KIP is to address the
>>>> issue
>>>>>>>>>> where
>>>>>>>>>>>> the
>>>>>>>>>>>>>> GC has to wait for the heartbeat timeout due to the client
>>>>> leaving
>>>>>>>>>>>>> without
>>>>>>>>>>>>>> knowing its memberId, which leads to reduced throughput
>> and
>>>>>>>>>> limited
>>>>>>>>>>>>>> scalability.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The solution you suggest has also been proposed by David.
>>> The
>>>>>>>>>> concern
>>>>>>>>>>>>> with
>>>>>>>>>>>>>> this approach is that it introduces additional complexity
>>> for
>>>>>>>>>>>>>> compatibility, as the new server would not immediately add
>>> the
>>>>>>>>>> member
>>>>>>>>>>>> to
>>>>>>>>>>>>>> the group, while the old server would. This requires
>> clients
>>>> to
>>>>>>>>>>>>>> differentiate whether their memberId has been added to the
>>>> group
>>>>>>>>>> or
>>>>>>>>>>>> not,
>>>>>>>>>>>>>> which could result in unexpected logs.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>> TengYao
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Andrew Schofield <an...@live.com> 於 2024年8月14日
>>> 週三
>>>>>>>>>>>> 上午12:29寫道：
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi TengYao,
>>>>>>>>>>>>>>> Thanks for the KIP. I wonder if there’s a different way
>> to
>>>>> close
>>>>>>>>>>> what
>>>>>>>>>>>>>>> is quite a small window.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> AS1: It is true that the initial heartbeat is not
>>> idempotent,
>>>>> but
>>>>>>>>>>> this
>>>>>>>>>>>>>>> remains
>>>>>>>>>>>>>>> true with this KIP. It’s just differently not idempotent.
>>> If
>>>>> the
>>>>>>>>>>>> client
>>>>>>>>>>>>>>> makes its
>>>>>>>>>>>>>>> own member ID, sends a request and dies, the GC will
>> still
>>>> have
>>>>>>>>>>> added
>>>>>>>>>>>>>>> the member to the group and it will hang around until the
>>>>> session
>>>>>>>>>>>>> expires.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I wonder if the GC could still generate the member ID in
>>>>>>>>>> response to
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> first
>>>>>>>>>>>>>>> heartbeat, and put the member in a special PENDING state
>>> with
>>>>> no
>>>>>>>>>>>>>>> assignments until the client sends the next heartbeat,
>> thus
>>>>>>>>>>> confirming
>>>>>>>>>>>>> it
>>>>>>>>>>>>>>> has received the member ID. This would not be a protocol
>>>> change
>>>>>>>>>> at
>>>>>>>>>>>> all,
>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>> a change to the GC to keep a new member in the lobby
>> until
>>>> it’s
>>>>>>>>>>>>> comfirmed
>>>>>>>>>>>>>>> it knows its member ID.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Andrew
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On 13 Aug 2024, at 15:59, TengYao Chi <
>>> kitingiao@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi Chia-Ping,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks for review and suggestions.
>>>>>>>>>>>>>>>> I have updated the content of KIP accordingly.
>>>>>>>>>>>>>>>> Please take a look.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>> TengYao
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2024年8月13日 週二
>>>>> 下午9:45寫道：
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> hi TengYao
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> thanks for this KIP.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 1) could you please describe the before/after behavior
>> in
>>>> the
>>>>>>>>>>>>> "Proposed
>>>>>>>>>>>>>>>>> Changes" section? IIRC, current RPC allows HB having
>>> member
>>>>> id
>>>>>>>>>>>>>>> generated by
>>>>>>>>>>>>>>>>> client, right? If HB has no member ID, server will
>>> generate
>>>>> one
>>>>>>>>>>> and
>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>> return. The new behavior will enforce HB "must" have
>>> member
>>>>> ID.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 2) could you please write the version number explicitly
>>> in
>>>>> the
>>>>>>>>>> KIP
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 3) how new client code handle the old HB? Does it
>> always
>>>>>>>>>> generate
>>>>>>>>>>>>> member
>>>>>>>>>>>>>>>>> ID on client-side even though that is not restricted?
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On 2024/08/13 06:20:42 TengYao Chi wrote:
>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I would like to start a discussion thread on KIP-1082,
>>>> which
>>>>>>>>>>>> proposes
>>>>>>>>>>>>>>>>>> enabling id generation for clients over the
>>>>>>>>>>> ConsumerGroupHeartbeat
>>>>>>>>>>>>> RPC.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Here is the KIP Link: KIP-1082
>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>> 
>>>>> 
>>>> 
>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Please take a look and let me know what you think,
>> and I
>>>>> would
>>>>>>>>>>>>>>> appreciate
>>>>>>>>>>>>>>>>>> any suggestions and feedback.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>> TengYao
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>> 
>>>> 
>>> 
>> 

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[TengYao Chi <ki...@gmail.com>]

Hi David,

Thank you for pointing that out.
I have updated the content of the KIP based on Lianet's and your feedback.
Please take a look and let me know your thoughts.

Best regards,
TengYao

David Jacot <dj...@confluent.io.invalid> 於 2024年8月29日 週四 下午3:20寫道：

> Hi TengYao,
>
> Thanks for the update. I haven't fully read it yet but I will soon.
>
> LM4: This is incorrect. The consumer must keep its member id during its
> entire lifetime (until the process stops or dies). The protocol stipulates
> that a member must rejoin with the same member id and the member epoch set
> to zero when an FENCED_MEMBER_EPOCH occurs. This allows the member to
> resynchronize itself. We should not change this behavior. I think that we
> should see the client side generation id as an incarnation id of the
> application. It is generated once and kept until it stops or dies.
>
> Best,
> David
>
> On Thu, Aug 29, 2024 at 6:21 AM TengYao Chi <ki...@gmail.com> wrote:
>
> > Hello Lianet !
> >
> > Thanks for the reviews and suggestions!
> >
> > LM1: Indeed, we plan to enforce client-side ID generation in the future,
> > and it is not an alternative. I will change the title accordingly.
> >
> > LM2: Yes, that's the expectation. I will add that statement to the public
> > interface section.
> >
> > LM3: Thank you for the high-level perspective review. I think you're
> right;
> > our intention isn't very clear since it was placed at the end of the
> > section. I will try to rephrase that section to make it more obvious.
> >
> > LM4: Regarding the definition of "session" in this KIP, I believe it
> refers
> > to the period between the *first-time heartbeat* and when the *consumer
> > leaves the group* (whether through a graceful shutdown or a heartbeat
> > timeout). The consumer should reuse its UUID if it has been generated
> > before. The only situation in which it will regenerate the UUID is if the
> > coordinator finds that there is already a consumer with the same UUID.
> > IIRC, the coordinator should compare the member epochs, and the
> > later-joined consumer should be fenced off by the coordinator due to
> having
> > a lower member epoch. Once the consumer receives a `FENCED_MEMBER_EPOCH`
> > error, it will generate a new UUID and attempt to rejoin. I will clarify
> > this in the KIP.
> >
> > Thanks again for your reviews, I really appreciate it.
> >
> > Best regards,
> > TengYao
> >
> > Lianet M. <li...@gmail.com> 於 2024年8月28日 週三 下午7:12寫道：
> >
> > > Hello TengYao! Thanks for taking on this issue, we've been going around
> > it
> > > for a while.
> > >
> > > LM1: About the title of the KIP: "Enable ID Generation for Clients over
> > the
> > > ConsumerGroupHeartbeat RPC". I find it confusing because it hints that
> > > we're adding it as an alternative (which was discussed and discarded,
> in
> > > favour of really enforcing it). It's also missing the core change imo,
> > > which is "where" the generation happens. So, maybe more to the point
> with
> > > something along the lines of "Client-side generated ID for clients over
> > > ConsumerGroupHeartbeat RPC"?
> > >
> > > LM2: On the public interfaces section, the KIP states that "the server
> > will
> > > reject the request", but we should agree on the specific error type. I
> > > expect it should fail with an InvalidRequestException, is that the
> > > intention? (This was also suggested in the discussion thread before but
> > is
> > > not in the KIP).
> > >
> > > LM3. Related to my previous point, I find that to be the true
> > public-facing
> > > change (member ID mandatory at the protocol level), but it's only at
> the
> > > end of the Public interfaces changes, kind of lost among details of how
> > > we're going to do it. Should we rephrase that section with the actual
> > > change first, and the hows after (ex. Bumping the version is not the
> > > public-facing change in this case, it's just the mechanism to properly
> > > introduce our change)
> > >
> > > LM4. Regarding the lifetime of the UUID: the KIP states we will "Verify
> > > that the UUID remains consistent across all subsequent heartbeats
> during
> > > the session". What is this "session" referring to here? I would expect
> > that
> > > the UUID is associated to a consumer instance (generated for the
> consumer
> > > the first time it needs to send a HB if it doesn't have the UUID yet.
> > From
> > > there on, every time it needs to send a "first HB" again, it will reuse
> > its
> > > UUID, is that the intention? Note that we should consider that the same
> > > consumer instance may have many "first heartbeats", meaning heartbeats
> to
> > > join the group when it's not part of it (ex. consumer unsubscribe +
> > > subscribe, fenced, stale). Is this the intention or are you considering
> > the
> > > lifetime differently? We should clarify it in the KIP.
> > >
> > > Thanks!
> > >
> > > Lianet
> > >
> > > On Tue, Aug 27, 2024 at 2:27 AM TengYao Chi <ki...@gmail.com>
> wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I have revised this KIP multiple times based on the feedback from our
> > > > discussions.
> > > > I would greatly appreciate it if you could review it when you have
> the
> > > > time.
> > > > If there are no further comments or suggestions, I plan to proceed
> with
> > > > initiating a vote soon.
> > > >
> > > > Best regards,
> > > > TengYao
> > > >
> > > > TengYao Chi <ki...@gmail.com> 於 2024年8月23日 週五 下午2:43寫道：
> > > >
> > > > > Hi Andrew,
> > > > > Thank you for your previous feedback and insights.
> > > > > Your contributions have added valuable perspectives to the
> > discussions.
> > > > > And we also benefit from the comparison of different solutions.
> > > > > I’m also looking forward to seeing an initial version in KIP-932,
> as
> > it
> > > > > will provide a good reference for future implementations.
> > > > >
> > > > > Regarding your comment on AS2, I wanted to clarify that my
> > > specification
> > > > > references org.apache.kafka.common.Uuid.
> > > > > I believe we’re referring to the same class, and it might just be a
> > > small
> > > > > oversight due to the busy schedule.
> > > > >
> > > > > I want to express my gratitude once again for your many insightful
> > > > > comments, which have helped the discussion progress smoothly.
> > > > >
> > > > > Best regards,
> > > > > TengYao
> > > > >
> > > > >
> > > > > Andrew Schofield <an...@live.com> 於 2024年8月22日 週四
> > > 下午11:28寫道：
> > > > >
> > > > >> Hi TengYao,
> > > > >> I’ve been reading through the comments and I’m happy that the
> lobby
> > > > >> approach has not gained support.
> > > > >>
> > > > >> Assuming that this KIP is voted, I will be happy to change KIP-932
> > so
> > > > >> that it only supports client-side member ID generation. Because
> that
> > > KIP
> > > > >> is still
> > > > >> under development, I can do this in the first version of
> > > > >> ShareGroupHeartbeat.
> > > > >>
> > > > >> AS2: For the encoding section, I suppose the specific encoding
> which
> > > > >> is used is what org.apache.kafka.utils.Uuid uses.
> > > > >>
> > > > >> Thanks,
> > > > >> Andrew
> > > > >>
> > > > >> > On 14 Aug 2024, at 17:00, TengYao Chi <ki...@gmail.com>
> > wrote:
> > > > >> >
> > > > >> > Hello Apoorv,
> > > > >> > Thank you for your feedback.
> > > > >> > Regarding the questions you raised, unfortunately, this KIP
> cannot
> > > > >> > guarantee the order of heartbeats. As with many classic
> > distributed
> > > > >> system
> > > > >> > challenges, what we can do is make our best effort to ensure
> that
> > > > there
> > > > >> are
> > > > >> > no idle members or stale assignments under normal circumstances.
> > > > >> >
> > > > >> > As for the lobby approach, I’m not a fan of it because it
> requires
> > > > >> adding a
> > > > >> > mechanism to maintain client state within the ConsumerGroup,
> > which,
> > > in
> > > > >> my
> > > > >> > view, resembles something like a two-phase commit. This would
> > > > introduce
> > > > >> > more complexity than the proposal in this KIP, which is
> something
> > we
> > > > >> want
> > > > >> > to avoid. KIP-848 aims to simplify the existing protocol, and
> > while
> > > > the
> > > > >> > lobby approach is a good one, I believe it is not the right fit
> > for
> > > > this
> > > > >> > particular situation.
> > > > >> >
> > > > >> > Best regards,
> > > > >> > TengYao
> > > > >> >
> > > > >> > TengYao Chi <ki...@gmail.com> 於 2024年8月14日 週三 下午11:45寫道：
> > > > >> >
> > > > >> >> Hi David,
> > > > >> >>
> > > > >> >> I really appreciate your review and suggestions. As I am still
> > > > gaining
> > > > >> >> experience in writing KIPs, your input has been incredibly
> > > helpful. I
> > > > >> am
> > > > >> >> currently applying your suggestions to the KIP and will
> complete
> > it
> > > > as
> > > > >> soon
> > > > >> >> as possible.
> > > > >> >> Regarding the UUID part, I think we haven’t reached a
> conclusion
> > > > >> yet.(So
> > > > >> >> far according to this thread)
> > > > >> >> However, I will review the current implementation in the Kafka
> > > `Uuid`
> > > > >> >> class and include a brief specification in the KIP.
> > > > >> >>
> > > > >> >> Once again, thank you so much for your help.
> > > > >> >>
> > > > >> >> Best regards,
> > > > >> >> TengYao
> > > > >> >>
> > > > >> >> Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月14日 週三 下午11:14寫道：
> > > > >> >>
> > > > >> >>> hi Apoorv
> > > > >> >>>
> > > > >> >>>> As the memberId is now known to the client, and client might
> > send
> > > > the
> > > > >> >>> leave
> > > > >> >>> group heartbeat on shutdown prior to receiving the initial
> > > heartbeat
> > > > >> >>> response. If that's true then how do we guarantee that the 2
> > > > requests
> > > > >> to
> > > > >> >>> join and leave will be processed in order, which could still
> > leave
> > > > >> stale
> > > > >> >>> members or throw unknown member id exceptions?
> > > > >> >>>
> > > > >> >>> This is definitely a good question. the short answer: no
> > guarantee
> > > > but
> > > > >> >>> best
> > > > >> >>> efforts
> > > > >> >>>
> > > > >> >>> Please notice the root cause is "we have no enough time to
> wait
> > > > >> member id
> > > > >> >>> (response) when closing consumer". Sadly, we can' guarantee
> the
> > > > >> request
> > > > >> >>> order due to the same reason.
> > > > >> >>>
> > > > >> >>> However, in contrast to previous behavior, there is one big
> > > benefit
> > > > >> of new
> > > > >> >>> approach - we can try STONITH because we know the member id
> > > > >> >>>
> > > > >> >>> Best,
> > > > >> >>> Chia-Ping
> > > > >> >>>
> > > > >> >>>
> > > > >> >>> Apoorv Mittal <ap...@gmail.com> 於 2024年8月14日 週三
> > > 下午8:55寫道：
> > > > >> >>>
> > > > >> >>>> Hi TengYao,
> > > > >> >>>> Thanks for the KIP. Continuing on the point which Andrew
> > > mentioned
> > > > as
> > > > >> >>> AS1.
> > > > >> >>>>
> > > > >> >>>> As the memberId is now known to the client, and client might
> > send
> > > > the
> > > > >> >>> leave
> > > > >> >>>> group heartbeat on shutdown prior to receiving the initial
> > > > heartbeat
> > > > >> >>>> response. If that's true then how do we guarantee that the 2
> > > > >> requests to
> > > > >> >>>> join and leave will be processed in order, which could still
> > > leave
> > > > >> stale
> > > > >> >>>> members or throw unknown member id exceptions?
> > > > >> >>>>
> > > > >> >>>> Though the client side member id generation is helpful which
> > will
> > > > >> >>> represent
> > > > >> >>>> the same group perspective as from client and broker's end.
> > But I
> > > > >> think
> > > > >> >>> the
> > > > >> >>>> major concern we want to solve here is Stale Partition
> > > Assignments
> > > > >> which
> > > > >> >>>> might still exist with the new approach. I am leaning towards
> > the
> > > > >> >>>> suggestion mentioned by Andrew where partition assignment
> > > triggers
> > > > on
> > > > >> >>>> subsequent heartbeat when client acknowledges the initial
> > > > heartbeat,
> > > > >> >>>> delayed partition assignment.
> > > > >> >>>>
> > > > >> >>>> Though on a separate note, I have a different question. What
> > > > happens
> > > > >> >>> when
> > > > >> >>>> there is an issue with the client which sends the initial
> > > heartbeat
> > > > >> >>> without
> > > > >> >>>> memberId, then crashes and restarts? I think we must be
> > > > re-triggering
> > > > >> >>>> assignments and expiring members only after the heartbeat
> > session
> > > > >> >>> timeout?
> > > > >> >>>> If that's true then shall delayed partition assignment can
> help
> > > > >> benefit
> > > > >> >>> us
> > > > >> >>>> from this situation as well?
> > > > >> >>>>
> > > > >> >>>> Regards,
> > > > >> >>>> Apoorv Mittal
> > > > >> >>>>
> > > > >> >>>>
> > > > >> >>>> On Wed, Aug 14, 2024 at 12:51 PM David Jacot
> > > > >> >>> <dj...@confluent.io.invalid>
> > > > >> >>>> wrote:
> > > > >> >>>>
> > > > >> >>>>> Hi Andrew,
> > > > >> >>>>>
> > > > >> >>>>> Personally, I don't like the lobby approach. It makes things
> > > more
> > > > >> >>>>> complicated and it would require changing the records on the
> > > > server
> > > > >> >>> too.
> > > > >> >>>>> This is why I initially suggested the rejected alternative
> #2
> > > > which
> > > > >> is
> > > > >> >>>>> pretty close but also not perfect.
> > > > >> >>>>>
> > > > >> >>>>> I'd like to clarify one thing. The ConsumerGroupHeartbeat
> API
> > > > >> already
> > > > >> >>>>> supports generating the member id on the client so we don't
> > need
> > > > any
> > > > >> >>>>> conditional logic on the client side. This is actually what
> we
> > > > >> wanted
> > > > >> >>> to
> > > > >> >>>> do
> > > > >> >>>>> in the first place but the idea got pushed back by Magnus
> back
> > > > then
> > > > >> >>>> because
> > > > >> >>>>> generating uuid from librdkafka required a new dependency.
> It
> > > > turns
> > > > >> >>> out
> > > > >> >>>>> that librdkafka has that dependency today. In retrospect, we
> > > > should
> > > > >> >>> have
> > > > >> >>>>> pushed back on this. Long story short, we can just do it.
> The
> > > > >> >>> proposal in
> > > > >> >>>>> this KIP is to make the member id required in future
> versions.
> > > We
> > > > >> >>> could
> > > > >> >>>>> also decide not to do it and to keep supporting both
> > > approaches. I
> > > > >> >>> would
> > > > >> >>>>> also be fine with this.
> > > > >> >>>>>
> > > > >> >>>>> Best,
> > > > >> >>>>> David
> > > > >> >>>>>
> > > > >> >>>>> On Wed, Aug 14, 2024 at 12:30 PM Andrew Schofield <
> > > > >> >>>>> andrew_schofield@live.com>
> > > > >> >>>>> wrote:
> > > > >> >>>>>
> > > > >> >>>>>> Hi TengYao,
> > > > >> >>>>>> Thanks for your response. I’ll have just one more try to
> > > > persuade.
> > > > >> >>>>>> I feel that I will need to follow the approach with KIP-932
> > > when
> > > > >> >>> we’ve
> > > > >> >>>>>> made a decision, so I do have more than a passing interest
> in
> > > > this.
> > > > >> >>>>>>
> > > > >> >>>>>> A group member in the lobby is in the group, but it does
> not
> > > have
> > > > >> >>> any
> > > > >> >>>>>> assignments. A member of a consumer group can have no
> > assigned
> > > > >> >>>>>> partitions (such as 5 CG members subscribed to a topic
> with 4
> > > > >> >>>>> partitions),
> > > > >> >>>>>> so it’s a situation that consumer group members already
> > expect.
> > > > >> >>>>>>
> > > > >> >>>>>> One of Kafka’s strengths is the way that we handle API
> > > > versioning.
> > > > >> >>>>>> But, there is a cost - the behaviour is different depending
> > on
> > > > the
> > > > >> >>> RPC
> > > > >> >>>>>> version. KIP-848 is on the cusp of completion, but we’re
> > > already
> > > > >> >>> adding
> > > > >> >>>>>> conditional logic for v0/v1 for ConsumerGroupHeartbeat.
> > That’s
> > > a
> > > > >> >>> pity.
> > > > >> >>>>>> Only a minor issue, but it’s unfortunate.
> > > > >> >>>>>>
> > > > >> >>>>>> Thanks,
> > > > >> >>>>>> Andrew
> > > > >> >>>>>>
> > > > >> >>>>>>> On 14 Aug 2024, at 08:47, TengYao Chi <
> kitingiao@gmail.com>
> > > > >> >>> wrote:
> > > > >> >>>>>>>
> > > > >> >>>>>>> Hello Andrew
> > > > >> >>>>>>> Thank you for your thoughtful suggestions and getting the
> > > > >> >>> discussion
> > > > >> >>>>>> going.
> > > > >> >>>>>>>
> > > > >> >>>>>>> To AS1:
> > > > >> >>>>>>> In the current scenario where the server generates the
> UUID,
> > > if
> > > > >> >>> the
> > > > >> >>>>>> client
> > > > >> >>>>>>> shuts down before receiving the memberId generated by the
> GC
> > > > >> >>>>> (regardless
> > > > >> >>>>>> of
> > > > >> >>>>>>> whether it’s a graceful shutdown or not), the GC will
> still
> > > have
> > > > >> >>> to
> > > > >> >>>>> wait
> > > > >> >>>>>>> for the heartbeat timeout because the client doesn’t know
> > its
> > > > >> >>>> memberId.
> > > > >> >>>>>>> This KIP indeed cannot completely resolve the idempotency
> > > issue,
> > > > >> >>> but
> > > > >> >>>> it
> > > > >> >>>>>> can
> > > > >> >>>>>>> better handle shutdown scenarios under normal
> circumstances
> > > > >> >>> because
> > > > >> >>>> the
> > > > >> >>>>>>> client always knows its memberId. Even if the client shuts
> > > down
> > > > >> >>>>>> immediately
> > > > >> >>>>>>> after the initial heartbeat, as long as it performs a
> > graceful
> > > > >> >>>> shutdown
> > > > >> >>>>>> and
> > > > >> >>>>>>> sends a leave heartbeat, the GC can manage the situation
> and
> > > > >> >>> remove
> > > > >> >>>> the
> > > > >> >>>>>>> member. Therefore, the goal of this KIP is to address the
> > > issue
> > > > >> >>> where
> > > > >> >>>>> the
> > > > >> >>>>>>> GC has to wait for the heartbeat timeout due to the client
> > > > leaving
> > > > >> >>>>>> without
> > > > >> >>>>>>> knowing its memberId, which leads to reduced throughput
> and
> > > > >> >>> limited
> > > > >> >>>>>>> scalability.
> > > > >> >>>>>>>
> > > > >> >>>>>>> The solution you suggest has also been proposed by David.
> > The
> > > > >> >>> concern
> > > > >> >>>>>> with
> > > > >> >>>>>>> this approach is that it introduces additional complexity
> > for
> > > > >> >>>>>>> compatibility, as the new server would not immediately add
> > the
> > > > >> >>> member
> > > > >> >>>>> to
> > > > >> >>>>>>> the group, while the old server would. This requires
> clients
> > > to
> > > > >> >>>>>>> differentiate whether their memberId has been added to the
> > > group
> > > > >> >>> or
> > > > >> >>>>> not,
> > > > >> >>>>>>> which could result in unexpected logs.
> > > > >> >>>>>>>
> > > > >> >>>>>>> Best Regards,
> > > > >> >>>>>>> TengYao
> > > > >> >>>>>>>
> > > > >> >>>>>>> Andrew Schofield <an...@live.com> 於 2024年8月14日
> > 週三
> > > > >> >>>>> 上午12:29寫道：
> > > > >> >>>>>>>
> > > > >> >>>>>>>> Hi TengYao,
> > > > >> >>>>>>>> Thanks for the KIP. I wonder if there’s a different way
> to
> > > > close
> > > > >> >>>> what
> > > > >> >>>>>>>> is quite a small window.
> > > > >> >>>>>>>>
> > > > >> >>>>>>>> AS1: It is true that the initial heartbeat is not
> > idempotent,
> > > > but
> > > > >> >>>> this
> > > > >> >>>>>>>> remains
> > > > >> >>>>>>>> true with this KIP. It’s just differently not idempotent.
> > If
> > > > the
> > > > >> >>>>> client
> > > > >> >>>>>>>> makes its
> > > > >> >>>>>>>> own member ID, sends a request and dies, the GC will
> still
> > > have
> > > > >> >>>> added
> > > > >> >>>>>>>> the member to the group and it will hang around until the
> > > > session
> > > > >> >>>>>> expires.
> > > > >> >>>>>>>>
> > > > >> >>>>>>>> I wonder if the GC could still generate the member ID in
> > > > >> >>> response to
> > > > >> >>>>> the
> > > > >> >>>>>>>> first
> > > > >> >>>>>>>> heartbeat, and put the member in a special PENDING state
> > with
> > > > no
> > > > >> >>>>>>>> assignments until the client sends the next heartbeat,
> thus
> > > > >> >>>> confirming
> > > > >> >>>>>> it
> > > > >> >>>>>>>> has received the member ID. This would not be a protocol
> > > change
> > > > >> >>> at
> > > > >> >>>>> all,
> > > > >> >>>>>>>> just
> > > > >> >>>>>>>> a change to the GC to keep a new member in the lobby
> until
> > > it’s
> > > > >> >>>>>> comfirmed
> > > > >> >>>>>>>> it knows its member ID.
> > > > >> >>>>>>>>
> > > > >> >>>>>>>>
> > > > >> >>>>>>>> Thanks,
> > > > >> >>>>>>>> Andrew
> > > > >> >>>>>>>>
> > > > >> >>>>>>>>> On 13 Aug 2024, at 15:59, TengYao Chi <
> > kitingiao@gmail.com>
> > > > >> >>> wrote:
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>> Hi Chia-Ping,
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>> Thanks for review and suggestions.
> > > > >> >>>>>>>>> I have updated the content of KIP accordingly.
> > > > >> >>>>>>>>> Please take a look.
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>> Best regards,
> > > > >> >>>>>>>>> TengYao
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2024年8月13日 週二
> > > > 下午9:45寫道：
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>>> hi TengYao
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> thanks for this KIP.
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> 1) could you please describe the before/after behavior
> in
> > > the
> > > > >> >>>>>> "Proposed
> > > > >> >>>>>>>>>> Changes" section? IIRC, current RPC allows HB having
> > member
> > > > id
> > > > >> >>>>>>>> generated by
> > > > >> >>>>>>>>>> client, right? If HB has no member ID, server will
> > generate
> > > > one
> > > > >> >>>> and
> > > > >> >>>>>> then
> > > > >> >>>>>>>>>> return. The new behavior will enforce HB "must" have
> > member
> > > > ID.
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> 2) could you please write the version number explicitly
> > in
> > > > the
> > > > >> >>> KIP
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> 3) how new client code handle the old HB? Does it
> always
> > > > >> >>> generate
> > > > >> >>>>>> member
> > > > >> >>>>>>>>>> ID on client-side even though that is not restricted?
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> Best,
> > > > >> >>>>>>>>>> Chia-Ping
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> On 2024/08/13 06:20:42 TengYao Chi wrote:
> > > > >> >>>>>>>>>>> Hello everyone,
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> I would like to start a discussion thread on KIP-1082,
> > > which
> > > > >> >>>>> proposes
> > > > >> >>>>>>>>>>> enabling id generation for clients over the
> > > > >> >>>> ConsumerGroupHeartbeat
> > > > >> >>>>>> RPC.
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> Here is the KIP Link: KIP-1082
> > > > >> >>>>>>>>>>> <
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>
> > > > >> >>>>>>
> > > > >> >>>>>
> > > > >> >>>>
> > > > >> >>>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> Please take a look and let me know what you think,
> and I
> > > > would
> > > > >> >>>>>>>> appreciate
> > > > >> >>>>>>>>>>> any suggestions and feedback.
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> Best regards,
> > > > >> >>>>>>>>>>> TengYao
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>
> > > > >> >>>>>>>>
> > > > >> >>>>>>
> > > > >> >>>>>>
> > > > >> >>>>>
> > > > >> >>>>
> > > > >> >>>
> > > > >> >>
> > > > >>
> > > > >>
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[David Jacot <dj...@confluent.io.INVALID>]

Hi TengYao,

Thanks for the update. I haven't fully read it yet but I will soon.

LM4: This is incorrect. The consumer must keep its member id during its
entire lifetime (until the process stops or dies). The protocol stipulates
that a member must rejoin with the same member id and the member epoch set
to zero when an FENCED_MEMBER_EPOCH occurs. This allows the member to
resynchronize itself. We should not change this behavior. I think that we
should see the client side generation id as an incarnation id of the
application. It is generated once and kept until it stops or dies.

Best,
David

On Thu, Aug 29, 2024 at 6:21 AM TengYao Chi <ki...@gmail.com> wrote:

> Hello Lianet !
>
> Thanks for the reviews and suggestions!
>
> LM1: Indeed, we plan to enforce client-side ID generation in the future,
> and it is not an alternative. I will change the title accordingly.
>
> LM2: Yes, that's the expectation. I will add that statement to the public
> interface section.
>
> LM3: Thank you for the high-level perspective review. I think you're right;
> our intention isn't very clear since it was placed at the end of the
> section. I will try to rephrase that section to make it more obvious.
>
> LM4: Regarding the definition of "session" in this KIP, I believe it refers
> to the period between the *first-time heartbeat* and when the *consumer
> leaves the group* (whether through a graceful shutdown or a heartbeat
> timeout). The consumer should reuse its UUID if it has been generated
> before. The only situation in which it will regenerate the UUID is if the
> coordinator finds that there is already a consumer with the same UUID.
> IIRC, the coordinator should compare the member epochs, and the
> later-joined consumer should be fenced off by the coordinator due to having
> a lower member epoch. Once the consumer receives a `FENCED_MEMBER_EPOCH`
> error, it will generate a new UUID and attempt to rejoin. I will clarify
> this in the KIP.
>
> Thanks again for your reviews, I really appreciate it.
>
> Best regards,
> TengYao
>
> Lianet M. <li...@gmail.com> 於 2024年8月28日 週三 下午7:12寫道：
>
> > Hello TengYao! Thanks for taking on this issue, we've been going around
> it
> > for a while.
> >
> > LM1: About the title of the KIP: "Enable ID Generation for Clients over
> the
> > ConsumerGroupHeartbeat RPC". I find it confusing because it hints that
> > we're adding it as an alternative (which was discussed and discarded, in
> > favour of really enforcing it). It's also missing the core change imo,
> > which is "where" the generation happens. So, maybe more to the point with
> > something along the lines of "Client-side generated ID for clients over
> > ConsumerGroupHeartbeat RPC"?
> >
> > LM2: On the public interfaces section, the KIP states that "the server
> will
> > reject the request", but we should agree on the specific error type. I
> > expect it should fail with an InvalidRequestException, is that the
> > intention? (This was also suggested in the discussion thread before but
> is
> > not in the KIP).
> >
> > LM3. Related to my previous point, I find that to be the true
> public-facing
> > change (member ID mandatory at the protocol level), but it's only at the
> > end of the Public interfaces changes, kind of lost among details of how
> > we're going to do it. Should we rephrase that section with the actual
> > change first, and the hows after (ex. Bumping the version is not the
> > public-facing change in this case, it's just the mechanism to properly
> > introduce our change)
> >
> > LM4. Regarding the lifetime of the UUID: the KIP states we will "Verify
> > that the UUID remains consistent across all subsequent heartbeats during
> > the session". What is this "session" referring to here? I would expect
> that
> > the UUID is associated to a consumer instance (generated for the consumer
> > the first time it needs to send a HB if it doesn't have the UUID yet.
> From
> > there on, every time it needs to send a "first HB" again, it will reuse
> its
> > UUID, is that the intention? Note that we should consider that the same
> > consumer instance may have many "first heartbeats", meaning heartbeats to
> > join the group when it's not part of it (ex. consumer unsubscribe +
> > subscribe, fenced, stale). Is this the intention or are you considering
> the
> > lifetime differently? We should clarify it in the KIP.
> >
> > Thanks!
> >
> > Lianet
> >
> > On Tue, Aug 27, 2024 at 2:27 AM TengYao Chi <ki...@gmail.com> wrote:
> >
> > > Hi everyone,
> > >
> > > I have revised this KIP multiple times based on the feedback from our
> > > discussions.
> > > I would greatly appreciate it if you could review it when you have the
> > > time.
> > > If there are no further comments or suggestions, I plan to proceed with
> > > initiating a vote soon.
> > >
> > > Best regards,
> > > TengYao
> > >
> > > TengYao Chi <ki...@gmail.com> 於 2024年8月23日 週五 下午2:43寫道：
> > >
> > > > Hi Andrew,
> > > > Thank you for your previous feedback and insights.
> > > > Your contributions have added valuable perspectives to the
> discussions.
> > > > And we also benefit from the comparison of different solutions.
> > > > I’m also looking forward to seeing an initial version in KIP-932, as
> it
> > > > will provide a good reference for future implementations.
> > > >
> > > > Regarding your comment on AS2, I wanted to clarify that my
> > specification
> > > > references org.apache.kafka.common.Uuid.
> > > > I believe we’re referring to the same class, and it might just be a
> > small
> > > > oversight due to the busy schedule.
> > > >
> > > > I want to express my gratitude once again for your many insightful
> > > > comments, which have helped the discussion progress smoothly.
> > > >
> > > > Best regards,
> > > > TengYao
> > > >
> > > >
> > > > Andrew Schofield <an...@live.com> 於 2024年8月22日 週四
> > 下午11:28寫道：
> > > >
> > > >> Hi TengYao,
> > > >> I’ve been reading through the comments and I’m happy that the lobby
> > > >> approach has not gained support.
> > > >>
> > > >> Assuming that this KIP is voted, I will be happy to change KIP-932
> so
> > > >> that it only supports client-side member ID generation. Because that
> > KIP
> > > >> is still
> > > >> under development, I can do this in the first version of
> > > >> ShareGroupHeartbeat.
> > > >>
> > > >> AS2: For the encoding section, I suppose the specific encoding which
> > > >> is used is what org.apache.kafka.utils.Uuid uses.
> > > >>
> > > >> Thanks,
> > > >> Andrew
> > > >>
> > > >> > On 14 Aug 2024, at 17:00, TengYao Chi <ki...@gmail.com>
> wrote:
> > > >> >
> > > >> > Hello Apoorv,
> > > >> > Thank you for your feedback.
> > > >> > Regarding the questions you raised, unfortunately, this KIP cannot
> > > >> > guarantee the order of heartbeats. As with many classic
> distributed
> > > >> system
> > > >> > challenges, what we can do is make our best effort to ensure that
> > > there
> > > >> are
> > > >> > no idle members or stale assignments under normal circumstances.
> > > >> >
> > > >> > As for the lobby approach, I’m not a fan of it because it requires
> > > >> adding a
> > > >> > mechanism to maintain client state within the ConsumerGroup,
> which,
> > in
> > > >> my
> > > >> > view, resembles something like a two-phase commit. This would
> > > introduce
> > > >> > more complexity than the proposal in this KIP, which is something
> we
> > > >> want
> > > >> > to avoid. KIP-848 aims to simplify the existing protocol, and
> while
> > > the
> > > >> > lobby approach is a good one, I believe it is not the right fit
> for
> > > this
> > > >> > particular situation.
> > > >> >
> > > >> > Best regards,
> > > >> > TengYao
> > > >> >
> > > >> > TengYao Chi <ki...@gmail.com> 於 2024年8月14日 週三 下午11:45寫道：
> > > >> >
> > > >> >> Hi David,
> > > >> >>
> > > >> >> I really appreciate your review and suggestions. As I am still
> > > gaining
> > > >> >> experience in writing KIPs, your input has been incredibly
> > helpful. I
> > > >> am
> > > >> >> currently applying your suggestions to the KIP and will complete
> it
> > > as
> > > >> soon
> > > >> >> as possible.
> > > >> >> Regarding the UUID part, I think we haven’t reached a conclusion
> > > >> yet.(So
> > > >> >> far according to this thread)
> > > >> >> However, I will review the current implementation in the Kafka
> > `Uuid`
> > > >> >> class and include a brief specification in the KIP.
> > > >> >>
> > > >> >> Once again, thank you so much for your help.
> > > >> >>
> > > >> >> Best regards,
> > > >> >> TengYao
> > > >> >>
> > > >> >> Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月14日 週三 下午11:14寫道：
> > > >> >>
> > > >> >>> hi Apoorv
> > > >> >>>
> > > >> >>>> As the memberId is now known to the client, and client might
> send
> > > the
> > > >> >>> leave
> > > >> >>> group heartbeat on shutdown prior to receiving the initial
> > heartbeat
> > > >> >>> response. If that's true then how do we guarantee that the 2
> > > requests
> > > >> to
> > > >> >>> join and leave will be processed in order, which could still
> leave
> > > >> stale
> > > >> >>> members or throw unknown member id exceptions?
> > > >> >>>
> > > >> >>> This is definitely a good question. the short answer: no
> guarantee
> > > but
> > > >> >>> best
> > > >> >>> efforts
> > > >> >>>
> > > >> >>> Please notice the root cause is "we have no enough time to wait
> > > >> member id
> > > >> >>> (response) when closing consumer". Sadly, we can' guarantee the
> > > >> request
> > > >> >>> order due to the same reason.
> > > >> >>>
> > > >> >>> However, in contrast to previous behavior, there is one big
> > benefit
> > > >> of new
> > > >> >>> approach - we can try STONITH because we know the member id
> > > >> >>>
> > > >> >>> Best,
> > > >> >>> Chia-Ping
> > > >> >>>
> > > >> >>>
> > > >> >>> Apoorv Mittal <ap...@gmail.com> 於 2024年8月14日 週三
> > 下午8:55寫道：
> > > >> >>>
> > > >> >>>> Hi TengYao,
> > > >> >>>> Thanks for the KIP. Continuing on the point which Andrew
> > mentioned
> > > as
> > > >> >>> AS1.
> > > >> >>>>
> > > >> >>>> As the memberId is now known to the client, and client might
> send
> > > the
> > > >> >>> leave
> > > >> >>>> group heartbeat on shutdown prior to receiving the initial
> > > heartbeat
> > > >> >>>> response. If that's true then how do we guarantee that the 2
> > > >> requests to
> > > >> >>>> join and leave will be processed in order, which could still
> > leave
> > > >> stale
> > > >> >>>> members or throw unknown member id exceptions?
> > > >> >>>>
> > > >> >>>> Though the client side member id generation is helpful which
> will
> > > >> >>> represent
> > > >> >>>> the same group perspective as from client and broker's end.
> But I
> > > >> think
> > > >> >>> the
> > > >> >>>> major concern we want to solve here is Stale Partition
> > Assignments
> > > >> which
> > > >> >>>> might still exist with the new approach. I am leaning towards
> the
> > > >> >>>> suggestion mentioned by Andrew where partition assignment
> > triggers
> > > on
> > > >> >>>> subsequent heartbeat when client acknowledges the initial
> > > heartbeat,
> > > >> >>>> delayed partition assignment.
> > > >> >>>>
> > > >> >>>> Though on a separate note, I have a different question. What
> > > happens
> > > >> >>> when
> > > >> >>>> there is an issue with the client which sends the initial
> > heartbeat
> > > >> >>> without
> > > >> >>>> memberId, then crashes and restarts? I think we must be
> > > re-triggering
> > > >> >>>> assignments and expiring members only after the heartbeat
> session
> > > >> >>> timeout?
> > > >> >>>> If that's true then shall delayed partition assignment can help
> > > >> benefit
> > > >> >>> us
> > > >> >>>> from this situation as well?
> > > >> >>>>
> > > >> >>>> Regards,
> > > >> >>>> Apoorv Mittal
> > > >> >>>>
> > > >> >>>>
> > > >> >>>> On Wed, Aug 14, 2024 at 12:51 PM David Jacot
> > > >> >>> <dj...@confluent.io.invalid>
> > > >> >>>> wrote:
> > > >> >>>>
> > > >> >>>>> Hi Andrew,
> > > >> >>>>>
> > > >> >>>>> Personally, I don't like the lobby approach. It makes things
> > more
> > > >> >>>>> complicated and it would require changing the records on the
> > > server
> > > >> >>> too.
> > > >> >>>>> This is why I initially suggested the rejected alternative #2
> > > which
> > > >> is
> > > >> >>>>> pretty close but also not perfect.
> > > >> >>>>>
> > > >> >>>>> I'd like to clarify one thing. The ConsumerGroupHeartbeat API
> > > >> already
> > > >> >>>>> supports generating the member id on the client so we don't
> need
> > > any
> > > >> >>>>> conditional logic on the client side. This is actually what we
> > > >> wanted
> > > >> >>> to
> > > >> >>>> do
> > > >> >>>>> in the first place but the idea got pushed back by Magnus back
> > > then
> > > >> >>>> because
> > > >> >>>>> generating uuid from librdkafka required a new dependency. It
> > > turns
> > > >> >>> out
> > > >> >>>>> that librdkafka has that dependency today. In retrospect, we
> > > should
> > > >> >>> have
> > > >> >>>>> pushed back on this. Long story short, we can just do it. The
> > > >> >>> proposal in
> > > >> >>>>> this KIP is to make the member id required in future versions.
> > We
> > > >> >>> could
> > > >> >>>>> also decide not to do it and to keep supporting both
> > approaches. I
> > > >> >>> would
> > > >> >>>>> also be fine with this.
> > > >> >>>>>
> > > >> >>>>> Best,
> > > >> >>>>> David
> > > >> >>>>>
> > > >> >>>>> On Wed, Aug 14, 2024 at 12:30 PM Andrew Schofield <
> > > >> >>>>> andrew_schofield@live.com>
> > > >> >>>>> wrote:
> > > >> >>>>>
> > > >> >>>>>> Hi TengYao,
> > > >> >>>>>> Thanks for your response. I’ll have just one more try to
> > > persuade.
> > > >> >>>>>> I feel that I will need to follow the approach with KIP-932
> > when
> > > >> >>> we’ve
> > > >> >>>>>> made a decision, so I do have more than a passing interest in
> > > this.
> > > >> >>>>>>
> > > >> >>>>>> A group member in the lobby is in the group, but it does not
> > have
> > > >> >>> any
> > > >> >>>>>> assignments. A member of a consumer group can have no
> assigned
> > > >> >>>>>> partitions (such as 5 CG members subscribed to a topic with 4
> > > >> >>>>> partitions),
> > > >> >>>>>> so it’s a situation that consumer group members already
> expect.
> > > >> >>>>>>
> > > >> >>>>>> One of Kafka’s strengths is the way that we handle API
> > > versioning.
> > > >> >>>>>> But, there is a cost - the behaviour is different depending
> on
> > > the
> > > >> >>> RPC
> > > >> >>>>>> version. KIP-848 is on the cusp of completion, but we’re
> > already
> > > >> >>> adding
> > > >> >>>>>> conditional logic for v0/v1 for ConsumerGroupHeartbeat.
> That’s
> > a
> > > >> >>> pity.
> > > >> >>>>>> Only a minor issue, but it’s unfortunate.
> > > >> >>>>>>
> > > >> >>>>>> Thanks,
> > > >> >>>>>> Andrew
> > > >> >>>>>>
> > > >> >>>>>>> On 14 Aug 2024, at 08:47, TengYao Chi <ki...@gmail.com>
> > > >> >>> wrote:
> > > >> >>>>>>>
> > > >> >>>>>>> Hello Andrew
> > > >> >>>>>>> Thank you for your thoughtful suggestions and getting the
> > > >> >>> discussion
> > > >> >>>>>> going.
> > > >> >>>>>>>
> > > >> >>>>>>> To AS1:
> > > >> >>>>>>> In the current scenario where the server generates the UUID,
> > if
> > > >> >>> the
> > > >> >>>>>> client
> > > >> >>>>>>> shuts down before receiving the memberId generated by the GC
> > > >> >>>>> (regardless
> > > >> >>>>>> of
> > > >> >>>>>>> whether it’s a graceful shutdown or not), the GC will still
> > have
> > > >> >>> to
> > > >> >>>>> wait
> > > >> >>>>>>> for the heartbeat timeout because the client doesn’t know
> its
> > > >> >>>> memberId.
> > > >> >>>>>>> This KIP indeed cannot completely resolve the idempotency
> > issue,
> > > >> >>> but
> > > >> >>>> it
> > > >> >>>>>> can
> > > >> >>>>>>> better handle shutdown scenarios under normal circumstances
> > > >> >>> because
> > > >> >>>> the
> > > >> >>>>>>> client always knows its memberId. Even if the client shuts
> > down
> > > >> >>>>>> immediately
> > > >> >>>>>>> after the initial heartbeat, as long as it performs a
> graceful
> > > >> >>>> shutdown
> > > >> >>>>>> and
> > > >> >>>>>>> sends a leave heartbeat, the GC can manage the situation and
> > > >> >>> remove
> > > >> >>>> the
> > > >> >>>>>>> member. Therefore, the goal of this KIP is to address the
> > issue
> > > >> >>> where
> > > >> >>>>> the
> > > >> >>>>>>> GC has to wait for the heartbeat timeout due to the client
> > > leaving
> > > >> >>>>>> without
> > > >> >>>>>>> knowing its memberId, which leads to reduced throughput and
> > > >> >>> limited
> > > >> >>>>>>> scalability.
> > > >> >>>>>>>
> > > >> >>>>>>> The solution you suggest has also been proposed by David.
> The
> > > >> >>> concern
> > > >> >>>>>> with
> > > >> >>>>>>> this approach is that it introduces additional complexity
> for
> > > >> >>>>>>> compatibility, as the new server would not immediately add
> the
> > > >> >>> member
> > > >> >>>>> to
> > > >> >>>>>>> the group, while the old server would. This requires clients
> > to
> > > >> >>>>>>> differentiate whether their memberId has been added to the
> > group
> > > >> >>> or
> > > >> >>>>> not,
> > > >> >>>>>>> which could result in unexpected logs.
> > > >> >>>>>>>
> > > >> >>>>>>> Best Regards,
> > > >> >>>>>>> TengYao
> > > >> >>>>>>>
> > > >> >>>>>>> Andrew Schofield <an...@live.com> 於 2024年8月14日
> 週三
> > > >> >>>>> 上午12:29寫道：
> > > >> >>>>>>>
> > > >> >>>>>>>> Hi TengYao,
> > > >> >>>>>>>> Thanks for the KIP. I wonder if there’s a different way to
> > > close
> > > >> >>>> what
> > > >> >>>>>>>> is quite a small window.
> > > >> >>>>>>>>
> > > >> >>>>>>>> AS1: It is true that the initial heartbeat is not
> idempotent,
> > > but
> > > >> >>>> this
> > > >> >>>>>>>> remains
> > > >> >>>>>>>> true with this KIP. It’s just differently not idempotent.
> If
> > > the
> > > >> >>>>> client
> > > >> >>>>>>>> makes its
> > > >> >>>>>>>> own member ID, sends a request and dies, the GC will still
> > have
> > > >> >>>> added
> > > >> >>>>>>>> the member to the group and it will hang around until the
> > > session
> > > >> >>>>>> expires.
> > > >> >>>>>>>>
> > > >> >>>>>>>> I wonder if the GC could still generate the member ID in
> > > >> >>> response to
> > > >> >>>>> the
> > > >> >>>>>>>> first
> > > >> >>>>>>>> heartbeat, and put the member in a special PENDING state
> with
> > > no
> > > >> >>>>>>>> assignments until the client sends the next heartbeat, thus
> > > >> >>>> confirming
> > > >> >>>>>> it
> > > >> >>>>>>>> has received the member ID. This would not be a protocol
> > change
> > > >> >>> at
> > > >> >>>>> all,
> > > >> >>>>>>>> just
> > > >> >>>>>>>> a change to the GC to keep a new member in the lobby until
> > it’s
> > > >> >>>>>> comfirmed
> > > >> >>>>>>>> it knows its member ID.
> > > >> >>>>>>>>
> > > >> >>>>>>>>
> > > >> >>>>>>>> Thanks,
> > > >> >>>>>>>> Andrew
> > > >> >>>>>>>>
> > > >> >>>>>>>>> On 13 Aug 2024, at 15:59, TengYao Chi <
> kitingiao@gmail.com>
> > > >> >>> wrote:
> > > >> >>>>>>>>>
> > > >> >>>>>>>>> Hi Chia-Ping,
> > > >> >>>>>>>>>
> > > >> >>>>>>>>> Thanks for review and suggestions.
> > > >> >>>>>>>>> I have updated the content of KIP accordingly.
> > > >> >>>>>>>>> Please take a look.
> > > >> >>>>>>>>>
> > > >> >>>>>>>>> Best regards,
> > > >> >>>>>>>>> TengYao
> > > >> >>>>>>>>>
> > > >> >>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2024年8月13日 週二
> > > 下午9:45寫道：
> > > >> >>>>>>>>>
> > > >> >>>>>>>>>> hi TengYao
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> thanks for this KIP.
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> 1) could you please describe the before/after behavior in
> > the
> > > >> >>>>>> "Proposed
> > > >> >>>>>>>>>> Changes" section? IIRC, current RPC allows HB having
> member
> > > id
> > > >> >>>>>>>> generated by
> > > >> >>>>>>>>>> client, right? If HB has no member ID, server will
> generate
> > > one
> > > >> >>>> and
> > > >> >>>>>> then
> > > >> >>>>>>>>>> return. The new behavior will enforce HB "must" have
> member
> > > ID.
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> 2) could you please write the version number explicitly
> in
> > > the
> > > >> >>> KIP
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> 3) how new client code handle the old HB? Does it always
> > > >> >>> generate
> > > >> >>>>>> member
> > > >> >>>>>>>>>> ID on client-side even though that is not restricted?
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> Best,
> > > >> >>>>>>>>>> Chia-Ping
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> On 2024/08/13 06:20:42 TengYao Chi wrote:
> > > >> >>>>>>>>>>> Hello everyone,
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> I would like to start a discussion thread on KIP-1082,
> > which
> > > >> >>>>> proposes
> > > >> >>>>>>>>>>> enabling id generation for clients over the
> > > >> >>>> ConsumerGroupHeartbeat
> > > >> >>>>>> RPC.
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> Here is the KIP Link: KIP-1082
> > > >> >>>>>>>>>>> <
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>
> > > >> >>>>>>
> > > >> >>>>>
> > > >> >>>>
> > > >> >>>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> Please take a look and let me know what you think, and I
> > > would
> > > >> >>>>>>>> appreciate
> > > >> >>>>>>>>>>> any suggestions and feedback.
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> Best regards,
> > > >> >>>>>>>>>>> TengYao
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>
> > > >> >>>>>>>>
> > > >> >>>>>>
> > > >> >>>>>>
> > > >> >>>>>
> > > >> >>>>
> > > >> >>>
> > > >> >>
> > > >>
> > > >>
> > >
> >
>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[TengYao Chi <ki...@gmail.com>]

Hello Lianet !

Thanks for the reviews and suggestions!

LM1: Indeed, we plan to enforce client-side ID generation in the future,
and it is not an alternative. I will change the title accordingly.

LM2: Yes, that's the expectation. I will add that statement to the public
interface section.

LM3: Thank you for the high-level perspective review. I think you're right;
our intention isn't very clear since it was placed at the end of the
section. I will try to rephrase that section to make it more obvious.

LM4: Regarding the definition of "session" in this KIP, I believe it refers
to the period between the *first-time heartbeat* and when the *consumer
leaves the group* (whether through a graceful shutdown or a heartbeat
timeout). The consumer should reuse its UUID if it has been generated
before. The only situation in which it will regenerate the UUID is if the
coordinator finds that there is already a consumer with the same UUID.
IIRC, the coordinator should compare the member epochs, and the
later-joined consumer should be fenced off by the coordinator due to having
a lower member epoch. Once the consumer receives a `FENCED_MEMBER_EPOCH`
error, it will generate a new UUID and attempt to rejoin. I will clarify
this in the KIP.

Thanks again for your reviews, I really appreciate it.

Best regards,
TengYao

Lianet M. <li...@gmail.com> 於 2024年8月28日 週三 下午7:12寫道：

> Hello TengYao! Thanks for taking on this issue, we've been going around it
> for a while.
>
> LM1: About the title of the KIP: "Enable ID Generation for Clients over the
> ConsumerGroupHeartbeat RPC". I find it confusing because it hints that
> we're adding it as an alternative (which was discussed and discarded, in
> favour of really enforcing it). It's also missing the core change imo,
> which is "where" the generation happens. So, maybe more to the point with
> something along the lines of "Client-side generated ID for clients over
> ConsumerGroupHeartbeat RPC"?
>
> LM2: On the public interfaces section, the KIP states that "the server will
> reject the request", but we should agree on the specific error type. I
> expect it should fail with an InvalidRequestException, is that the
> intention? (This was also suggested in the discussion thread before but is
> not in the KIP).
>
> LM3. Related to my previous point, I find that to be the true public-facing
> change (member ID mandatory at the protocol level), but it's only at the
> end of the Public interfaces changes, kind of lost among details of how
> we're going to do it. Should we rephrase that section with the actual
> change first, and the hows after (ex. Bumping the version is not the
> public-facing change in this case, it's just the mechanism to properly
> introduce our change)
>
> LM4. Regarding the lifetime of the UUID: the KIP states we will "Verify
> that the UUID remains consistent across all subsequent heartbeats during
> the session". What is this "session" referring to here? I would expect that
> the UUID is associated to a consumer instance (generated for the consumer
> the first time it needs to send a HB if it doesn't have the UUID yet. From
> there on, every time it needs to send a "first HB" again, it will reuse its
> UUID, is that the intention? Note that we should consider that the same
> consumer instance may have many "first heartbeats", meaning heartbeats to
> join the group when it's not part of it (ex. consumer unsubscribe +
> subscribe, fenced, stale). Is this the intention or are you considering the
> lifetime differently? We should clarify it in the KIP.
>
> Thanks!
>
> Lianet
>
> On Tue, Aug 27, 2024 at 2:27 AM TengYao Chi <ki...@gmail.com> wrote:
>
> > Hi everyone,
> >
> > I have revised this KIP multiple times based on the feedback from our
> > discussions.
> > I would greatly appreciate it if you could review it when you have the
> > time.
> > If there are no further comments or suggestions, I plan to proceed with
> > initiating a vote soon.
> >
> > Best regards,
> > TengYao
> >
> > TengYao Chi <ki...@gmail.com> 於 2024年8月23日 週五 下午2:43寫道：
> >
> > > Hi Andrew,
> > > Thank you for your previous feedback and insights.
> > > Your contributions have added valuable perspectives to the discussions.
> > > And we also benefit from the comparison of different solutions.
> > > I’m also looking forward to seeing an initial version in KIP-932, as it
> > > will provide a good reference for future implementations.
> > >
> > > Regarding your comment on AS2, I wanted to clarify that my
> specification
> > > references org.apache.kafka.common.Uuid.
> > > I believe we’re referring to the same class, and it might just be a
> small
> > > oversight due to the busy schedule.
> > >
> > > I want to express my gratitude once again for your many insightful
> > > comments, which have helped the discussion progress smoothly.
> > >
> > > Best regards,
> > > TengYao
> > >
> > >
> > > Andrew Schofield <an...@live.com> 於 2024年8月22日 週四
> 下午11:28寫道：
> > >
> > >> Hi TengYao,
> > >> I’ve been reading through the comments and I’m happy that the lobby
> > >> approach has not gained support.
> > >>
> > >> Assuming that this KIP is voted, I will be happy to change KIP-932 so
> > >> that it only supports client-side member ID generation. Because that
> KIP
> > >> is still
> > >> under development, I can do this in the first version of
> > >> ShareGroupHeartbeat.
> > >>
> > >> AS2: For the encoding section, I suppose the specific encoding which
> > >> is used is what org.apache.kafka.utils.Uuid uses.
> > >>
> > >> Thanks,
> > >> Andrew
> > >>
> > >> > On 14 Aug 2024, at 17:00, TengYao Chi <ki...@gmail.com> wrote:
> > >> >
> > >> > Hello Apoorv,
> > >> > Thank you for your feedback.
> > >> > Regarding the questions you raised, unfortunately, this KIP cannot
> > >> > guarantee the order of heartbeats. As with many classic distributed
> > >> system
> > >> > challenges, what we can do is make our best effort to ensure that
> > there
> > >> are
> > >> > no idle members or stale assignments under normal circumstances.
> > >> >
> > >> > As for the lobby approach, I’m not a fan of it because it requires
> > >> adding a
> > >> > mechanism to maintain client state within the ConsumerGroup, which,
> in
> > >> my
> > >> > view, resembles something like a two-phase commit. This would
> > introduce
> > >> > more complexity than the proposal in this KIP, which is something we
> > >> want
> > >> > to avoid. KIP-848 aims to simplify the existing protocol, and while
> > the
> > >> > lobby approach is a good one, I believe it is not the right fit for
> > this
> > >> > particular situation.
> > >> >
> > >> > Best regards,
> > >> > TengYao
> > >> >
> > >> > TengYao Chi <ki...@gmail.com> 於 2024年8月14日 週三 下午11:45寫道：
> > >> >
> > >> >> Hi David,
> > >> >>
> > >> >> I really appreciate your review and suggestions. As I am still
> > gaining
> > >> >> experience in writing KIPs, your input has been incredibly
> helpful. I
> > >> am
> > >> >> currently applying your suggestions to the KIP and will complete it
> > as
> > >> soon
> > >> >> as possible.
> > >> >> Regarding the UUID part, I think we haven’t reached a conclusion
> > >> yet.(So
> > >> >> far according to this thread)
> > >> >> However, I will review the current implementation in the Kafka
> `Uuid`
> > >> >> class and include a brief specification in the KIP.
> > >> >>
> > >> >> Once again, thank you so much for your help.
> > >> >>
> > >> >> Best regards,
> > >> >> TengYao
> > >> >>
> > >> >> Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月14日 週三 下午11:14寫道：
> > >> >>
> > >> >>> hi Apoorv
> > >> >>>
> > >> >>>> As the memberId is now known to the client, and client might send
> > the
> > >> >>> leave
> > >> >>> group heartbeat on shutdown prior to receiving the initial
> heartbeat
> > >> >>> response. If that's true then how do we guarantee that the 2
> > requests
> > >> to
> > >> >>> join and leave will be processed in order, which could still leave
> > >> stale
> > >> >>> members or throw unknown member id exceptions?
> > >> >>>
> > >> >>> This is definitely a good question. the short answer: no guarantee
> > but
> > >> >>> best
> > >> >>> efforts
> > >> >>>
> > >> >>> Please notice the root cause is "we have no enough time to wait
> > >> member id
> > >> >>> (response) when closing consumer". Sadly, we can' guarantee the
> > >> request
> > >> >>> order due to the same reason.
> > >> >>>
> > >> >>> However, in contrast to previous behavior, there is one big
> benefit
> > >> of new
> > >> >>> approach - we can try STONITH because we know the member id
> > >> >>>
> > >> >>> Best,
> > >> >>> Chia-Ping
> > >> >>>
> > >> >>>
> > >> >>> Apoorv Mittal <ap...@gmail.com> 於 2024年8月14日 週三
> 下午8:55寫道：
> > >> >>>
> > >> >>>> Hi TengYao,
> > >> >>>> Thanks for the KIP. Continuing on the point which Andrew
> mentioned
> > as
> > >> >>> AS1.
> > >> >>>>
> > >> >>>> As the memberId is now known to the client, and client might send
> > the
> > >> >>> leave
> > >> >>>> group heartbeat on shutdown prior to receiving the initial
> > heartbeat
> > >> >>>> response. If that's true then how do we guarantee that the 2
> > >> requests to
> > >> >>>> join and leave will be processed in order, which could still
> leave
> > >> stale
> > >> >>>> members or throw unknown member id exceptions?
> > >> >>>>
> > >> >>>> Though the client side member id generation is helpful which will
> > >> >>> represent
> > >> >>>> the same group perspective as from client and broker's end. But I
> > >> think
> > >> >>> the
> > >> >>>> major concern we want to solve here is Stale Partition
> Assignments
> > >> which
> > >> >>>> might still exist with the new approach. I am leaning towards the
> > >> >>>> suggestion mentioned by Andrew where partition assignment
> triggers
> > on
> > >> >>>> subsequent heartbeat when client acknowledges the initial
> > heartbeat,
> > >> >>>> delayed partition assignment.
> > >> >>>>
> > >> >>>> Though on a separate note, I have a different question. What
> > happens
> > >> >>> when
> > >> >>>> there is an issue with the client which sends the initial
> heartbeat
> > >> >>> without
> > >> >>>> memberId, then crashes and restarts? I think we must be
> > re-triggering
> > >> >>>> assignments and expiring members only after the heartbeat session
> > >> >>> timeout?
> > >> >>>> If that's true then shall delayed partition assignment can help
> > >> benefit
> > >> >>> us
> > >> >>>> from this situation as well?
> > >> >>>>
> > >> >>>> Regards,
> > >> >>>> Apoorv Mittal
> > >> >>>>
> > >> >>>>
> > >> >>>> On Wed, Aug 14, 2024 at 12:51 PM David Jacot
> > >> >>> <dj...@confluent.io.invalid>
> > >> >>>> wrote:
> > >> >>>>
> > >> >>>>> Hi Andrew,
> > >> >>>>>
> > >> >>>>> Personally, I don't like the lobby approach. It makes things
> more
> > >> >>>>> complicated and it would require changing the records on the
> > server
> > >> >>> too.
> > >> >>>>> This is why I initially suggested the rejected alternative #2
> > which
> > >> is
> > >> >>>>> pretty close but also not perfect.
> > >> >>>>>
> > >> >>>>> I'd like to clarify one thing. The ConsumerGroupHeartbeat API
> > >> already
> > >> >>>>> supports generating the member id on the client so we don't need
> > any
> > >> >>>>> conditional logic on the client side. This is actually what we
> > >> wanted
> > >> >>> to
> > >> >>>> do
> > >> >>>>> in the first place but the idea got pushed back by Magnus back
> > then
> > >> >>>> because
> > >> >>>>> generating uuid from librdkafka required a new dependency. It
> > turns
> > >> >>> out
> > >> >>>>> that librdkafka has that dependency today. In retrospect, we
> > should
> > >> >>> have
> > >> >>>>> pushed back on this. Long story short, we can just do it. The
> > >> >>> proposal in
> > >> >>>>> this KIP is to make the member id required in future versions.
> We
> > >> >>> could
> > >> >>>>> also decide not to do it and to keep supporting both
> approaches. I
> > >> >>> would
> > >> >>>>> also be fine with this.
> > >> >>>>>
> > >> >>>>> Best,
> > >> >>>>> David
> > >> >>>>>
> > >> >>>>> On Wed, Aug 14, 2024 at 12:30 PM Andrew Schofield <
> > >> >>>>> andrew_schofield@live.com>
> > >> >>>>> wrote:
> > >> >>>>>
> > >> >>>>>> Hi TengYao,
> > >> >>>>>> Thanks for your response. I’ll have just one more try to
> > persuade.
> > >> >>>>>> I feel that I will need to follow the approach with KIP-932
> when
> > >> >>> we’ve
> > >> >>>>>> made a decision, so I do have more than a passing interest in
> > this.
> > >> >>>>>>
> > >> >>>>>> A group member in the lobby is in the group, but it does not
> have
> > >> >>> any
> > >> >>>>>> assignments. A member of a consumer group can have no assigned
> > >> >>>>>> partitions (such as 5 CG members subscribed to a topic with 4
> > >> >>>>> partitions),
> > >> >>>>>> so it’s a situation that consumer group members already expect.
> > >> >>>>>>
> > >> >>>>>> One of Kafka’s strengths is the way that we handle API
> > versioning.
> > >> >>>>>> But, there is a cost - the behaviour is different depending on
> > the
> > >> >>> RPC
> > >> >>>>>> version. KIP-848 is on the cusp of completion, but we’re
> already
> > >> >>> adding
> > >> >>>>>> conditional logic for v0/v1 for ConsumerGroupHeartbeat. That’s
> a
> > >> >>> pity.
> > >> >>>>>> Only a minor issue, but it’s unfortunate.
> > >> >>>>>>
> > >> >>>>>> Thanks,
> > >> >>>>>> Andrew
> > >> >>>>>>
> > >> >>>>>>> On 14 Aug 2024, at 08:47, TengYao Chi <ki...@gmail.com>
> > >> >>> wrote:
> > >> >>>>>>>
> > >> >>>>>>> Hello Andrew
> > >> >>>>>>> Thank you for your thoughtful suggestions and getting the
> > >> >>> discussion
> > >> >>>>>> going.
> > >> >>>>>>>
> > >> >>>>>>> To AS1:
> > >> >>>>>>> In the current scenario where the server generates the UUID,
> if
> > >> >>> the
> > >> >>>>>> client
> > >> >>>>>>> shuts down before receiving the memberId generated by the GC
> > >> >>>>> (regardless
> > >> >>>>>> of
> > >> >>>>>>> whether it’s a graceful shutdown or not), the GC will still
> have
> > >> >>> to
> > >> >>>>> wait
> > >> >>>>>>> for the heartbeat timeout because the client doesn’t know its
> > >> >>>> memberId.
> > >> >>>>>>> This KIP indeed cannot completely resolve the idempotency
> issue,
> > >> >>> but
> > >> >>>> it
> > >> >>>>>> can
> > >> >>>>>>> better handle shutdown scenarios under normal circumstances
> > >> >>> because
> > >> >>>> the
> > >> >>>>>>> client always knows its memberId. Even if the client shuts
> down
> > >> >>>>>> immediately
> > >> >>>>>>> after the initial heartbeat, as long as it performs a graceful
> > >> >>>> shutdown
> > >> >>>>>> and
> > >> >>>>>>> sends a leave heartbeat, the GC can manage the situation and
> > >> >>> remove
> > >> >>>> the
> > >> >>>>>>> member. Therefore, the goal of this KIP is to address the
> issue
> > >> >>> where
> > >> >>>>> the
> > >> >>>>>>> GC has to wait for the heartbeat timeout due to the client
> > leaving
> > >> >>>>>> without
> > >> >>>>>>> knowing its memberId, which leads to reduced throughput and
> > >> >>> limited
> > >> >>>>>>> scalability.
> > >> >>>>>>>
> > >> >>>>>>> The solution you suggest has also been proposed by David. The
> > >> >>> concern
> > >> >>>>>> with
> > >> >>>>>>> this approach is that it introduces additional complexity for
> > >> >>>>>>> compatibility, as the new server would not immediately add the
> > >> >>> member
> > >> >>>>> to
> > >> >>>>>>> the group, while the old server would. This requires clients
> to
> > >> >>>>>>> differentiate whether their memberId has been added to the
> group
> > >> >>> or
> > >> >>>>> not,
> > >> >>>>>>> which could result in unexpected logs.
> > >> >>>>>>>
> > >> >>>>>>> Best Regards,
> > >> >>>>>>> TengYao
> > >> >>>>>>>
> > >> >>>>>>> Andrew Schofield <an...@live.com> 於 2024年8月14日 週三
> > >> >>>>> 上午12:29寫道：
> > >> >>>>>>>
> > >> >>>>>>>> Hi TengYao,
> > >> >>>>>>>> Thanks for the KIP. I wonder if there’s a different way to
> > close
> > >> >>>> what
> > >> >>>>>>>> is quite a small window.
> > >> >>>>>>>>
> > >> >>>>>>>> AS1: It is true that the initial heartbeat is not idempotent,
> > but
> > >> >>>> this
> > >> >>>>>>>> remains
> > >> >>>>>>>> true with this KIP. It’s just differently not idempotent. If
> > the
> > >> >>>>> client
> > >> >>>>>>>> makes its
> > >> >>>>>>>> own member ID, sends a request and dies, the GC will still
> have
> > >> >>>> added
> > >> >>>>>>>> the member to the group and it will hang around until the
> > session
> > >> >>>>>> expires.
> > >> >>>>>>>>
> > >> >>>>>>>> I wonder if the GC could still generate the member ID in
> > >> >>> response to
> > >> >>>>> the
> > >> >>>>>>>> first
> > >> >>>>>>>> heartbeat, and put the member in a special PENDING state with
> > no
> > >> >>>>>>>> assignments until the client sends the next heartbeat, thus
> > >> >>>> confirming
> > >> >>>>>> it
> > >> >>>>>>>> has received the member ID. This would not be a protocol
> change
> > >> >>> at
> > >> >>>>> all,
> > >> >>>>>>>> just
> > >> >>>>>>>> a change to the GC to keep a new member in the lobby until
> it’s
> > >> >>>>>> comfirmed
> > >> >>>>>>>> it knows its member ID.
> > >> >>>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>>> Thanks,
> > >> >>>>>>>> Andrew
> > >> >>>>>>>>
> > >> >>>>>>>>> On 13 Aug 2024, at 15:59, TengYao Chi <ki...@gmail.com>
> > >> >>> wrote:
> > >> >>>>>>>>>
> > >> >>>>>>>>> Hi Chia-Ping,
> > >> >>>>>>>>>
> > >> >>>>>>>>> Thanks for review and suggestions.
> > >> >>>>>>>>> I have updated the content of KIP accordingly.
> > >> >>>>>>>>> Please take a look.
> > >> >>>>>>>>>
> > >> >>>>>>>>> Best regards,
> > >> >>>>>>>>> TengYao
> > >> >>>>>>>>>
> > >> >>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2024年8月13日 週二
> > 下午9:45寫道：
> > >> >>>>>>>>>
> > >> >>>>>>>>>> hi TengYao
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> thanks for this KIP.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> 1) could you please describe the before/after behavior in
> the
> > >> >>>>>> "Proposed
> > >> >>>>>>>>>> Changes" section? IIRC, current RPC allows HB having member
> > id
> > >> >>>>>>>> generated by
> > >> >>>>>>>>>> client, right? If HB has no member ID, server will generate
> > one
> > >> >>>> and
> > >> >>>>>> then
> > >> >>>>>>>>>> return. The new behavior will enforce HB "must" have member
> > ID.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> 2) could you please write the version number explicitly in
> > the
> > >> >>> KIP
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> 3) how new client code handle the old HB? Does it always
> > >> >>> generate
> > >> >>>>>> member
> > >> >>>>>>>>>> ID on client-side even though that is not restricted?
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> Best,
> > >> >>>>>>>>>> Chia-Ping
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> On 2024/08/13 06:20:42 TengYao Chi wrote:
> > >> >>>>>>>>>>> Hello everyone,
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> I would like to start a discussion thread on KIP-1082,
> which
> > >> >>>>> proposes
> > >> >>>>>>>>>>> enabling id generation for clients over the
> > >> >>>> ConsumerGroupHeartbeat
> > >> >>>>>> RPC.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Here is the KIP Link: KIP-1082
> > >> >>>>>>>>>>> <
> > >> >>>>>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>
> > >> >>>>>
> > >> >>>>
> > >> >>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Please take a look and let me know what you think, and I
> > would
> > >> >>>>>>>> appreciate
> > >> >>>>>>>>>>> any suggestions and feedback.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Best regards,
> > >> >>>>>>>>>>> TengYao
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>
> > >> >>>>
> > >> >>>
> > >> >>
> > >>
> > >>
> >
>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

["Lianet M." <li...@gmail.com>]

Hello TengYao! Thanks for taking on this issue, we've been going around it
for a while.

LM1: About the title of the KIP: "Enable ID Generation for Clients over the
ConsumerGroupHeartbeat RPC". I find it confusing because it hints that
we're adding it as an alternative (which was discussed and discarded, in
favour of really enforcing it). It's also missing the core change imo,
which is "where" the generation happens. So, maybe more to the point with
something along the lines of "Client-side generated ID for clients over
ConsumerGroupHeartbeat RPC"?

LM2: On the public interfaces section, the KIP states that "the server will
reject the request", but we should agree on the specific error type. I
expect it should fail with an InvalidRequestException, is that the
intention? (This was also suggested in the discussion thread before but is
not in the KIP).

LM3. Related to my previous point, I find that to be the true public-facing
change (member ID mandatory at the protocol level), but it's only at the
end of the Public interfaces changes, kind of lost among details of how
we're going to do it. Should we rephrase that section with the actual
change first, and the hows after (ex. Bumping the version is not the
public-facing change in this case, it's just the mechanism to properly
introduce our change)

LM4. Regarding the lifetime of the UUID: the KIP states we will "Verify
that the UUID remains consistent across all subsequent heartbeats during
the session". What is this "session" referring to here? I would expect that
the UUID is associated to a consumer instance (generated for the consumer
the first time it needs to send a HB if it doesn't have the UUID yet. From
there on, every time it needs to send a "first HB" again, it will reuse its
UUID, is that the intention? Note that we should consider that the same
consumer instance may have many "first heartbeats", meaning heartbeats to
join the group when it's not part of it (ex. consumer unsubscribe +
subscribe, fenced, stale). Is this the intention or are you considering the
lifetime differently? We should clarify it in the KIP.

Thanks!

Lianet

On Tue, Aug 27, 2024 at 2:27 AM TengYao Chi <ki...@gmail.com> wrote:

> Hi everyone,
>
> I have revised this KIP multiple times based on the feedback from our
> discussions.
> I would greatly appreciate it if you could review it when you have the
> time.
> If there are no further comments or suggestions, I plan to proceed with
> initiating a vote soon.
>
> Best regards,
> TengYao
>
> TengYao Chi <ki...@gmail.com> 於 2024年8月23日 週五 下午2:43寫道：
>
> > Hi Andrew,
> > Thank you for your previous feedback and insights.
> > Your contributions have added valuable perspectives to the discussions.
> > And we also benefit from the comparison of different solutions.
> > I’m also looking forward to seeing an initial version in KIP-932, as it
> > will provide a good reference for future implementations.
> >
> > Regarding your comment on AS2, I wanted to clarify that my specification
> > references org.apache.kafka.common.Uuid.
> > I believe we’re referring to the same class, and it might just be a small
> > oversight due to the busy schedule.
> >
> > I want to express my gratitude once again for your many insightful
> > comments, which have helped the discussion progress smoothly.
> >
> > Best regards,
> > TengYao
> >
> >
> > Andrew Schofield <an...@live.com> 於 2024年8月22日 週四 下午11:28寫道：
> >
> >> Hi TengYao,
> >> I’ve been reading through the comments and I’m happy that the lobby
> >> approach has not gained support.
> >>
> >> Assuming that this KIP is voted, I will be happy to change KIP-932 so
> >> that it only supports client-side member ID generation. Because that KIP
> >> is still
> >> under development, I can do this in the first version of
> >> ShareGroupHeartbeat.
> >>
> >> AS2: For the encoding section, I suppose the specific encoding which
> >> is used is what org.apache.kafka.utils.Uuid uses.
> >>
> >> Thanks,
> >> Andrew
> >>
> >> > On 14 Aug 2024, at 17:00, TengYao Chi <ki...@gmail.com> wrote:
> >> >
> >> > Hello Apoorv,
> >> > Thank you for your feedback.
> >> > Regarding the questions you raised, unfortunately, this KIP cannot
> >> > guarantee the order of heartbeats. As with many classic distributed
> >> system
> >> > challenges, what we can do is make our best effort to ensure that
> there
> >> are
> >> > no idle members or stale assignments under normal circumstances.
> >> >
> >> > As for the lobby approach, I’m not a fan of it because it requires
> >> adding a
> >> > mechanism to maintain client state within the ConsumerGroup, which, in
> >> my
> >> > view, resembles something like a two-phase commit. This would
> introduce
> >> > more complexity than the proposal in this KIP, which is something we
> >> want
> >> > to avoid. KIP-848 aims to simplify the existing protocol, and while
> the
> >> > lobby approach is a good one, I believe it is not the right fit for
> this
> >> > particular situation.
> >> >
> >> > Best regards,
> >> > TengYao
> >> >
> >> > TengYao Chi <ki...@gmail.com> 於 2024年8月14日 週三 下午11:45寫道：
> >> >
> >> >> Hi David,
> >> >>
> >> >> I really appreciate your review and suggestions. As I am still
> gaining
> >> >> experience in writing KIPs, your input has been incredibly helpful. I
> >> am
> >> >> currently applying your suggestions to the KIP and will complete it
> as
> >> soon
> >> >> as possible.
> >> >> Regarding the UUID part, I think we haven’t reached a conclusion
> >> yet.(So
> >> >> far according to this thread)
> >> >> However, I will review the current implementation in the Kafka `Uuid`
> >> >> class and include a brief specification in the KIP.
> >> >>
> >> >> Once again, thank you so much for your help.
> >> >>
> >> >> Best regards,
> >> >> TengYao
> >> >>
> >> >> Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月14日 週三 下午11:14寫道：
> >> >>
> >> >>> hi Apoorv
> >> >>>
> >> >>>> As the memberId is now known to the client, and client might send
> the
> >> >>> leave
> >> >>> group heartbeat on shutdown prior to receiving the initial heartbeat
> >> >>> response. If that's true then how do we guarantee that the 2
> requests
> >> to
> >> >>> join and leave will be processed in order, which could still leave
> >> stale
> >> >>> members or throw unknown member id exceptions?
> >> >>>
> >> >>> This is definitely a good question. the short answer: no guarantee
> but
> >> >>> best
> >> >>> efforts
> >> >>>
> >> >>> Please notice the root cause is "we have no enough time to wait
> >> member id
> >> >>> (response) when closing consumer". Sadly, we can' guarantee the
> >> request
> >> >>> order due to the same reason.
> >> >>>
> >> >>> However, in contrast to previous behavior, there is one big benefit
> >> of new
> >> >>> approach - we can try STONITH because we know the member id
> >> >>>
> >> >>> Best,
> >> >>> Chia-Ping
> >> >>>
> >> >>>
> >> >>> Apoorv Mittal <ap...@gmail.com> 於 2024年8月14日 週三 下午8:55寫道：
> >> >>>
> >> >>>> Hi TengYao,
> >> >>>> Thanks for the KIP. Continuing on the point which Andrew mentioned
> as
> >> >>> AS1.
> >> >>>>
> >> >>>> As the memberId is now known to the client, and client might send
> the
> >> >>> leave
> >> >>>> group heartbeat on shutdown prior to receiving the initial
> heartbeat
> >> >>>> response. If that's true then how do we guarantee that the 2
> >> requests to
> >> >>>> join and leave will be processed in order, which could still leave
> >> stale
> >> >>>> members or throw unknown member id exceptions?
> >> >>>>
> >> >>>> Though the client side member id generation is helpful which will
> >> >>> represent
> >> >>>> the same group perspective as from client and broker's end. But I
> >> think
> >> >>> the
> >> >>>> major concern we want to solve here is Stale Partition Assignments
> >> which
> >> >>>> might still exist with the new approach. I am leaning towards the
> >> >>>> suggestion mentioned by Andrew where partition assignment triggers
> on
> >> >>>> subsequent heartbeat when client acknowledges the initial
> heartbeat,
> >> >>>> delayed partition assignment.
> >> >>>>
> >> >>>> Though on a separate note, I have a different question. What
> happens
> >> >>> when
> >> >>>> there is an issue with the client which sends the initial heartbeat
> >> >>> without
> >> >>>> memberId, then crashes and restarts? I think we must be
> re-triggering
> >> >>>> assignments and expiring members only after the heartbeat session
> >> >>> timeout?
> >> >>>> If that's true then shall delayed partition assignment can help
> >> benefit
> >> >>> us
> >> >>>> from this situation as well?
> >> >>>>
> >> >>>> Regards,
> >> >>>> Apoorv Mittal
> >> >>>>
> >> >>>>
> >> >>>> On Wed, Aug 14, 2024 at 12:51 PM David Jacot
> >> >>> <dj...@confluent.io.invalid>
> >> >>>> wrote:
> >> >>>>
> >> >>>>> Hi Andrew,
> >> >>>>>
> >> >>>>> Personally, I don't like the lobby approach. It makes things more
> >> >>>>> complicated and it would require changing the records on the
> server
> >> >>> too.
> >> >>>>> This is why I initially suggested the rejected alternative #2
> which
> >> is
> >> >>>>> pretty close but also not perfect.
> >> >>>>>
> >> >>>>> I'd like to clarify one thing. The ConsumerGroupHeartbeat API
> >> already
> >> >>>>> supports generating the member id on the client so we don't need
> any
> >> >>>>> conditional logic on the client side. This is actually what we
> >> wanted
> >> >>> to
> >> >>>> do
> >> >>>>> in the first place but the idea got pushed back by Magnus back
> then
> >> >>>> because
> >> >>>>> generating uuid from librdkafka required a new dependency. It
> turns
> >> >>> out
> >> >>>>> that librdkafka has that dependency today. In retrospect, we
> should
> >> >>> have
> >> >>>>> pushed back on this. Long story short, we can just do it. The
> >> >>> proposal in
> >> >>>>> this KIP is to make the member id required in future versions. We
> >> >>> could
> >> >>>>> also decide not to do it and to keep supporting both approaches. I
> >> >>> would
> >> >>>>> also be fine with this.
> >> >>>>>
> >> >>>>> Best,
> >> >>>>> David
> >> >>>>>
> >> >>>>> On Wed, Aug 14, 2024 at 12:30 PM Andrew Schofield <
> >> >>>>> andrew_schofield@live.com>
> >> >>>>> wrote:
> >> >>>>>
> >> >>>>>> Hi TengYao,
> >> >>>>>> Thanks for your response. I’ll have just one more try to
> persuade.
> >> >>>>>> I feel that I will need to follow the approach with KIP-932 when
> >> >>> we’ve
> >> >>>>>> made a decision, so I do have more than a passing interest in
> this.
> >> >>>>>>
> >> >>>>>> A group member in the lobby is in the group, but it does not have
> >> >>> any
> >> >>>>>> assignments. A member of a consumer group can have no assigned
> >> >>>>>> partitions (such as 5 CG members subscribed to a topic with 4
> >> >>>>> partitions),
> >> >>>>>> so it’s a situation that consumer group members already expect.
> >> >>>>>>
> >> >>>>>> One of Kafka’s strengths is the way that we handle API
> versioning.
> >> >>>>>> But, there is a cost - the behaviour is different depending on
> the
> >> >>> RPC
> >> >>>>>> version. KIP-848 is on the cusp of completion, but we’re already
> >> >>> adding
> >> >>>>>> conditional logic for v0/v1 for ConsumerGroupHeartbeat. That’s a
> >> >>> pity.
> >> >>>>>> Only a minor issue, but it’s unfortunate.
> >> >>>>>>
> >> >>>>>> Thanks,
> >> >>>>>> Andrew
> >> >>>>>>
> >> >>>>>>> On 14 Aug 2024, at 08:47, TengYao Chi <ki...@gmail.com>
> >> >>> wrote:
> >> >>>>>>>
> >> >>>>>>> Hello Andrew
> >> >>>>>>> Thank you for your thoughtful suggestions and getting the
> >> >>> discussion
> >> >>>>>> going.
> >> >>>>>>>
> >> >>>>>>> To AS1:
> >> >>>>>>> In the current scenario where the server generates the UUID, if
> >> >>> the
> >> >>>>>> client
> >> >>>>>>> shuts down before receiving the memberId generated by the GC
> >> >>>>> (regardless
> >> >>>>>> of
> >> >>>>>>> whether it’s a graceful shutdown or not), the GC will still have
> >> >>> to
> >> >>>>> wait
> >> >>>>>>> for the heartbeat timeout because the client doesn’t know its
> >> >>>> memberId.
> >> >>>>>>> This KIP indeed cannot completely resolve the idempotency issue,
> >> >>> but
> >> >>>> it
> >> >>>>>> can
> >> >>>>>>> better handle shutdown scenarios under normal circumstances
> >> >>> because
> >> >>>> the
> >> >>>>>>> client always knows its memberId. Even if the client shuts down
> >> >>>>>> immediately
> >> >>>>>>> after the initial heartbeat, as long as it performs a graceful
> >> >>>> shutdown
> >> >>>>>> and
> >> >>>>>>> sends a leave heartbeat, the GC can manage the situation and
> >> >>> remove
> >> >>>> the
> >> >>>>>>> member. Therefore, the goal of this KIP is to address the issue
> >> >>> where
> >> >>>>> the
> >> >>>>>>> GC has to wait for the heartbeat timeout due to the client
> leaving
> >> >>>>>> without
> >> >>>>>>> knowing its memberId, which leads to reduced throughput and
> >> >>> limited
> >> >>>>>>> scalability.
> >> >>>>>>>
> >> >>>>>>> The solution you suggest has also been proposed by David. The
> >> >>> concern
> >> >>>>>> with
> >> >>>>>>> this approach is that it introduces additional complexity for
> >> >>>>>>> compatibility, as the new server would not immediately add the
> >> >>> member
> >> >>>>> to
> >> >>>>>>> the group, while the old server would. This requires clients to
> >> >>>>>>> differentiate whether their memberId has been added to the group
> >> >>> or
> >> >>>>> not,
> >> >>>>>>> which could result in unexpected logs.
> >> >>>>>>>
> >> >>>>>>> Best Regards,
> >> >>>>>>> TengYao
> >> >>>>>>>
> >> >>>>>>> Andrew Schofield <an...@live.com> 於 2024年8月14日 週三
> >> >>>>> 上午12:29寫道：
> >> >>>>>>>
> >> >>>>>>>> Hi TengYao,
> >> >>>>>>>> Thanks for the KIP. I wonder if there’s a different way to
> close
> >> >>>> what
> >> >>>>>>>> is quite a small window.
> >> >>>>>>>>
> >> >>>>>>>> AS1: It is true that the initial heartbeat is not idempotent,
> but
> >> >>>> this
> >> >>>>>>>> remains
> >> >>>>>>>> true with this KIP. It’s just differently not idempotent. If
> the
> >> >>>>> client
> >> >>>>>>>> makes its
> >> >>>>>>>> own member ID, sends a request and dies, the GC will still have
> >> >>>> added
> >> >>>>>>>> the member to the group and it will hang around until the
> session
> >> >>>>>> expires.
> >> >>>>>>>>
> >> >>>>>>>> I wonder if the GC could still generate the member ID in
> >> >>> response to
> >> >>>>> the
> >> >>>>>>>> first
> >> >>>>>>>> heartbeat, and put the member in a special PENDING state with
> no
> >> >>>>>>>> assignments until the client sends the next heartbeat, thus
> >> >>>> confirming
> >> >>>>>> it
> >> >>>>>>>> has received the member ID. This would not be a protocol change
> >> >>> at
> >> >>>>> all,
> >> >>>>>>>> just
> >> >>>>>>>> a change to the GC to keep a new member in the lobby until it’s
> >> >>>>>> comfirmed
> >> >>>>>>>> it knows its member ID.
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>> Thanks,
> >> >>>>>>>> Andrew
> >> >>>>>>>>
> >> >>>>>>>>> On 13 Aug 2024, at 15:59, TengYao Chi <ki...@gmail.com>
> >> >>> wrote:
> >> >>>>>>>>>
> >> >>>>>>>>> Hi Chia-Ping,
> >> >>>>>>>>>
> >> >>>>>>>>> Thanks for review and suggestions.
> >> >>>>>>>>> I have updated the content of KIP accordingly.
> >> >>>>>>>>> Please take a look.
> >> >>>>>>>>>
> >> >>>>>>>>> Best regards,
> >> >>>>>>>>> TengYao
> >> >>>>>>>>>
> >> >>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2024年8月13日 週二
> 下午9:45寫道：
> >> >>>>>>>>>
> >> >>>>>>>>>> hi TengYao
> >> >>>>>>>>>>
> >> >>>>>>>>>> thanks for this KIP.
> >> >>>>>>>>>>
> >> >>>>>>>>>> 1) could you please describe the before/after behavior in the
> >> >>>>>> "Proposed
> >> >>>>>>>>>> Changes" section? IIRC, current RPC allows HB having member
> id
> >> >>>>>>>> generated by
> >> >>>>>>>>>> client, right? If HB has no member ID, server will generate
> one
> >> >>>> and
> >> >>>>>> then
> >> >>>>>>>>>> return. The new behavior will enforce HB "must" have member
> ID.
> >> >>>>>>>>>>
> >> >>>>>>>>>> 2) could you please write the version number explicitly in
> the
> >> >>> KIP
> >> >>>>>>>>>>
> >> >>>>>>>>>> 3) how new client code handle the old HB? Does it always
> >> >>> generate
> >> >>>>>> member
> >> >>>>>>>>>> ID on client-side even though that is not restricted?
> >> >>>>>>>>>>
> >> >>>>>>>>>> Best,
> >> >>>>>>>>>> Chia-Ping
> >> >>>>>>>>>>
> >> >>>>>>>>>> On 2024/08/13 06:20:42 TengYao Chi wrote:
> >> >>>>>>>>>>> Hello everyone,
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> I would like to start a discussion thread on KIP-1082, which
> >> >>>>> proposes
> >> >>>>>>>>>>> enabling id generation for clients over the
> >> >>>> ConsumerGroupHeartbeat
> >> >>>>>> RPC.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Here is the KIP Link: KIP-1082
> >> >>>>>>>>>>> <
> >> >>>>>>>>>>
> >> >>>>>>>>
> >> >>>>>>
> >> >>>>>
> >> >>>>
> >> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Please take a look and let me know what you think, and I
> would
> >> >>>>>>>> appreciate
> >> >>>>>>>>>>> any suggestions and feedback.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Best regards,
> >> >>>>>>>>>>> TengYao
> >> >>>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>
> >> >>>>
> >> >>>
> >> >>
> >>
> >>
>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[TengYao Chi <ki...@gmail.com>]

Hi everyone,

I have revised this KIP multiple times based on the feedback from our
discussions.
I would greatly appreciate it if you could review it when you have the
time.
If there are no further comments or suggestions, I plan to proceed with
initiating a vote soon.

Best regards,
TengYao

TengYao Chi <ki...@gmail.com> 於 2024年8月23日 週五 下午2:43寫道：

> Hi Andrew,
> Thank you for your previous feedback and insights.
> Your contributions have added valuable perspectives to the discussions.
> And we also benefit from the comparison of different solutions.
> I’m also looking forward to seeing an initial version in KIP-932, as it
> will provide a good reference for future implementations.
>
> Regarding your comment on AS2, I wanted to clarify that my specification
> references org.apache.kafka.common.Uuid.
> I believe we’re referring to the same class, and it might just be a small
> oversight due to the busy schedule.
>
> I want to express my gratitude once again for your many insightful
> comments, which have helped the discussion progress smoothly.
>
> Best regards,
> TengYao
>
>
> Andrew Schofield <an...@live.com> 於 2024年8月22日 週四 下午11:28寫道：
>
>> Hi TengYao,
>> I’ve been reading through the comments and I’m happy that the lobby
>> approach has not gained support.
>>
>> Assuming that this KIP is voted, I will be happy to change KIP-932 so
>> that it only supports client-side member ID generation. Because that KIP
>> is still
>> under development, I can do this in the first version of
>> ShareGroupHeartbeat.
>>
>> AS2: For the encoding section, I suppose the specific encoding which
>> is used is what org.apache.kafka.utils.Uuid uses.
>>
>> Thanks,
>> Andrew
>>
>> > On 14 Aug 2024, at 17:00, TengYao Chi <ki...@gmail.com> wrote:
>> >
>> > Hello Apoorv,
>> > Thank you for your feedback.
>> > Regarding the questions you raised, unfortunately, this KIP cannot
>> > guarantee the order of heartbeats. As with many classic distributed
>> system
>> > challenges, what we can do is make our best effort to ensure that there
>> are
>> > no idle members or stale assignments under normal circumstances.
>> >
>> > As for the lobby approach, I’m not a fan of it because it requires
>> adding a
>> > mechanism to maintain client state within the ConsumerGroup, which, in
>> my
>> > view, resembles something like a two-phase commit. This would introduce
>> > more complexity than the proposal in this KIP, which is something we
>> want
>> > to avoid. KIP-848 aims to simplify the existing protocol, and while the
>> > lobby approach is a good one, I believe it is not the right fit for this
>> > particular situation.
>> >
>> > Best regards,
>> > TengYao
>> >
>> > TengYao Chi <ki...@gmail.com> 於 2024年8月14日 週三 下午11:45寫道：
>> >
>> >> Hi David,
>> >>
>> >> I really appreciate your review and suggestions. As I am still gaining
>> >> experience in writing KIPs, your input has been incredibly helpful. I
>> am
>> >> currently applying your suggestions to the KIP and will complete it as
>> soon
>> >> as possible.
>> >> Regarding the UUID part, I think we haven’t reached a conclusion
>> yet.(So
>> >> far according to this thread)
>> >> However, I will review the current implementation in the Kafka `Uuid`
>> >> class and include a brief specification in the KIP.
>> >>
>> >> Once again, thank you so much for your help.
>> >>
>> >> Best regards,
>> >> TengYao
>> >>
>> >> Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月14日 週三 下午11:14寫道：
>> >>
>> >>> hi Apoorv
>> >>>
>> >>>> As the memberId is now known to the client, and client might send the
>> >>> leave
>> >>> group heartbeat on shutdown prior to receiving the initial heartbeat
>> >>> response. If that's true then how do we guarantee that the 2 requests
>> to
>> >>> join and leave will be processed in order, which could still leave
>> stale
>> >>> members or throw unknown member id exceptions?
>> >>>
>> >>> This is definitely a good question. the short answer: no guarantee but
>> >>> best
>> >>> efforts
>> >>>
>> >>> Please notice the root cause is "we have no enough time to wait
>> member id
>> >>> (response) when closing consumer". Sadly, we can' guarantee the
>> request
>> >>> order due to the same reason.
>> >>>
>> >>> However, in contrast to previous behavior, there is one big benefit
>> of new
>> >>> approach - we can try STONITH because we know the member id
>> >>>
>> >>> Best,
>> >>> Chia-Ping
>> >>>
>> >>>
>> >>> Apoorv Mittal <ap...@gmail.com> 於 2024年8月14日 週三 下午8:55寫道：
>> >>>
>> >>>> Hi TengYao,
>> >>>> Thanks for the KIP. Continuing on the point which Andrew mentioned as
>> >>> AS1.
>> >>>>
>> >>>> As the memberId is now known to the client, and client might send the
>> >>> leave
>> >>>> group heartbeat on shutdown prior to receiving the initial heartbeat
>> >>>> response. If that's true then how do we guarantee that the 2
>> requests to
>> >>>> join and leave will be processed in order, which could still leave
>> stale
>> >>>> members or throw unknown member id exceptions?
>> >>>>
>> >>>> Though the client side member id generation is helpful which will
>> >>> represent
>> >>>> the same group perspective as from client and broker's end. But I
>> think
>> >>> the
>> >>>> major concern we want to solve here is Stale Partition Assignments
>> which
>> >>>> might still exist with the new approach. I am leaning towards the
>> >>>> suggestion mentioned by Andrew where partition assignment triggers on
>> >>>> subsequent heartbeat when client acknowledges the initial heartbeat,
>> >>>> delayed partition assignment.
>> >>>>
>> >>>> Though on a separate note, I have a different question. What happens
>> >>> when
>> >>>> there is an issue with the client which sends the initial heartbeat
>> >>> without
>> >>>> memberId, then crashes and restarts? I think we must be re-triggering
>> >>>> assignments and expiring members only after the heartbeat session
>> >>> timeout?
>> >>>> If that's true then shall delayed partition assignment can help
>> benefit
>> >>> us
>> >>>> from this situation as well?
>> >>>>
>> >>>> Regards,
>> >>>> Apoorv Mittal
>> >>>>
>> >>>>
>> >>>> On Wed, Aug 14, 2024 at 12:51 PM David Jacot
>> >>> <dj...@confluent.io.invalid>
>> >>>> wrote:
>> >>>>
>> >>>>> Hi Andrew,
>> >>>>>
>> >>>>> Personally, I don't like the lobby approach. It makes things more
>> >>>>> complicated and it would require changing the records on the server
>> >>> too.
>> >>>>> This is why I initially suggested the rejected alternative #2 which
>> is
>> >>>>> pretty close but also not perfect.
>> >>>>>
>> >>>>> I'd like to clarify one thing. The ConsumerGroupHeartbeat API
>> already
>> >>>>> supports generating the member id on the client so we don't need any
>> >>>>> conditional logic on the client side. This is actually what we
>> wanted
>> >>> to
>> >>>> do
>> >>>>> in the first place but the idea got pushed back by Magnus back then
>> >>>> because
>> >>>>> generating uuid from librdkafka required a new dependency. It turns
>> >>> out
>> >>>>> that librdkafka has that dependency today. In retrospect, we should
>> >>> have
>> >>>>> pushed back on this. Long story short, we can just do it. The
>> >>> proposal in
>> >>>>> this KIP is to make the member id required in future versions. We
>> >>> could
>> >>>>> also decide not to do it and to keep supporting both approaches. I
>> >>> would
>> >>>>> also be fine with this.
>> >>>>>
>> >>>>> Best,
>> >>>>> David
>> >>>>>
>> >>>>> On Wed, Aug 14, 2024 at 12:30 PM Andrew Schofield <
>> >>>>> andrew_schofield@live.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>>> Hi TengYao,
>> >>>>>> Thanks for your response. I’ll have just one more try to persuade.
>> >>>>>> I feel that I will need to follow the approach with KIP-932 when
>> >>> we’ve
>> >>>>>> made a decision, so I do have more than a passing interest in this.
>> >>>>>>
>> >>>>>> A group member in the lobby is in the group, but it does not have
>> >>> any
>> >>>>>> assignments. A member of a consumer group can have no assigned
>> >>>>>> partitions (such as 5 CG members subscribed to a topic with 4
>> >>>>> partitions),
>> >>>>>> so it’s a situation that consumer group members already expect.
>> >>>>>>
>> >>>>>> One of Kafka’s strengths is the way that we handle API versioning.
>> >>>>>> But, there is a cost - the behaviour is different depending on the
>> >>> RPC
>> >>>>>> version. KIP-848 is on the cusp of completion, but we’re already
>> >>> adding
>> >>>>>> conditional logic for v0/v1 for ConsumerGroupHeartbeat. That’s a
>> >>> pity.
>> >>>>>> Only a minor issue, but it’s unfortunate.
>> >>>>>>
>> >>>>>> Thanks,
>> >>>>>> Andrew
>> >>>>>>
>> >>>>>>> On 14 Aug 2024, at 08:47, TengYao Chi <ki...@gmail.com>
>> >>> wrote:
>> >>>>>>>
>> >>>>>>> Hello Andrew
>> >>>>>>> Thank you for your thoughtful suggestions and getting the
>> >>> discussion
>> >>>>>> going.
>> >>>>>>>
>> >>>>>>> To AS1:
>> >>>>>>> In the current scenario where the server generates the UUID, if
>> >>> the
>> >>>>>> client
>> >>>>>>> shuts down before receiving the memberId generated by the GC
>> >>>>> (regardless
>> >>>>>> of
>> >>>>>>> whether it’s a graceful shutdown or not), the GC will still have
>> >>> to
>> >>>>> wait
>> >>>>>>> for the heartbeat timeout because the client doesn’t know its
>> >>>> memberId.
>> >>>>>>> This KIP indeed cannot completely resolve the idempotency issue,
>> >>> but
>> >>>> it
>> >>>>>> can
>> >>>>>>> better handle shutdown scenarios under normal circumstances
>> >>> because
>> >>>> the
>> >>>>>>> client always knows its memberId. Even if the client shuts down
>> >>>>>> immediately
>> >>>>>>> after the initial heartbeat, as long as it performs a graceful
>> >>>> shutdown
>> >>>>>> and
>> >>>>>>> sends a leave heartbeat, the GC can manage the situation and
>> >>> remove
>> >>>> the
>> >>>>>>> member. Therefore, the goal of this KIP is to address the issue
>> >>> where
>> >>>>> the
>> >>>>>>> GC has to wait for the heartbeat timeout due to the client leaving
>> >>>>>> without
>> >>>>>>> knowing its memberId, which leads to reduced throughput and
>> >>> limited
>> >>>>>>> scalability.
>> >>>>>>>
>> >>>>>>> The solution you suggest has also been proposed by David. The
>> >>> concern
>> >>>>>> with
>> >>>>>>> this approach is that it introduces additional complexity for
>> >>>>>>> compatibility, as the new server would not immediately add the
>> >>> member
>> >>>>> to
>> >>>>>>> the group, while the old server would. This requires clients to
>> >>>>>>> differentiate whether their memberId has been added to the group
>> >>> or
>> >>>>> not,
>> >>>>>>> which could result in unexpected logs.
>> >>>>>>>
>> >>>>>>> Best Regards,
>> >>>>>>> TengYao
>> >>>>>>>
>> >>>>>>> Andrew Schofield <an...@live.com> 於 2024年8月14日 週三
>> >>>>> 上午12:29寫道：
>> >>>>>>>
>> >>>>>>>> Hi TengYao,
>> >>>>>>>> Thanks for the KIP. I wonder if there’s a different way to close
>> >>>> what
>> >>>>>>>> is quite a small window.
>> >>>>>>>>
>> >>>>>>>> AS1: It is true that the initial heartbeat is not idempotent, but
>> >>>> this
>> >>>>>>>> remains
>> >>>>>>>> true with this KIP. It’s just differently not idempotent. If the
>> >>>>> client
>> >>>>>>>> makes its
>> >>>>>>>> own member ID, sends a request and dies, the GC will still have
>> >>>> added
>> >>>>>>>> the member to the group and it will hang around until the session
>> >>>>>> expires.
>> >>>>>>>>
>> >>>>>>>> I wonder if the GC could still generate the member ID in
>> >>> response to
>> >>>>> the
>> >>>>>>>> first
>> >>>>>>>> heartbeat, and put the member in a special PENDING state with no
>> >>>>>>>> assignments until the client sends the next heartbeat, thus
>> >>>> confirming
>> >>>>>> it
>> >>>>>>>> has received the member ID. This would not be a protocol change
>> >>> at
>> >>>>> all,
>> >>>>>>>> just
>> >>>>>>>> a change to the GC to keep a new member in the lobby until it’s
>> >>>>>> comfirmed
>> >>>>>>>> it knows its member ID.
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> Thanks,
>> >>>>>>>> Andrew
>> >>>>>>>>
>> >>>>>>>>> On 13 Aug 2024, at 15:59, TengYao Chi <ki...@gmail.com>
>> >>> wrote:
>> >>>>>>>>>
>> >>>>>>>>> Hi Chia-Ping,
>> >>>>>>>>>
>> >>>>>>>>> Thanks for review and suggestions.
>> >>>>>>>>> I have updated the content of KIP accordingly.
>> >>>>>>>>> Please take a look.
>> >>>>>>>>>
>> >>>>>>>>> Best regards,
>> >>>>>>>>> TengYao
>> >>>>>>>>>
>> >>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2024年8月13日 週二 下午9:45寫道：
>> >>>>>>>>>
>> >>>>>>>>>> hi TengYao
>> >>>>>>>>>>
>> >>>>>>>>>> thanks for this KIP.
>> >>>>>>>>>>
>> >>>>>>>>>> 1) could you please describe the before/after behavior in the
>> >>>>>> "Proposed
>> >>>>>>>>>> Changes" section? IIRC, current RPC allows HB having member id
>> >>>>>>>> generated by
>> >>>>>>>>>> client, right? If HB has no member ID, server will generate one
>> >>>> and
>> >>>>>> then
>> >>>>>>>>>> return. The new behavior will enforce HB "must" have member ID.
>> >>>>>>>>>>
>> >>>>>>>>>> 2) could you please write the version number explicitly in the
>> >>> KIP
>> >>>>>>>>>>
>> >>>>>>>>>> 3) how new client code handle the old HB? Does it always
>> >>> generate
>> >>>>>> member
>> >>>>>>>>>> ID on client-side even though that is not restricted?
>> >>>>>>>>>>
>> >>>>>>>>>> Best,
>> >>>>>>>>>> Chia-Ping
>> >>>>>>>>>>
>> >>>>>>>>>> On 2024/08/13 06:20:42 TengYao Chi wrote:
>> >>>>>>>>>>> Hello everyone,
>> >>>>>>>>>>>
>> >>>>>>>>>>> I would like to start a discussion thread on KIP-1082, which
>> >>>>> proposes
>> >>>>>>>>>>> enabling id generation for clients over the
>> >>>> ConsumerGroupHeartbeat
>> >>>>>> RPC.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Here is the KIP Link: KIP-1082
>> >>>>>>>>>>> <
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
>> >>>>>>>>>>>
>> >>>>>>>>>>> Please take a look and let me know what you think, and I would
>> >>>>>>>> appreciate
>> >>>>>>>>>>> any suggestions and feedback.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Best regards,
>> >>>>>>>>>>> TengYao
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>>
>>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[TengYao Chi <ki...@gmail.com>]

Hi Andrew,
Thank you for your previous feedback and insights.
Your contributions have added valuable perspectives to the discussions. And
we also benefit from the comparison of different solutions.
I’m also looking forward to seeing an initial version in KIP-932, as it
will provide a good reference for future implementations.

Regarding your comment on AS2, I wanted to clarify that my specification
references org.apache.kafka.common.Uuid.
I believe we’re referring to the same class, and it might just be a small
oversight due to the busy schedule.

I want to express my gratitude once again for your many insightful
comments, which have helped the discussion progress smoothly.

Best regards,
TengYao


Andrew Schofield <an...@live.com> 於 2024年8月22日 週四 下午11:28寫道：

> Hi TengYao,
> I’ve been reading through the comments and I’m happy that the lobby
> approach has not gained support.
>
> Assuming that this KIP is voted, I will be happy to change KIP-932 so
> that it only supports client-side member ID generation. Because that KIP
> is still
> under development, I can do this in the first version of
> ShareGroupHeartbeat.
>
> AS2: For the encoding section, I suppose the specific encoding which
> is used is what org.apache.kafka.utils.Uuid uses.
>
> Thanks,
> Andrew
>
> > On 14 Aug 2024, at 17:00, TengYao Chi <ki...@gmail.com> wrote:
> >
> > Hello Apoorv,
> > Thank you for your feedback.
> > Regarding the questions you raised, unfortunately, this KIP cannot
> > guarantee the order of heartbeats. As with many classic distributed
> system
> > challenges, what we can do is make our best effort to ensure that there
> are
> > no idle members or stale assignments under normal circumstances.
> >
> > As for the lobby approach, I’m not a fan of it because it requires
> adding a
> > mechanism to maintain client state within the ConsumerGroup, which, in my
> > view, resembles something like a two-phase commit. This would introduce
> > more complexity than the proposal in this KIP, which is something we want
> > to avoid. KIP-848 aims to simplify the existing protocol, and while the
> > lobby approach is a good one, I believe it is not the right fit for this
> > particular situation.
> >
> > Best regards,
> > TengYao
> >
> > TengYao Chi <ki...@gmail.com> 於 2024年8月14日 週三 下午11:45寫道：
> >
> >> Hi David,
> >>
> >> I really appreciate your review and suggestions. As I am still gaining
> >> experience in writing KIPs, your input has been incredibly helpful. I am
> >> currently applying your suggestions to the KIP and will complete it as
> soon
> >> as possible.
> >> Regarding the UUID part, I think we haven’t reached a conclusion yet.(So
> >> far according to this thread)
> >> However, I will review the current implementation in the Kafka `Uuid`
> >> class and include a brief specification in the KIP.
> >>
> >> Once again, thank you so much for your help.
> >>
> >> Best regards,
> >> TengYao
> >>
> >> Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月14日 週三 下午11:14寫道：
> >>
> >>> hi Apoorv
> >>>
> >>>> As the memberId is now known to the client, and client might send the
> >>> leave
> >>> group heartbeat on shutdown prior to receiving the initial heartbeat
> >>> response. If that's true then how do we guarantee that the 2 requests
> to
> >>> join and leave will be processed in order, which could still leave
> stale
> >>> members or throw unknown member id exceptions?
> >>>
> >>> This is definitely a good question. the short answer: no guarantee but
> >>> best
> >>> efforts
> >>>
> >>> Please notice the root cause is "we have no enough time to wait member
> id
> >>> (response) when closing consumer". Sadly, we can' guarantee the request
> >>> order due to the same reason.
> >>>
> >>> However, in contrast to previous behavior, there is one big benefit of
> new
> >>> approach - we can try STONITH because we know the member id
> >>>
> >>> Best,
> >>> Chia-Ping
> >>>
> >>>
> >>> Apoorv Mittal <ap...@gmail.com> 於 2024年8月14日 週三 下午8:55寫道：
> >>>
> >>>> Hi TengYao,
> >>>> Thanks for the KIP. Continuing on the point which Andrew mentioned as
> >>> AS1.
> >>>>
> >>>> As the memberId is now known to the client, and client might send the
> >>> leave
> >>>> group heartbeat on shutdown prior to receiving the initial heartbeat
> >>>> response. If that's true then how do we guarantee that the 2 requests
> to
> >>>> join and leave will be processed in order, which could still leave
> stale
> >>>> members or throw unknown member id exceptions?
> >>>>
> >>>> Though the client side member id generation is helpful which will
> >>> represent
> >>>> the same group perspective as from client and broker's end. But I
> think
> >>> the
> >>>> major concern we want to solve here is Stale Partition Assignments
> which
> >>>> might still exist with the new approach. I am leaning towards the
> >>>> suggestion mentioned by Andrew where partition assignment triggers on
> >>>> subsequent heartbeat when client acknowledges the initial heartbeat,
> >>>> delayed partition assignment.
> >>>>
> >>>> Though on a separate note, I have a different question. What happens
> >>> when
> >>>> there is an issue with the client which sends the initial heartbeat
> >>> without
> >>>> memberId, then crashes and restarts? I think we must be re-triggering
> >>>> assignments and expiring members only after the heartbeat session
> >>> timeout?
> >>>> If that's true then shall delayed partition assignment can help
> benefit
> >>> us
> >>>> from this situation as well?
> >>>>
> >>>> Regards,
> >>>> Apoorv Mittal
> >>>>
> >>>>
> >>>> On Wed, Aug 14, 2024 at 12:51 PM David Jacot
> >>> <dj...@confluent.io.invalid>
> >>>> wrote:
> >>>>
> >>>>> Hi Andrew,
> >>>>>
> >>>>> Personally, I don't like the lobby approach. It makes things more
> >>>>> complicated and it would require changing the records on the server
> >>> too.
> >>>>> This is why I initially suggested the rejected alternative #2 which
> is
> >>>>> pretty close but also not perfect.
> >>>>>
> >>>>> I'd like to clarify one thing. The ConsumerGroupHeartbeat API already
> >>>>> supports generating the member id on the client so we don't need any
> >>>>> conditional logic on the client side. This is actually what we wanted
> >>> to
> >>>> do
> >>>>> in the first place but the idea got pushed back by Magnus back then
> >>>> because
> >>>>> generating uuid from librdkafka required a new dependency. It turns
> >>> out
> >>>>> that librdkafka has that dependency today. In retrospect, we should
> >>> have
> >>>>> pushed back on this. Long story short, we can just do it. The
> >>> proposal in
> >>>>> this KIP is to make the member id required in future versions. We
> >>> could
> >>>>> also decide not to do it and to keep supporting both approaches. I
> >>> would
> >>>>> also be fine with this.
> >>>>>
> >>>>> Best,
> >>>>> David
> >>>>>
> >>>>> On Wed, Aug 14, 2024 at 12:30 PM Andrew Schofield <
> >>>>> andrew_schofield@live.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Hi TengYao,
> >>>>>> Thanks for your response. I’ll have just one more try to persuade.
> >>>>>> I feel that I will need to follow the approach with KIP-932 when
> >>> we’ve
> >>>>>> made a decision, so I do have more than a passing interest in this.
> >>>>>>
> >>>>>> A group member in the lobby is in the group, but it does not have
> >>> any
> >>>>>> assignments. A member of a consumer group can have no assigned
> >>>>>> partitions (such as 5 CG members subscribed to a topic with 4
> >>>>> partitions),
> >>>>>> so it’s a situation that consumer group members already expect.
> >>>>>>
> >>>>>> One of Kafka’s strengths is the way that we handle API versioning.
> >>>>>> But, there is a cost - the behaviour is different depending on the
> >>> RPC
> >>>>>> version. KIP-848 is on the cusp of completion, but we’re already
> >>> adding
> >>>>>> conditional logic for v0/v1 for ConsumerGroupHeartbeat. That’s a
> >>> pity.
> >>>>>> Only a minor issue, but it’s unfortunate.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Andrew
> >>>>>>
> >>>>>>> On 14 Aug 2024, at 08:47, TengYao Chi <ki...@gmail.com>
> >>> wrote:
> >>>>>>>
> >>>>>>> Hello Andrew
> >>>>>>> Thank you for your thoughtful suggestions and getting the
> >>> discussion
> >>>>>> going.
> >>>>>>>
> >>>>>>> To AS1:
> >>>>>>> In the current scenario where the server generates the UUID, if
> >>> the
> >>>>>> client
> >>>>>>> shuts down before receiving the memberId generated by the GC
> >>>>> (regardless
> >>>>>> of
> >>>>>>> whether it’s a graceful shutdown or not), the GC will still have
> >>> to
> >>>>> wait
> >>>>>>> for the heartbeat timeout because the client doesn’t know its
> >>>> memberId.
> >>>>>>> This KIP indeed cannot completely resolve the idempotency issue,
> >>> but
> >>>> it
> >>>>>> can
> >>>>>>> better handle shutdown scenarios under normal circumstances
> >>> because
> >>>> the
> >>>>>>> client always knows its memberId. Even if the client shuts down
> >>>>>> immediately
> >>>>>>> after the initial heartbeat, as long as it performs a graceful
> >>>> shutdown
> >>>>>> and
> >>>>>>> sends a leave heartbeat, the GC can manage the situation and
> >>> remove
> >>>> the
> >>>>>>> member. Therefore, the goal of this KIP is to address the issue
> >>> where
> >>>>> the
> >>>>>>> GC has to wait for the heartbeat timeout due to the client leaving
> >>>>>> without
> >>>>>>> knowing its memberId, which leads to reduced throughput and
> >>> limited
> >>>>>>> scalability.
> >>>>>>>
> >>>>>>> The solution you suggest has also been proposed by David. The
> >>> concern
> >>>>>> with
> >>>>>>> this approach is that it introduces additional complexity for
> >>>>>>> compatibility, as the new server would not immediately add the
> >>> member
> >>>>> to
> >>>>>>> the group, while the old server would. This requires clients to
> >>>>>>> differentiate whether their memberId has been added to the group
> >>> or
> >>>>> not,
> >>>>>>> which could result in unexpected logs.
> >>>>>>>
> >>>>>>> Best Regards,
> >>>>>>> TengYao
> >>>>>>>
> >>>>>>> Andrew Schofield <an...@live.com> 於 2024年8月14日 週三
> >>>>> 上午12:29寫道：
> >>>>>>>
> >>>>>>>> Hi TengYao,
> >>>>>>>> Thanks for the KIP. I wonder if there’s a different way to close
> >>>> what
> >>>>>>>> is quite a small window.
> >>>>>>>>
> >>>>>>>> AS1: It is true that the initial heartbeat is not idempotent, but
> >>>> this
> >>>>>>>> remains
> >>>>>>>> true with this KIP. It’s just differently not idempotent. If the
> >>>>> client
> >>>>>>>> makes its
> >>>>>>>> own member ID, sends a request and dies, the GC will still have
> >>>> added
> >>>>>>>> the member to the group and it will hang around until the session
> >>>>>> expires.
> >>>>>>>>
> >>>>>>>> I wonder if the GC could still generate the member ID in
> >>> response to
> >>>>> the
> >>>>>>>> first
> >>>>>>>> heartbeat, and put the member in a special PENDING state with no
> >>>>>>>> assignments until the client sends the next heartbeat, thus
> >>>> confirming
> >>>>>> it
> >>>>>>>> has received the member ID. This would not be a protocol change
> >>> at
> >>>>> all,
> >>>>>>>> just
> >>>>>>>> a change to the GC to keep a new member in the lobby until it’s
> >>>>>> comfirmed
> >>>>>>>> it knows its member ID.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Andrew
> >>>>>>>>
> >>>>>>>>> On 13 Aug 2024, at 15:59, TengYao Chi <ki...@gmail.com>
> >>> wrote:
> >>>>>>>>>
> >>>>>>>>> Hi Chia-Ping,
> >>>>>>>>>
> >>>>>>>>> Thanks for review and suggestions.
> >>>>>>>>> I have updated the content of KIP accordingly.
> >>>>>>>>> Please take a look.
> >>>>>>>>>
> >>>>>>>>> Best regards,
> >>>>>>>>> TengYao
> >>>>>>>>>
> >>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2024年8月13日 週二 下午9:45寫道：
> >>>>>>>>>
> >>>>>>>>>> hi TengYao
> >>>>>>>>>>
> >>>>>>>>>> thanks for this KIP.
> >>>>>>>>>>
> >>>>>>>>>> 1) could you please describe the before/after behavior in the
> >>>>>> "Proposed
> >>>>>>>>>> Changes" section? IIRC, current RPC allows HB having member id
> >>>>>>>> generated by
> >>>>>>>>>> client, right? If HB has no member ID, server will generate one
> >>>> and
> >>>>>> then
> >>>>>>>>>> return. The new behavior will enforce HB "must" have member ID.
> >>>>>>>>>>
> >>>>>>>>>> 2) could you please write the version number explicitly in the
> >>> KIP
> >>>>>>>>>>
> >>>>>>>>>> 3) how new client code handle the old HB? Does it always
> >>> generate
> >>>>>> member
> >>>>>>>>>> ID on client-side even though that is not restricted?
> >>>>>>>>>>
> >>>>>>>>>> Best,
> >>>>>>>>>> Chia-Ping
> >>>>>>>>>>
> >>>>>>>>>> On 2024/08/13 06:20:42 TengYao Chi wrote:
> >>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>
> >>>>>>>>>>> I would like to start a discussion thread on KIP-1082, which
> >>>>> proposes
> >>>>>>>>>>> enabling id generation for clients over the
> >>>> ConsumerGroupHeartbeat
> >>>>>> RPC.
> >>>>>>>>>>>
> >>>>>>>>>>> Here is the KIP Link: KIP-1082
> >>>>>>>>>>> <
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
> >>>>>>>>>>>
> >>>>>>>>>>> Please take a look and let me know what you think, and I would
> >>>>>>>> appreciate
> >>>>>>>>>>> any suggestions and feedback.
> >>>>>>>>>>>
> >>>>>>>>>>> Best regards,
> >>>>>>>>>>> TengYao
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[Andrew Schofield <an...@live.com>]

Hi TengYao,
I’ve been reading through the comments and I’m happy that the lobby
approach has not gained support.

Assuming that this KIP is voted, I will be happy to change KIP-932 so
that it only supports client-side member ID generation. Because that KIP is still
under development, I can do this in the first version of ShareGroupHeartbeat.

AS2: For the encoding section, I suppose the specific encoding which
is used is what org.apache.kafka.utils.Uuid uses.

Thanks,
Andrew

> On 14 Aug 2024, at 17:00, TengYao Chi <ki...@gmail.com> wrote:
>
> Hello Apoorv,
> Thank you for your feedback.
> Regarding the questions you raised, unfortunately, this KIP cannot
> guarantee the order of heartbeats. As with many classic distributed system
> challenges, what we can do is make our best effort to ensure that there are
> no idle members or stale assignments under normal circumstances.
>
> As for the lobby approach, I’m not a fan of it because it requires adding a
> mechanism to maintain client state within the ConsumerGroup, which, in my
> view, resembles something like a two-phase commit. This would introduce
> more complexity than the proposal in this KIP, which is something we want
> to avoid. KIP-848 aims to simplify the existing protocol, and while the
> lobby approach is a good one, I believe it is not the right fit for this
> particular situation.
>
> Best regards,
> TengYao
>
> TengYao Chi <ki...@gmail.com> 於 2024年8月14日 週三 下午11:45寫道：
>
>> Hi David,
>>
>> I really appreciate your review and suggestions. As I am still gaining
>> experience in writing KIPs, your input has been incredibly helpful. I am
>> currently applying your suggestions to the KIP and will complete it as soon
>> as possible.
>> Regarding the UUID part, I think we haven’t reached a conclusion yet.(So
>> far according to this thread)
>> However, I will review the current implementation in the Kafka `Uuid`
>> class and include a brief specification in the KIP.
>>
>> Once again, thank you so much for your help.
>>
>> Best regards,
>> TengYao
>>
>> Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月14日 週三 下午11:14寫道：
>>
>>> hi Apoorv
>>>
>>>> As the memberId is now known to the client, and client might send the
>>> leave
>>> group heartbeat on shutdown prior to receiving the initial heartbeat
>>> response. If that's true then how do we guarantee that the 2 requests to
>>> join and leave will be processed in order, which could still leave stale
>>> members or throw unknown member id exceptions?
>>>
>>> This is definitely a good question. the short answer: no guarantee but
>>> best
>>> efforts
>>>
>>> Please notice the root cause is "we have no enough time to wait member id
>>> (response) when closing consumer". Sadly, we can' guarantee the request
>>> order due to the same reason.
>>>
>>> However, in contrast to previous behavior, there is one big benefit of new
>>> approach - we can try STONITH because we know the member id
>>>
>>> Best,
>>> Chia-Ping
>>>
>>>
>>> Apoorv Mittal <ap...@gmail.com> 於 2024年8月14日 週三 下午8:55寫道：
>>>
>>>> Hi TengYao,
>>>> Thanks for the KIP. Continuing on the point which Andrew mentioned as
>>> AS1.
>>>>
>>>> As the memberId is now known to the client, and client might send the
>>> leave
>>>> group heartbeat on shutdown prior to receiving the initial heartbeat
>>>> response. If that's true then how do we guarantee that the 2 requests to
>>>> join and leave will be processed in order, which could still leave stale
>>>> members or throw unknown member id exceptions?
>>>>
>>>> Though the client side member id generation is helpful which will
>>> represent
>>>> the same group perspective as from client and broker's end. But I think
>>> the
>>>> major concern we want to solve here is Stale Partition Assignments which
>>>> might still exist with the new approach. I am leaning towards the
>>>> suggestion mentioned by Andrew where partition assignment triggers on
>>>> subsequent heartbeat when client acknowledges the initial heartbeat,
>>>> delayed partition assignment.
>>>>
>>>> Though on a separate note, I have a different question. What happens
>>> when
>>>> there is an issue with the client which sends the initial heartbeat
>>> without
>>>> memberId, then crashes and restarts? I think we must be re-triggering
>>>> assignments and expiring members only after the heartbeat session
>>> timeout?
>>>> If that's true then shall delayed partition assignment can help benefit
>>> us
>>>> from this situation as well?
>>>>
>>>> Regards,
>>>> Apoorv Mittal
>>>>
>>>>
>>>> On Wed, Aug 14, 2024 at 12:51 PM David Jacot
>>> <dj...@confluent.io.invalid>
>>>> wrote:
>>>>
>>>>> Hi Andrew,
>>>>>
>>>>> Personally, I don't like the lobby approach. It makes things more
>>>>> complicated and it would require changing the records on the server
>>> too.
>>>>> This is why I initially suggested the rejected alternative #2 which is
>>>>> pretty close but also not perfect.
>>>>>
>>>>> I'd like to clarify one thing. The ConsumerGroupHeartbeat API already
>>>>> supports generating the member id on the client so we don't need any
>>>>> conditional logic on the client side. This is actually what we wanted
>>> to
>>>> do
>>>>> in the first place but the idea got pushed back by Magnus back then
>>>> because
>>>>> generating uuid from librdkafka required a new dependency. It turns
>>> out
>>>>> that librdkafka has that dependency today. In retrospect, we should
>>> have
>>>>> pushed back on this. Long story short, we can just do it. The
>>> proposal in
>>>>> this KIP is to make the member id required in future versions. We
>>> could
>>>>> also decide not to do it and to keep supporting both approaches. I
>>> would
>>>>> also be fine with this.
>>>>>
>>>>> Best,
>>>>> David
>>>>>
>>>>> On Wed, Aug 14, 2024 at 12:30 PM Andrew Schofield <
>>>>> andrew_schofield@live.com>
>>>>> wrote:
>>>>>
>>>>>> Hi TengYao,
>>>>>> Thanks for your response. I’ll have just one more try to persuade.
>>>>>> I feel that I will need to follow the approach with KIP-932 when
>>> we’ve
>>>>>> made a decision, so I do have more than a passing interest in this.
>>>>>>
>>>>>> A group member in the lobby is in the group, but it does not have
>>> any
>>>>>> assignments. A member of a consumer group can have no assigned
>>>>>> partitions (such as 5 CG members subscribed to a topic with 4
>>>>> partitions),
>>>>>> so it’s a situation that consumer group members already expect.
>>>>>>
>>>>>> One of Kafka’s strengths is the way that we handle API versioning.
>>>>>> But, there is a cost - the behaviour is different depending on the
>>> RPC
>>>>>> version. KIP-848 is on the cusp of completion, but we’re already
>>> adding
>>>>>> conditional logic for v0/v1 for ConsumerGroupHeartbeat. That’s a
>>> pity.
>>>>>> Only a minor issue, but it’s unfortunate.
>>>>>>
>>>>>> Thanks,
>>>>>> Andrew
>>>>>>
>>>>>>> On 14 Aug 2024, at 08:47, TengYao Chi <ki...@gmail.com>
>>> wrote:
>>>>>>>
>>>>>>> Hello Andrew
>>>>>>> Thank you for your thoughtful suggestions and getting the
>>> discussion
>>>>>> going.
>>>>>>>
>>>>>>> To AS1:
>>>>>>> In the current scenario where the server generates the UUID, if
>>> the
>>>>>> client
>>>>>>> shuts down before receiving the memberId generated by the GC
>>>>> (regardless
>>>>>> of
>>>>>>> whether it’s a graceful shutdown or not), the GC will still have
>>> to
>>>>> wait
>>>>>>> for the heartbeat timeout because the client doesn’t know its
>>>> memberId.
>>>>>>> This KIP indeed cannot completely resolve the idempotency issue,
>>> but
>>>> it
>>>>>> can
>>>>>>> better handle shutdown scenarios under normal circumstances
>>> because
>>>> the
>>>>>>> client always knows its memberId. Even if the client shuts down
>>>>>> immediately
>>>>>>> after the initial heartbeat, as long as it performs a graceful
>>>> shutdown
>>>>>> and
>>>>>>> sends a leave heartbeat, the GC can manage the situation and
>>> remove
>>>> the
>>>>>>> member. Therefore, the goal of this KIP is to address the issue
>>> where
>>>>> the
>>>>>>> GC has to wait for the heartbeat timeout due to the client leaving
>>>>>> without
>>>>>>> knowing its memberId, which leads to reduced throughput and
>>> limited
>>>>>>> scalability.
>>>>>>>
>>>>>>> The solution you suggest has also been proposed by David. The
>>> concern
>>>>>> with
>>>>>>> this approach is that it introduces additional complexity for
>>>>>>> compatibility, as the new server would not immediately add the
>>> member
>>>>> to
>>>>>>> the group, while the old server would. This requires clients to
>>>>>>> differentiate whether their memberId has been added to the group
>>> or
>>>>> not,
>>>>>>> which could result in unexpected logs.
>>>>>>>
>>>>>>> Best Regards,
>>>>>>> TengYao
>>>>>>>
>>>>>>> Andrew Schofield <an...@live.com> 於 2024年8月14日 週三
>>>>> 上午12:29寫道：
>>>>>>>
>>>>>>>> Hi TengYao,
>>>>>>>> Thanks for the KIP. I wonder if there’s a different way to close
>>>> what
>>>>>>>> is quite a small window.
>>>>>>>>
>>>>>>>> AS1: It is true that the initial heartbeat is not idempotent, but
>>>> this
>>>>>>>> remains
>>>>>>>> true with this KIP. It’s just differently not idempotent. If the
>>>>> client
>>>>>>>> makes its
>>>>>>>> own member ID, sends a request and dies, the GC will still have
>>>> added
>>>>>>>> the member to the group and it will hang around until the session
>>>>>> expires.
>>>>>>>>
>>>>>>>> I wonder if the GC could still generate the member ID in
>>> response to
>>>>> the
>>>>>>>> first
>>>>>>>> heartbeat, and put the member in a special PENDING state with no
>>>>>>>> assignments until the client sends the next heartbeat, thus
>>>> confirming
>>>>>> it
>>>>>>>> has received the member ID. This would not be a protocol change
>>> at
>>>>> all,
>>>>>>>> just
>>>>>>>> a change to the GC to keep a new member in the lobby until it’s
>>>>>> comfirmed
>>>>>>>> it knows its member ID.
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Andrew
>>>>>>>>
>>>>>>>>> On 13 Aug 2024, at 15:59, TengYao Chi <ki...@gmail.com>
>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi Chia-Ping,
>>>>>>>>>
>>>>>>>>> Thanks for review and suggestions.
>>>>>>>>> I have updated the content of KIP accordingly.
>>>>>>>>> Please take a look.
>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>> TengYao
>>>>>>>>>
>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2024年8月13日 週二 下午9:45寫道：
>>>>>>>>>
>>>>>>>>>> hi TengYao
>>>>>>>>>>
>>>>>>>>>> thanks for this KIP.
>>>>>>>>>>
>>>>>>>>>> 1) could you please describe the before/after behavior in the
>>>>>> "Proposed
>>>>>>>>>> Changes" section? IIRC, current RPC allows HB having member id
>>>>>>>> generated by
>>>>>>>>>> client, right? If HB has no member ID, server will generate one
>>>> and
>>>>>> then
>>>>>>>>>> return. The new behavior will enforce HB "must" have member ID.
>>>>>>>>>>
>>>>>>>>>> 2) could you please write the version number explicitly in the
>>> KIP
>>>>>>>>>>
>>>>>>>>>> 3) how new client code handle the old HB? Does it always
>>> generate
>>>>>> member
>>>>>>>>>> ID on client-side even though that is not restricted?
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Chia-Ping
>>>>>>>>>>
>>>>>>>>>> On 2024/08/13 06:20:42 TengYao Chi wrote:
>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>
>>>>>>>>>>> I would like to start a discussion thread on KIP-1082, which
>>>>> proposes
>>>>>>>>>>> enabling id generation for clients over the
>>>> ConsumerGroupHeartbeat
>>>>>> RPC.
>>>>>>>>>>>
>>>>>>>>>>> Here is the KIP Link: KIP-1082
>>>>>>>>>>> <
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
>>>>>>>>>>>
>>>>>>>>>>> Please take a look and let me know what you think, and I would
>>>>>>>> appreciate
>>>>>>>>>>> any suggestions and feedback.
>>>>>>>>>>>
>>>>>>>>>>> Best regards,
>>>>>>>>>>> TengYao
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[TengYao Chi <ki...@gmail.com>]

Hello Apoorv,
Thank you for your feedback.
Regarding the questions you raised, unfortunately, this KIP cannot
guarantee the order of heartbeats. As with many classic distributed system
challenges, what we can do is make our best effort to ensure that there are
no idle members or stale assignments under normal circumstances.

As for the lobby approach, I’m not a fan of it because it requires adding a
mechanism to maintain client state within the ConsumerGroup, which, in my
view, resembles something like a two-phase commit. This would introduce
more complexity than the proposal in this KIP, which is something we want
to avoid. KIP-848 aims to simplify the existing protocol, and while the
lobby approach is a good one, I believe it is not the right fit for this
particular situation.

Best regards,
TengYao

TengYao Chi <ki...@gmail.com> 於 2024年8月14日 週三 下午11:45寫道：

> Hi David,
>
> I really appreciate your review and suggestions. As I am still gaining
> experience in writing KIPs, your input has been incredibly helpful. I am
> currently applying your suggestions to the KIP and will complete it as soon
> as possible.
> Regarding the UUID part, I think we haven’t reached a conclusion yet.(So
> far according to this thread)
> However, I will review the current implementation in the Kafka `Uuid`
> class and include a brief specification in the KIP.
>
> Once again, thank you so much for your help.
>
> Best regards,
> TengYao
>
> Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月14日 週三 下午11:14寫道：
>
>> hi Apoorv
>>
>> > As the memberId is now known to the client, and client might send the
>> leave
>> group heartbeat on shutdown prior to receiving the initial heartbeat
>> response. If that's true then how do we guarantee that the 2 requests to
>> join and leave will be processed in order, which could still leave stale
>> members or throw unknown member id exceptions?
>>
>> This is definitely a good question. the short answer: no guarantee but
>> best
>> efforts
>>
>> Please notice the root cause is "we have no enough time to wait member id
>> (response) when closing consumer". Sadly, we can' guarantee the request
>> order due to the same reason.
>>
>> However, in contrast to previous behavior, there is one big benefit of new
>> approach - we can try STONITH because we know the member id
>>
>> Best,
>> Chia-Ping
>>
>>
>> Apoorv Mittal <ap...@gmail.com> 於 2024年8月14日 週三 下午8:55寫道：
>>
>> > Hi TengYao,
>> > Thanks for the KIP. Continuing on the point which Andrew mentioned as
>> AS1.
>> >
>> > As the memberId is now known to the client, and client might send the
>> leave
>> > group heartbeat on shutdown prior to receiving the initial heartbeat
>> > response. If that's true then how do we guarantee that the 2 requests to
>> > join and leave will be processed in order, which could still leave stale
>> > members or throw unknown member id exceptions?
>> >
>> > Though the client side member id generation is helpful which will
>> represent
>> > the same group perspective as from client and broker's end. But I think
>> the
>> > major concern we want to solve here is Stale Partition Assignments which
>> > might still exist with the new approach. I am leaning towards the
>> > suggestion mentioned by Andrew where partition assignment triggers on
>> > subsequent heartbeat when client acknowledges the initial heartbeat,
>> > delayed partition assignment.
>> >
>> > Though on a separate note, I have a different question. What happens
>> when
>> > there is an issue with the client which sends the initial heartbeat
>> without
>> > memberId, then crashes and restarts? I think we must be re-triggering
>> > assignments and expiring members only after the heartbeat session
>> timeout?
>> > If that's true then shall delayed partition assignment can help benefit
>> us
>> > from this situation as well?
>> >
>> > Regards,
>> > Apoorv Mittal
>> >
>> >
>> > On Wed, Aug 14, 2024 at 12:51 PM David Jacot
>> <dj...@confluent.io.invalid>
>> > wrote:
>> >
>> > > Hi Andrew,
>> > >
>> > > Personally, I don't like the lobby approach. It makes things more
>> > > complicated and it would require changing the records on the server
>> too.
>> > > This is why I initially suggested the rejected alternative #2 which is
>> > > pretty close but also not perfect.
>> > >
>> > > I'd like to clarify one thing. The ConsumerGroupHeartbeat API already
>> > > supports generating the member id on the client so we don't need any
>> > > conditional logic on the client side. This is actually what we wanted
>> to
>> > do
>> > > in the first place but the idea got pushed back by Magnus back then
>> > because
>> > > generating uuid from librdkafka required a new dependency. It turns
>> out
>> > > that librdkafka has that dependency today. In retrospect, we should
>> have
>> > > pushed back on this. Long story short, we can just do it. The
>> proposal in
>> > > this KIP is to make the member id required in future versions. We
>> could
>> > > also decide not to do it and to keep supporting both approaches. I
>> would
>> > > also be fine with this.
>> > >
>> > > Best,
>> > > David
>> > >
>> > > On Wed, Aug 14, 2024 at 12:30 PM Andrew Schofield <
>> > > andrew_schofield@live.com>
>> > > wrote:
>> > >
>> > > > Hi TengYao,
>> > > > Thanks for your response. I’ll have just one more try to persuade.
>> > > > I feel that I will need to follow the approach with KIP-932 when
>> we’ve
>> > > > made a decision, so I do have more than a passing interest in this.
>> > > >
>> > > > A group member in the lobby is in the group, but it does not have
>> any
>> > > > assignments. A member of a consumer group can have no assigned
>> > > > partitions (such as 5 CG members subscribed to a topic with 4
>> > > partitions),
>> > > > so it’s a situation that consumer group members already expect.
>> > > >
>> > > > One of Kafka’s strengths is the way that we handle API versioning.
>> > > > But, there is a cost - the behaviour is different depending on the
>> RPC
>> > > > version. KIP-848 is on the cusp of completion, but we’re already
>> adding
>> > > > conditional logic for v0/v1 for ConsumerGroupHeartbeat. That’s a
>> pity.
>> > > > Only a minor issue, but it’s unfortunate.
>> > > >
>> > > > Thanks,
>> > > > Andrew
>> > > >
>> > > > > On 14 Aug 2024, at 08:47, TengYao Chi <ki...@gmail.com>
>> wrote:
>> > > > >
>> > > > > Hello Andrew
>> > > > > Thank you for your thoughtful suggestions and getting the
>> discussion
>> > > > going.
>> > > > >
>> > > > > To AS1:
>> > > > > In the current scenario where the server generates the UUID, if
>> the
>> > > > client
>> > > > > shuts down before receiving the memberId generated by the GC
>> > > (regardless
>> > > > of
>> > > > > whether it’s a graceful shutdown or not), the GC will still have
>> to
>> > > wait
>> > > > > for the heartbeat timeout because the client doesn’t know its
>> > memberId.
>> > > > > This KIP indeed cannot completely resolve the idempotency issue,
>> but
>> > it
>> > > > can
>> > > > > better handle shutdown scenarios under normal circumstances
>> because
>> > the
>> > > > > client always knows its memberId. Even if the client shuts down
>> > > > immediately
>> > > > > after the initial heartbeat, as long as it performs a graceful
>> > shutdown
>> > > > and
>> > > > > sends a leave heartbeat, the GC can manage the situation and
>> remove
>> > the
>> > > > > member. Therefore, the goal of this KIP is to address the issue
>> where
>> > > the
>> > > > > GC has to wait for the heartbeat timeout due to the client leaving
>> > > > without
>> > > > > knowing its memberId, which leads to reduced throughput and
>> limited
>> > > > > scalability.
>> > > > >
>> > > > > The solution you suggest has also been proposed by David. The
>> concern
>> > > > with
>> > > > > this approach is that it introduces additional complexity for
>> > > > > compatibility, as the new server would not immediately add the
>> member
>> > > to
>> > > > > the group, while the old server would. This requires clients to
>> > > > > differentiate whether their memberId has been added to the group
>> or
>> > > not,
>> > > > > which could result in unexpected logs.
>> > > > >
>> > > > > Best Regards,
>> > > > > TengYao
>> > > > >
>> > > > > Andrew Schofield <an...@live.com> 於 2024年8月14日 週三
>> > > 上午12:29寫道：
>> > > > >
>> > > > >> Hi TengYao,
>> > > > >> Thanks for the KIP. I wonder if there’s a different way to close
>> > what
>> > > > >> is quite a small window.
>> > > > >>
>> > > > >> AS1: It is true that the initial heartbeat is not idempotent, but
>> > this
>> > > > >> remains
>> > > > >> true with this KIP. It’s just differently not idempotent. If the
>> > > client
>> > > > >> makes its
>> > > > >> own member ID, sends a request and dies, the GC will still have
>> > added
>> > > > >> the member to the group and it will hang around until the session
>> > > > expires.
>> > > > >>
>> > > > >> I wonder if the GC could still generate the member ID in
>> response to
>> > > the
>> > > > >> first
>> > > > >> heartbeat, and put the member in a special PENDING state with no
>> > > > >> assignments until the client sends the next heartbeat, thus
>> > confirming
>> > > > it
>> > > > >> has received the member ID. This would not be a protocol change
>> at
>> > > all,
>> > > > >> just
>> > > > >> a change to the GC to keep a new member in the lobby until it’s
>> > > > comfirmed
>> > > > >> it knows its member ID.
>> > > > >>
>> > > > >>
>> > > > >> Thanks,
>> > > > >> Andrew
>> > > > >>
>> > > > >>> On 13 Aug 2024, at 15:59, TengYao Chi <ki...@gmail.com>
>> wrote:
>> > > > >>>
>> > > > >>> Hi Chia-Ping,
>> > > > >>>
>> > > > >>> Thanks for review and suggestions.
>> > > > >>> I have updated the content of KIP accordingly.
>> > > > >>> Please take a look.
>> > > > >>>
>> > > > >>> Best regards,
>> > > > >>> TengYao
>> > > > >>>
>> > > > >>> Chia-Ping Tsai <ch...@apache.org> 於 2024年8月13日 週二 下午9:45寫道：
>> > > > >>>
>> > > > >>>> hi TengYao
>> > > > >>>>
>> > > > >>>> thanks for this KIP.
>> > > > >>>>
>> > > > >>>> 1) could you please describe the before/after behavior in the
>> > > > "Proposed
>> > > > >>>> Changes" section? IIRC, current RPC allows HB having member id
>> > > > >> generated by
>> > > > >>>> client, right? If HB has no member ID, server will generate one
>> > and
>> > > > then
>> > > > >>>> return. The new behavior will enforce HB "must" have member ID.
>> > > > >>>>
>> > > > >>>> 2) could you please write the version number explicitly in the
>> KIP
>> > > > >>>>
>> > > > >>>> 3) how new client code handle the old HB? Does it always
>> generate
>> > > > member
>> > > > >>>> ID on client-side even though that is not restricted?
>> > > > >>>>
>> > > > >>>> Best,
>> > > > >>>> Chia-Ping
>> > > > >>>>
>> > > > >>>> On 2024/08/13 06:20:42 TengYao Chi wrote:
>> > > > >>>>> Hello everyone,
>> > > > >>>>>
>> > > > >>>>> I would like to start a discussion thread on KIP-1082, which
>> > > proposes
>> > > > >>>>> enabling id generation for clients over the
>> > ConsumerGroupHeartbeat
>> > > > RPC.
>> > > > >>>>>
>> > > > >>>>> Here is the KIP Link: KIP-1082
>> > > > >>>>> <
>> > > > >>>>
>> > > > >>
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
>> > > > >>>>>
>> > > > >>>>> Please take a look and let me know what you think, and I would
>> > > > >> appreciate
>> > > > >>>>> any suggestions and feedback.
>> > > > >>>>>
>> > > > >>>>> Best regards,
>> > > > >>>>> TengYao
>> > > > >>>>>
>> > > > >>>>
>> > > > >>
>> > > > >>
>> > > >
>> > > >
>> > >
>> >
>>
>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[TengYao Chi <ki...@gmail.com>]

Hi David,

I really appreciate your review and suggestions. As I am still gaining
experience in writing KIPs, your input has been incredibly helpful. I am
currently applying your suggestions to the KIP and will complete it as soon
as possible.
Regarding the UUID part, I think we haven’t reached a conclusion yet.(So
far according to this thread)
However, I will review the current implementation in the Kafka `Uuid` class
and include a brief specification in the KIP.

Once again, thank you so much for your help.

Best regards,
TengYao

Chia-Ping Tsai <ch...@gmail.com> 於 2024年8月14日 週三 下午11:14寫道：

> hi Apoorv
>
> > As the memberId is now known to the client, and client might send the
> leave
> group heartbeat on shutdown prior to receiving the initial heartbeat
> response. If that's true then how do we guarantee that the 2 requests to
> join and leave will be processed in order, which could still leave stale
> members or throw unknown member id exceptions?
>
> This is definitely a good question. the short answer: no guarantee but best
> efforts
>
> Please notice the root cause is "we have no enough time to wait member id
> (response) when closing consumer". Sadly, we can' guarantee the request
> order due to the same reason.
>
> However, in contrast to previous behavior, there is one big benefit of new
> approach - we can try STONITH because we know the member id
>
> Best,
> Chia-Ping
>
>
> Apoorv Mittal <ap...@gmail.com> 於 2024年8月14日 週三 下午8:55寫道：
>
> > Hi TengYao,
> > Thanks for the KIP. Continuing on the point which Andrew mentioned as
> AS1.
> >
> > As the memberId is now known to the client, and client might send the
> leave
> > group heartbeat on shutdown prior to receiving the initial heartbeat
> > response. If that's true then how do we guarantee that the 2 requests to
> > join and leave will be processed in order, which could still leave stale
> > members or throw unknown member id exceptions?
> >
> > Though the client side member id generation is helpful which will
> represent
> > the same group perspective as from client and broker's end. But I think
> the
> > major concern we want to solve here is Stale Partition Assignments which
> > might still exist with the new approach. I am leaning towards the
> > suggestion mentioned by Andrew where partition assignment triggers on
> > subsequent heartbeat when client acknowledges the initial heartbeat,
> > delayed partition assignment.
> >
> > Though on a separate note, I have a different question. What happens when
> > there is an issue with the client which sends the initial heartbeat
> without
> > memberId, then crashes and restarts? I think we must be re-triggering
> > assignments and expiring members only after the heartbeat session
> timeout?
> > If that's true then shall delayed partition assignment can help benefit
> us
> > from this situation as well?
> >
> > Regards,
> > Apoorv Mittal
> >
> >
> > On Wed, Aug 14, 2024 at 12:51 PM David Jacot <djacot@confluent.io.invalid
> >
> > wrote:
> >
> > > Hi Andrew,
> > >
> > > Personally, I don't like the lobby approach. It makes things more
> > > complicated and it would require changing the records on the server
> too.
> > > This is why I initially suggested the rejected alternative #2 which is
> > > pretty close but also not perfect.
> > >
> > > I'd like to clarify one thing. The ConsumerGroupHeartbeat API already
> > > supports generating the member id on the client so we don't need any
> > > conditional logic on the client side. This is actually what we wanted
> to
> > do
> > > in the first place but the idea got pushed back by Magnus back then
> > because
> > > generating uuid from librdkafka required a new dependency. It turns out
> > > that librdkafka has that dependency today. In retrospect, we should
> have
> > > pushed back on this. Long story short, we can just do it. The proposal
> in
> > > this KIP is to make the member id required in future versions. We could
> > > also decide not to do it and to keep supporting both approaches. I
> would
> > > also be fine with this.
> > >
> > > Best,
> > > David
> > >
> > > On Wed, Aug 14, 2024 at 12:30 PM Andrew Schofield <
> > > andrew_schofield@live.com>
> > > wrote:
> > >
> > > > Hi TengYao,
> > > > Thanks for your response. I’ll have just one more try to persuade.
> > > > I feel that I will need to follow the approach with KIP-932 when
> we’ve
> > > > made a decision, so I do have more than a passing interest in this.
> > > >
> > > > A group member in the lobby is in the group, but it does not have any
> > > > assignments. A member of a consumer group can have no assigned
> > > > partitions (such as 5 CG members subscribed to a topic with 4
> > > partitions),
> > > > so it’s a situation that consumer group members already expect.
> > > >
> > > > One of Kafka’s strengths is the way that we handle API versioning.
> > > > But, there is a cost - the behaviour is different depending on the
> RPC
> > > > version. KIP-848 is on the cusp of completion, but we’re already
> adding
> > > > conditional logic for v0/v1 for ConsumerGroupHeartbeat. That’s a
> pity.
> > > > Only a minor issue, but it’s unfortunate.
> > > >
> > > > Thanks,
> > > > Andrew
> > > >
> > > > > On 14 Aug 2024, at 08:47, TengYao Chi <ki...@gmail.com> wrote:
> > > > >
> > > > > Hello Andrew
> > > > > Thank you for your thoughtful suggestions and getting the
> discussion
> > > > going.
> > > > >
> > > > > To AS1:
> > > > > In the current scenario where the server generates the UUID, if the
> > > > client
> > > > > shuts down before receiving the memberId generated by the GC
> > > (regardless
> > > > of
> > > > > whether it’s a graceful shutdown or not), the GC will still have to
> > > wait
> > > > > for the heartbeat timeout because the client doesn’t know its
> > memberId.
> > > > > This KIP indeed cannot completely resolve the idempotency issue,
> but
> > it
> > > > can
> > > > > better handle shutdown scenarios under normal circumstances because
> > the
> > > > > client always knows its memberId. Even if the client shuts down
> > > > immediately
> > > > > after the initial heartbeat, as long as it performs a graceful
> > shutdown
> > > > and
> > > > > sends a leave heartbeat, the GC can manage the situation and remove
> > the
> > > > > member. Therefore, the goal of this KIP is to address the issue
> where
> > > the
> > > > > GC has to wait for the heartbeat timeout due to the client leaving
> > > > without
> > > > > knowing its memberId, which leads to reduced throughput and limited
> > > > > scalability.
> > > > >
> > > > > The solution you suggest has also been proposed by David. The
> concern
> > > > with
> > > > > this approach is that it introduces additional complexity for
> > > > > compatibility, as the new server would not immediately add the
> member
> > > to
> > > > > the group, while the old server would. This requires clients to
> > > > > differentiate whether their memberId has been added to the group or
> > > not,
> > > > > which could result in unexpected logs.
> > > > >
> > > > > Best Regards,
> > > > > TengYao
> > > > >
> > > > > Andrew Schofield <an...@live.com> 於 2024年8月14日 週三
> > > 上午12:29寫道：
> > > > >
> > > > >> Hi TengYao,
> > > > >> Thanks for the KIP. I wonder if there’s a different way to close
> > what
> > > > >> is quite a small window.
> > > > >>
> > > > >> AS1: It is true that the initial heartbeat is not idempotent, but
> > this
> > > > >> remains
> > > > >> true with this KIP. It’s just differently not idempotent. If the
> > > client
> > > > >> makes its
> > > > >> own member ID, sends a request and dies, the GC will still have
> > added
> > > > >> the member to the group and it will hang around until the session
> > > > expires.
> > > > >>
> > > > >> I wonder if the GC could still generate the member ID in response
> to
> > > the
> > > > >> first
> > > > >> heartbeat, and put the member in a special PENDING state with no
> > > > >> assignments until the client sends the next heartbeat, thus
> > confirming
> > > > it
> > > > >> has received the member ID. This would not be a protocol change at
> > > all,
> > > > >> just
> > > > >> a change to the GC to keep a new member in the lobby until it’s
> > > > comfirmed
> > > > >> it knows its member ID.
> > > > >>
> > > > >>
> > > > >> Thanks,
> > > > >> Andrew
> > > > >>
> > > > >>> On 13 Aug 2024, at 15:59, TengYao Chi <ki...@gmail.com>
> wrote:
> > > > >>>
> > > > >>> Hi Chia-Ping,
> > > > >>>
> > > > >>> Thanks for review and suggestions.
> > > > >>> I have updated the content of KIP accordingly.
> > > > >>> Please take a look.
> > > > >>>
> > > > >>> Best regards,
> > > > >>> TengYao
> > > > >>>
> > > > >>> Chia-Ping Tsai <ch...@apache.org> 於 2024年8月13日 週二 下午9:45寫道：
> > > > >>>
> > > > >>>> hi TengYao
> > > > >>>>
> > > > >>>> thanks for this KIP.
> > > > >>>>
> > > > >>>> 1) could you please describe the before/after behavior in the
> > > > "Proposed
> > > > >>>> Changes" section? IIRC, current RPC allows HB having member id
> > > > >> generated by
> > > > >>>> client, right? If HB has no member ID, server will generate one
> > and
> > > > then
> > > > >>>> return. The new behavior will enforce HB "must" have member ID.
> > > > >>>>
> > > > >>>> 2) could you please write the version number explicitly in the
> KIP
> > > > >>>>
> > > > >>>> 3) how new client code handle the old HB? Does it always
> generate
> > > > member
> > > > >>>> ID on client-side even though that is not restricted?
> > > > >>>>
> > > > >>>> Best,
> > > > >>>> Chia-Ping
> > > > >>>>
> > > > >>>> On 2024/08/13 06:20:42 TengYao Chi wrote:
> > > > >>>>> Hello everyone,
> > > > >>>>>
> > > > >>>>> I would like to start a discussion thread on KIP-1082, which
> > > proposes
> > > > >>>>> enabling id generation for clients over the
> > ConsumerGroupHeartbeat
> > > > RPC.
> > > > >>>>>
> > > > >>>>> Here is the KIP Link: KIP-1082
> > > > >>>>> <
> > > > >>>>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
> > > > >>>>>
> > > > >>>>> Please take a look and let me know what you think, and I would
> > > > >> appreciate
> > > > >>>>> any suggestions and feedback.
> > > > >>>>>
> > > > >>>>> Best regards,
> > > > >>>>> TengYao
> > > > >>>>>
> > > > >>>>
> > > > >>
> > > > >>
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[Chia-Ping Tsai <ch...@gmail.com>]

hi Apoorv

> As the memberId is now known to the client, and client might send the
leave
group heartbeat on shutdown prior to receiving the initial heartbeat
response. If that's true then how do we guarantee that the 2 requests to
join and leave will be processed in order, which could still leave stale
members or throw unknown member id exceptions?

This is definitely a good question. the short answer: no guarantee but best
efforts

Please notice the root cause is "we have no enough time to wait member id
(response) when closing consumer". Sadly, we can' guarantee the request
order due to the same reason.

However, in contrast to previous behavior, there is one big benefit of new
approach - we can try STONITH because we know the member id

Best,
Chia-Ping


Apoorv Mittal <ap...@gmail.com> 於 2024年8月14日 週三 下午8:55寫道：

> Hi TengYao,
> Thanks for the KIP. Continuing on the point which Andrew mentioned as AS1.
>
> As the memberId is now known to the client, and client might send the leave
> group heartbeat on shutdown prior to receiving the initial heartbeat
> response. If that's true then how do we guarantee that the 2 requests to
> join and leave will be processed in order, which could still leave stale
> members or throw unknown member id exceptions?
>
> Though the client side member id generation is helpful which will represent
> the same group perspective as from client and broker's end. But I think the
> major concern we want to solve here is Stale Partition Assignments which
> might still exist with the new approach. I am leaning towards the
> suggestion mentioned by Andrew where partition assignment triggers on
> subsequent heartbeat when client acknowledges the initial heartbeat,
> delayed partition assignment.
>
> Though on a separate note, I have a different question. What happens when
> there is an issue with the client which sends the initial heartbeat without
> memberId, then crashes and restarts? I think we must be re-triggering
> assignments and expiring members only after the heartbeat session timeout?
> If that's true then shall delayed partition assignment can help benefit us
> from this situation as well?
>
> Regards,
> Apoorv Mittal
>
>
> On Wed, Aug 14, 2024 at 12:51 PM David Jacot <dj...@confluent.io.invalid>
> wrote:
>
> > Hi Andrew,
> >
> > Personally, I don't like the lobby approach. It makes things more
> > complicated and it would require changing the records on the server too.
> > This is why I initially suggested the rejected alternative #2 which is
> > pretty close but also not perfect.
> >
> > I'd like to clarify one thing. The ConsumerGroupHeartbeat API already
> > supports generating the member id on the client so we don't need any
> > conditional logic on the client side. This is actually what we wanted to
> do
> > in the first place but the idea got pushed back by Magnus back then
> because
> > generating uuid from librdkafka required a new dependency. It turns out
> > that librdkafka has that dependency today. In retrospect, we should have
> > pushed back on this. Long story short, we can just do it. The proposal in
> > this KIP is to make the member id required in future versions. We could
> > also decide not to do it and to keep supporting both approaches. I would
> > also be fine with this.
> >
> > Best,
> > David
> >
> > On Wed, Aug 14, 2024 at 12:30 PM Andrew Schofield <
> > andrew_schofield@live.com>
> > wrote:
> >
> > > Hi TengYao,
> > > Thanks for your response. I’ll have just one more try to persuade.
> > > I feel that I will need to follow the approach with KIP-932 when we’ve
> > > made a decision, so I do have more than a passing interest in this.
> > >
> > > A group member in the lobby is in the group, but it does not have any
> > > assignments. A member of a consumer group can have no assigned
> > > partitions (such as 5 CG members subscribed to a topic with 4
> > partitions),
> > > so it’s a situation that consumer group members already expect.
> > >
> > > One of Kafka’s strengths is the way that we handle API versioning.
> > > But, there is a cost - the behaviour is different depending on the RPC
> > > version. KIP-848 is on the cusp of completion, but we’re already adding
> > > conditional logic for v0/v1 for ConsumerGroupHeartbeat. That’s a pity.
> > > Only a minor issue, but it’s unfortunate.
> > >
> > > Thanks,
> > > Andrew
> > >
> > > > On 14 Aug 2024, at 08:47, TengYao Chi <ki...@gmail.com> wrote:
> > > >
> > > > Hello Andrew
> > > > Thank you for your thoughtful suggestions and getting the discussion
> > > going.
> > > >
> > > > To AS1:
> > > > In the current scenario where the server generates the UUID, if the
> > > client
> > > > shuts down before receiving the memberId generated by the GC
> > (regardless
> > > of
> > > > whether it’s a graceful shutdown or not), the GC will still have to
> > wait
> > > > for the heartbeat timeout because the client doesn’t know its
> memberId.
> > > > This KIP indeed cannot completely resolve the idempotency issue, but
> it
> > > can
> > > > better handle shutdown scenarios under normal circumstances because
> the
> > > > client always knows its memberId. Even if the client shuts down
> > > immediately
> > > > after the initial heartbeat, as long as it performs a graceful
> shutdown
> > > and
> > > > sends a leave heartbeat, the GC can manage the situation and remove
> the
> > > > member. Therefore, the goal of this KIP is to address the issue where
> > the
> > > > GC has to wait for the heartbeat timeout due to the client leaving
> > > without
> > > > knowing its memberId, which leads to reduced throughput and limited
> > > > scalability.
> > > >
> > > > The solution you suggest has also been proposed by David. The concern
> > > with
> > > > this approach is that it introduces additional complexity for
> > > > compatibility, as the new server would not immediately add the member
> > to
> > > > the group, while the old server would. This requires clients to
> > > > differentiate whether their memberId has been added to the group or
> > not,
> > > > which could result in unexpected logs.
> > > >
> > > > Best Regards,
> > > > TengYao
> > > >
> > > > Andrew Schofield <an...@live.com> 於 2024年8月14日 週三
> > 上午12:29寫道：
> > > >
> > > >> Hi TengYao,
> > > >> Thanks for the KIP. I wonder if there’s a different way to close
> what
> > > >> is quite a small window.
> > > >>
> > > >> AS1: It is true that the initial heartbeat is not idempotent, but
> this
> > > >> remains
> > > >> true with this KIP. It’s just differently not idempotent. If the
> > client
> > > >> makes its
> > > >> own member ID, sends a request and dies, the GC will still have
> added
> > > >> the member to the group and it will hang around until the session
> > > expires.
> > > >>
> > > >> I wonder if the GC could still generate the member ID in response to
> > the
> > > >> first
> > > >> heartbeat, and put the member in a special PENDING state with no
> > > >> assignments until the client sends the next heartbeat, thus
> confirming
> > > it
> > > >> has received the member ID. This would not be a protocol change at
> > all,
> > > >> just
> > > >> a change to the GC to keep a new member in the lobby until it’s
> > > comfirmed
> > > >> it knows its member ID.
> > > >>
> > > >>
> > > >> Thanks,
> > > >> Andrew
> > > >>
> > > >>> On 13 Aug 2024, at 15:59, TengYao Chi <ki...@gmail.com> wrote:
> > > >>>
> > > >>> Hi Chia-Ping,
> > > >>>
> > > >>> Thanks for review and suggestions.
> > > >>> I have updated the content of KIP accordingly.
> > > >>> Please take a look.
> > > >>>
> > > >>> Best regards,
> > > >>> TengYao
> > > >>>
> > > >>> Chia-Ping Tsai <ch...@apache.org> 於 2024年8月13日 週二 下午9:45寫道：
> > > >>>
> > > >>>> hi TengYao
> > > >>>>
> > > >>>> thanks for this KIP.
> > > >>>>
> > > >>>> 1) could you please describe the before/after behavior in the
> > > "Proposed
> > > >>>> Changes" section? IIRC, current RPC allows HB having member id
> > > >> generated by
> > > >>>> client, right? If HB has no member ID, server will generate one
> and
> > > then
> > > >>>> return. The new behavior will enforce HB "must" have member ID.
> > > >>>>
> > > >>>> 2) could you please write the version number explicitly in the KIP
> > > >>>>
> > > >>>> 3) how new client code handle the old HB? Does it always generate
> > > member
> > > >>>> ID on client-side even though that is not restricted?
> > > >>>>
> > > >>>> Best,
> > > >>>> Chia-Ping
> > > >>>>
> > > >>>> On 2024/08/13 06:20:42 TengYao Chi wrote:
> > > >>>>> Hello everyone,
> > > >>>>>
> > > >>>>> I would like to start a discussion thread on KIP-1082, which
> > proposes
> > > >>>>> enabling id generation for clients over the
> ConsumerGroupHeartbeat
> > > RPC.
> > > >>>>>
> > > >>>>> Here is the KIP Link: KIP-1082
> > > >>>>> <
> > > >>>>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
> > > >>>>>
> > > >>>>> Please take a look and let me know what you think, and I would
> > > >> appreciate
> > > >>>>> any suggestions and feedback.
> > > >>>>>
> > > >>>>> Best regards,
> > > >>>>> TengYao
> > > >>>>>
> > > >>>>
> > > >>
> > > >>
> > >
> > >
> >
>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[Apoorv Mittal <ap...@gmail.com>]

Hi TengYao,
Thanks for the KIP. Continuing on the point which Andrew mentioned as AS1.

As the memberId is now known to the client, and client might send the leave
group heartbeat on shutdown prior to receiving the initial heartbeat
response. If that's true then how do we guarantee that the 2 requests to
join and leave will be processed in order, which could still leave stale
members or throw unknown member id exceptions?

Though the client side member id generation is helpful which will represent
the same group perspective as from client and broker's end. But I think the
major concern we want to solve here is Stale Partition Assignments which
might still exist with the new approach. I am leaning towards the
suggestion mentioned by Andrew where partition assignment triggers on
subsequent heartbeat when client acknowledges the initial heartbeat,
delayed partition assignment.

Though on a separate note, I have a different question. What happens when
there is an issue with the client which sends the initial heartbeat without
memberId, then crashes and restarts? I think we must be re-triggering
assignments and expiring members only after the heartbeat session timeout?
If that's true then shall delayed partition assignment can help benefit us
from this situation as well?

Regards,
Apoorv Mittal


On Wed, Aug 14, 2024 at 12:51 PM David Jacot <dj...@confluent.io.invalid>
wrote:

> Hi Andrew,
>
> Personally, I don't like the lobby approach. It makes things more
> complicated and it would require changing the records on the server too.
> This is why I initially suggested the rejected alternative #2 which is
> pretty close but also not perfect.
>
> I'd like to clarify one thing. The ConsumerGroupHeartbeat API already
> supports generating the member id on the client so we don't need any
> conditional logic on the client side. This is actually what we wanted to do
> in the first place but the idea got pushed back by Magnus back then because
> generating uuid from librdkafka required a new dependency. It turns out
> that librdkafka has that dependency today. In retrospect, we should have
> pushed back on this. Long story short, we can just do it. The proposal in
> this KIP is to make the member id required in future versions. We could
> also decide not to do it and to keep supporting both approaches. I would
> also be fine with this.
>
> Best,
> David
>
> On Wed, Aug 14, 2024 at 12:30 PM Andrew Schofield <
> andrew_schofield@live.com>
> wrote:
>
> > Hi TengYao,
> > Thanks for your response. I’ll have just one more try to persuade.
> > I feel that I will need to follow the approach with KIP-932 when we’ve
> > made a decision, so I do have more than a passing interest in this.
> >
> > A group member in the lobby is in the group, but it does not have any
> > assignments. A member of a consumer group can have no assigned
> > partitions (such as 5 CG members subscribed to a topic with 4
> partitions),
> > so it’s a situation that consumer group members already expect.
> >
> > One of Kafka’s strengths is the way that we handle API versioning.
> > But, there is a cost - the behaviour is different depending on the RPC
> > version. KIP-848 is on the cusp of completion, but we’re already adding
> > conditional logic for v0/v1 for ConsumerGroupHeartbeat. That’s a pity.
> > Only a minor issue, but it’s unfortunate.
> >
> > Thanks,
> > Andrew
> >
> > > On 14 Aug 2024, at 08:47, TengYao Chi <ki...@gmail.com> wrote:
> > >
> > > Hello Andrew
> > > Thank you for your thoughtful suggestions and getting the discussion
> > going.
> > >
> > > To AS1:
> > > In the current scenario where the server generates the UUID, if the
> > client
> > > shuts down before receiving the memberId generated by the GC
> (regardless
> > of
> > > whether it’s a graceful shutdown or not), the GC will still have to
> wait
> > > for the heartbeat timeout because the client doesn’t know its memberId.
> > > This KIP indeed cannot completely resolve the idempotency issue, but it
> > can
> > > better handle shutdown scenarios under normal circumstances because the
> > > client always knows its memberId. Even if the client shuts down
> > immediately
> > > after the initial heartbeat, as long as it performs a graceful shutdown
> > and
> > > sends a leave heartbeat, the GC can manage the situation and remove the
> > > member. Therefore, the goal of this KIP is to address the issue where
> the
> > > GC has to wait for the heartbeat timeout due to the client leaving
> > without
> > > knowing its memberId, which leads to reduced throughput and limited
> > > scalability.
> > >
> > > The solution you suggest has also been proposed by David. The concern
> > with
> > > this approach is that it introduces additional complexity for
> > > compatibility, as the new server would not immediately add the member
> to
> > > the group, while the old server would. This requires clients to
> > > differentiate whether their memberId has been added to the group or
> not,
> > > which could result in unexpected logs.
> > >
> > > Best Regards,
> > > TengYao
> > >
> > > Andrew Schofield <an...@live.com> 於 2024年8月14日 週三
> 上午12:29寫道：
> > >
> > >> Hi TengYao,
> > >> Thanks for the KIP. I wonder if there’s a different way to close what
> > >> is quite a small window.
> > >>
> > >> AS1: It is true that the initial heartbeat is not idempotent, but this
> > >> remains
> > >> true with this KIP. It’s just differently not idempotent. If the
> client
> > >> makes its
> > >> own member ID, sends a request and dies, the GC will still have added
> > >> the member to the group and it will hang around until the session
> > expires.
> > >>
> > >> I wonder if the GC could still generate the member ID in response to
> the
> > >> first
> > >> heartbeat, and put the member in a special PENDING state with no
> > >> assignments until the client sends the next heartbeat, thus confirming
> > it
> > >> has received the member ID. This would not be a protocol change at
> all,
> > >> just
> > >> a change to the GC to keep a new member in the lobby until it’s
> > comfirmed
> > >> it knows its member ID.
> > >>
> > >>
> > >> Thanks,
> > >> Andrew
> > >>
> > >>> On 13 Aug 2024, at 15:59, TengYao Chi <ki...@gmail.com> wrote:
> > >>>
> > >>> Hi Chia-Ping,
> > >>>
> > >>> Thanks for review and suggestions.
> > >>> I have updated the content of KIP accordingly.
> > >>> Please take a look.
> > >>>
> > >>> Best regards,
> > >>> TengYao
> > >>>
> > >>> Chia-Ping Tsai <ch...@apache.org> 於 2024年8月13日 週二 下午9:45寫道：
> > >>>
> > >>>> hi TengYao
> > >>>>
> > >>>> thanks for this KIP.
> > >>>>
> > >>>> 1) could you please describe the before/after behavior in the
> > "Proposed
> > >>>> Changes" section? IIRC, current RPC allows HB having member id
> > >> generated by
> > >>>> client, right? If HB has no member ID, server will generate one and
> > then
> > >>>> return. The new behavior will enforce HB "must" have member ID.
> > >>>>
> > >>>> 2) could you please write the version number explicitly in the KIP
> > >>>>
> > >>>> 3) how new client code handle the old HB? Does it always generate
> > member
> > >>>> ID on client-side even though that is not restricted?
> > >>>>
> > >>>> Best,
> > >>>> Chia-Ping
> > >>>>
> > >>>> On 2024/08/13 06:20:42 TengYao Chi wrote:
> > >>>>> Hello everyone,
> > >>>>>
> > >>>>> I would like to start a discussion thread on KIP-1082, which
> proposes
> > >>>>> enabling id generation for clients over the ConsumerGroupHeartbeat
> > RPC.
> > >>>>>
> > >>>>> Here is the KIP Link: KIP-1082
> > >>>>> <
> > >>>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
> > >>>>>
> > >>>>> Please take a look and let me know what you think, and I would
> > >> appreciate
> > >>>>> any suggestions and feedback.
> > >>>>>
> > >>>>> Best regards,
> > >>>>> TengYao
> > >>>>>
> > >>>>
> > >>
> > >>
> >
> >
>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[Chia-Ping Tsai <ch...@gmail.com>]

> I think that this should be defined. Otherwise, folks may start using
incorrect things. In the java client, we will generate a Kafka UUID.
The javadoc of the Uuid class has a good definition for it.

That makes sense to me. However, there is no binding agreement when we only
put the recommendations. I means we can't stop folks from using incorrect
things unless the field "member id" is defined as UUID (for example, change
the protocol type from string to UUID)

> The proposal in
this KIP is to make the member id required in future versions. We could
also decide not to do it and to keep supporting both approaches. I would
also be fine with this.

IMHO, that is a kind of "incorrect thing". If we all agree the behavior of
this version could cause bugs (maybe the bugs are rare but they are indeed
existent), then it is worth having the new behavior to protect folks from
the possible bugs.

Best,
Chia-Ping

David Jacot <dj...@confluent.io.invalid> 於 2024年8月14日 週三 下午7:50寫道：

> Hi Andrew,
>
> Personally, I don't like the lobby approach. It makes things more
> complicated and it would require changing the records on the server too.
> This is why I initially suggested the rejected alternative #2 which is
> pretty close but also not perfect.
>
> I'd like to clarify one thing. The ConsumerGroupHeartbeat API already
> supports generating the member id on the client so we don't need any
> conditional logic on the client side. This is actually what we wanted to do
> in the first place but the idea got pushed back by Magnus back then because
> generating uuid from librdkafka required a new dependency. It turns out
> that librdkafka has that dependency today. In retrospect, we should have
> pushed back on this. Long story short, we can just do it. The proposal in
> this KIP is to make the member id required in future versions. We could
> also decide not to do it and to keep supporting both approaches. I would
> also be fine with this.
>
> Best,
> David
>
> On Wed, Aug 14, 2024 at 12:30 PM Andrew Schofield <
> andrew_schofield@live.com>
> wrote:
>
> > Hi TengYao,
> > Thanks for your response. I’ll have just one more try to persuade.
> > I feel that I will need to follow the approach with KIP-932 when we’ve
> > made a decision, so I do have more than a passing interest in this.
> >
> > A group member in the lobby is in the group, but it does not have any
> > assignments. A member of a consumer group can have no assigned
> > partitions (such as 5 CG members subscribed to a topic with 4
> partitions),
> > so it’s a situation that consumer group members already expect.
> >
> > One of Kafka’s strengths is the way that we handle API versioning.
> > But, there is a cost - the behaviour is different depending on the RPC
> > version. KIP-848 is on the cusp of completion, but we’re already adding
> > conditional logic for v0/v1 for ConsumerGroupHeartbeat. That’s a pity.
> > Only a minor issue, but it’s unfortunate.
> >
> > Thanks,
> > Andrew
> >
> > > On 14 Aug 2024, at 08:47, TengYao Chi <ki...@gmail.com> wrote:
> > >
> > > Hello Andrew
> > > Thank you for your thoughtful suggestions and getting the discussion
> > going.
> > >
> > > To AS1:
> > > In the current scenario where the server generates the UUID, if the
> > client
> > > shuts down before receiving the memberId generated by the GC
> (regardless
> > of
> > > whether it’s a graceful shutdown or not), the GC will still have to
> wait
> > > for the heartbeat timeout because the client doesn’t know its memberId.
> > > This KIP indeed cannot completely resolve the idempotency issue, but it
> > can
> > > better handle shutdown scenarios under normal circumstances because the
> > > client always knows its memberId. Even if the client shuts down
> > immediately
> > > after the initial heartbeat, as long as it performs a graceful shutdown
> > and
> > > sends a leave heartbeat, the GC can manage the situation and remove the
> > > member. Therefore, the goal of this KIP is to address the issue where
> the
> > > GC has to wait for the heartbeat timeout due to the client leaving
> > without
> > > knowing its memberId, which leads to reduced throughput and limited
> > > scalability.
> > >
> > > The solution you suggest has also been proposed by David. The concern
> > with
> > > this approach is that it introduces additional complexity for
> > > compatibility, as the new server would not immediately add the member
> to
> > > the group, while the old server would. This requires clients to
> > > differentiate whether their memberId has been added to the group or
> not,
> > > which could result in unexpected logs.
> > >
> > > Best Regards,
> > > TengYao
> > >
> > > Andrew Schofield <an...@live.com> 於 2024年8月14日 週三
> 上午12:29寫道：
> > >
> > >> Hi TengYao,
> > >> Thanks for the KIP. I wonder if there’s a different way to close what
> > >> is quite a small window.
> > >>
> > >> AS1: It is true that the initial heartbeat is not idempotent, but this
> > >> remains
> > >> true with this KIP. It’s just differently not idempotent. If the
> client
> > >> makes its
> > >> own member ID, sends a request and dies, the GC will still have added
> > >> the member to the group and it will hang around until the session
> > expires.
> > >>
> > >> I wonder if the GC could still generate the member ID in response to
> the
> > >> first
> > >> heartbeat, and put the member in a special PENDING state with no
> > >> assignments until the client sends the next heartbeat, thus confirming
> > it
> > >> has received the member ID. This would not be a protocol change at
> all,
> > >> just
> > >> a change to the GC to keep a new member in the lobby until it’s
> > comfirmed
> > >> it knows its member ID.
> > >>
> > >>
> > >> Thanks,
> > >> Andrew
> > >>
> > >>> On 13 Aug 2024, at 15:59, TengYao Chi <ki...@gmail.com> wrote:
> > >>>
> > >>> Hi Chia-Ping,
> > >>>
> > >>> Thanks for review and suggestions.
> > >>> I have updated the content of KIP accordingly.
> > >>> Please take a look.
> > >>>
> > >>> Best regards,
> > >>> TengYao
> > >>>
> > >>> Chia-Ping Tsai <ch...@apache.org> 於 2024年8月13日 週二 下午9:45寫道：
> > >>>
> > >>>> hi TengYao
> > >>>>
> > >>>> thanks for this KIP.
> > >>>>
> > >>>> 1) could you please describe the before/after behavior in the
> > "Proposed
> > >>>> Changes" section? IIRC, current RPC allows HB having member id
> > >> generated by
> > >>>> client, right? If HB has no member ID, server will generate one and
> > then
> > >>>> return. The new behavior will enforce HB "must" have member ID.
> > >>>>
> > >>>> 2) could you please write the version number explicitly in the KIP
> > >>>>
> > >>>> 3) how new client code handle the old HB? Does it always generate
> > member
> > >>>> ID on client-side even though that is not restricted?
> > >>>>
> > >>>> Best,
> > >>>> Chia-Ping
> > >>>>
> > >>>> On 2024/08/13 06:20:42 TengYao Chi wrote:
> > >>>>> Hello everyone,
> > >>>>>
> > >>>>> I would like to start a discussion thread on KIP-1082, which
> proposes
> > >>>>> enabling id generation for clients over the ConsumerGroupHeartbeat
> > RPC.
> > >>>>>
> > >>>>> Here is the KIP Link: KIP-1082
> > >>>>> <
> > >>>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
> > >>>>>
> > >>>>> Please take a look and let me know what you think, and I would
> > >> appreciate
> > >>>>> any suggestions and feedback.
> > >>>>>
> > >>>>> Best regards,
> > >>>>> TengYao
> > >>>>>
> > >>>>
> > >>
> > >>
> >
> >
>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[David Jacot <dj...@confluent.io.INVALID>]

Hi Andrew,

Personally, I don't like the lobby approach. It makes things more
complicated and it would require changing the records on the server too.
This is why I initially suggested the rejected alternative #2 which is
pretty close but also not perfect.

I'd like to clarify one thing. The ConsumerGroupHeartbeat API already
supports generating the member id on the client so we don't need any
conditional logic on the client side. This is actually what we wanted to do
in the first place but the idea got pushed back by Magnus back then because
generating uuid from librdkafka required a new dependency. It turns out
that librdkafka has that dependency today. In retrospect, we should have
pushed back on this. Long story short, we can just do it. The proposal in
this KIP is to make the member id required in future versions. We could
also decide not to do it and to keep supporting both approaches. I would
also be fine with this.

Best,
David

On Wed, Aug 14, 2024 at 12:30 PM Andrew Schofield <an...@live.com>
wrote:

> Hi TengYao,
> Thanks for your response. I’ll have just one more try to persuade.
> I feel that I will need to follow the approach with KIP-932 when we’ve
> made a decision, so I do have more than a passing interest in this.
>
> A group member in the lobby is in the group, but it does not have any
> assignments. A member of a consumer group can have no assigned
> partitions (such as 5 CG members subscribed to a topic with 4 partitions),
> so it’s a situation that consumer group members already expect.
>
> One of Kafka’s strengths is the way that we handle API versioning.
> But, there is a cost - the behaviour is different depending on the RPC
> version. KIP-848 is on the cusp of completion, but we’re already adding
> conditional logic for v0/v1 for ConsumerGroupHeartbeat. That’s a pity.
> Only a minor issue, but it’s unfortunate.
>
> Thanks,
> Andrew
>
> > On 14 Aug 2024, at 08:47, TengYao Chi <ki...@gmail.com> wrote:
> >
> > Hello Andrew
> > Thank you for your thoughtful suggestions and getting the discussion
> going.
> >
> > To AS1:
> > In the current scenario where the server generates the UUID, if the
> client
> > shuts down before receiving the memberId generated by the GC (regardless
> of
> > whether it’s a graceful shutdown or not), the GC will still have to wait
> > for the heartbeat timeout because the client doesn’t know its memberId.
> > This KIP indeed cannot completely resolve the idempotency issue, but it
> can
> > better handle shutdown scenarios under normal circumstances because the
> > client always knows its memberId. Even if the client shuts down
> immediately
> > after the initial heartbeat, as long as it performs a graceful shutdown
> and
> > sends a leave heartbeat, the GC can manage the situation and remove the
> > member. Therefore, the goal of this KIP is to address the issue where the
> > GC has to wait for the heartbeat timeout due to the client leaving
> without
> > knowing its memberId, which leads to reduced throughput and limited
> > scalability.
> >
> > The solution you suggest has also been proposed by David. The concern
> with
> > this approach is that it introduces additional complexity for
> > compatibility, as the new server would not immediately add the member to
> > the group, while the old server would. This requires clients to
> > differentiate whether their memberId has been added to the group or not,
> > which could result in unexpected logs.
> >
> > Best Regards,
> > TengYao
> >
> > Andrew Schofield <an...@live.com> 於 2024年8月14日 週三 上午12:29寫道：
> >
> >> Hi TengYao,
> >> Thanks for the KIP. I wonder if there’s a different way to close what
> >> is quite a small window.
> >>
> >> AS1: It is true that the initial heartbeat is not idempotent, but this
> >> remains
> >> true with this KIP. It’s just differently not idempotent. If the client
> >> makes its
> >> own member ID, sends a request and dies, the GC will still have added
> >> the member to the group and it will hang around until the session
> expires.
> >>
> >> I wonder if the GC could still generate the member ID in response to the
> >> first
> >> heartbeat, and put the member in a special PENDING state with no
> >> assignments until the client sends the next heartbeat, thus confirming
> it
> >> has received the member ID. This would not be a protocol change at all,
> >> just
> >> a change to the GC to keep a new member in the lobby until it’s
> comfirmed
> >> it knows its member ID.
> >>
> >>
> >> Thanks,
> >> Andrew
> >>
> >>> On 13 Aug 2024, at 15:59, TengYao Chi <ki...@gmail.com> wrote:
> >>>
> >>> Hi Chia-Ping,
> >>>
> >>> Thanks for review and suggestions.
> >>> I have updated the content of KIP accordingly.
> >>> Please take a look.
> >>>
> >>> Best regards,
> >>> TengYao
> >>>
> >>> Chia-Ping Tsai <ch...@apache.org> 於 2024年8月13日 週二 下午9:45寫道：
> >>>
> >>>> hi TengYao
> >>>>
> >>>> thanks for this KIP.
> >>>>
> >>>> 1) could you please describe the before/after behavior in the
> "Proposed
> >>>> Changes" section? IIRC, current RPC allows HB having member id
> >> generated by
> >>>> client, right? If HB has no member ID, server will generate one and
> then
> >>>> return. The new behavior will enforce HB "must" have member ID.
> >>>>
> >>>> 2) could you please write the version number explicitly in the KIP
> >>>>
> >>>> 3) how new client code handle the old HB? Does it always generate
> member
> >>>> ID on client-side even though that is not restricted?
> >>>>
> >>>> Best,
> >>>> Chia-Ping
> >>>>
> >>>> On 2024/08/13 06:20:42 TengYao Chi wrote:
> >>>>> Hello everyone,
> >>>>>
> >>>>> I would like to start a discussion thread on KIP-1082, which proposes
> >>>>> enabling id generation for clients over the ConsumerGroupHeartbeat
> RPC.
> >>>>>
> >>>>> Here is the KIP Link: KIP-1082
> >>>>> <
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
> >>>>>
> >>>>> Please take a look and let me know what you think, and I would
> >> appreciate
> >>>>> any suggestions and feedback.
> >>>>>
> >>>>> Best regards,
> >>>>> TengYao
> >>>>>
> >>>>
> >>
> >>
>
>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[Andrew Schofield <an...@live.com>]

Hi TengYao,
Thanks for your response. I’ll have just one more try to persuade.
I feel that I will need to follow the approach with KIP-932 when we’ve
made a decision, so I do have more than a passing interest in this.

A group member in the lobby is in the group, but it does not have any
assignments. A member of a consumer group can have no assigned
partitions (such as 5 CG members subscribed to a topic with 4 partitions),
so it’s a situation that consumer group members already expect.

One of Kafka’s strengths is the way that we handle API versioning.
But, there is a cost - the behaviour is different depending on the RPC
version. KIP-848 is on the cusp of completion, but we’re already adding
conditional logic for v0/v1 for ConsumerGroupHeartbeat. That’s a pity.
Only a minor issue, but it’s unfortunate.

Thanks,
Andrew

> On 14 Aug 2024, at 08:47, TengYao Chi <ki...@gmail.com> wrote:
>
> Hello Andrew
> Thank you for your thoughtful suggestions and getting the discussion going.
>
> To AS1:
> In the current scenario where the server generates the UUID, if the client
> shuts down before receiving the memberId generated by the GC (regardless of
> whether it’s a graceful shutdown or not), the GC will still have to wait
> for the heartbeat timeout because the client doesn’t know its memberId.
> This KIP indeed cannot completely resolve the idempotency issue, but it can
> better handle shutdown scenarios under normal circumstances because the
> client always knows its memberId. Even if the client shuts down immediately
> after the initial heartbeat, as long as it performs a graceful shutdown and
> sends a leave heartbeat, the GC can manage the situation and remove the
> member. Therefore, the goal of this KIP is to address the issue where the
> GC has to wait for the heartbeat timeout due to the client leaving without
> knowing its memberId, which leads to reduced throughput and limited
> scalability.
>
> The solution you suggest has also been proposed by David. The concern with
> this approach is that it introduces additional complexity for
> compatibility, as the new server would not immediately add the member to
> the group, while the old server would. This requires clients to
> differentiate whether their memberId has been added to the group or not,
> which could result in unexpected logs.
>
> Best Regards,
> TengYao
>
> Andrew Schofield <an...@live.com> 於 2024年8月14日 週三 上午12:29寫道：
>
>> Hi TengYao,
>> Thanks for the KIP. I wonder if there’s a different way to close what
>> is quite a small window.
>>
>> AS1: It is true that the initial heartbeat is not idempotent, but this
>> remains
>> true with this KIP. It’s just differently not idempotent. If the client
>> makes its
>> own member ID, sends a request and dies, the GC will still have added
>> the member to the group and it will hang around until the session expires.
>>
>> I wonder if the GC could still generate the member ID in response to the
>> first
>> heartbeat, and put the member in a special PENDING state with no
>> assignments until the client sends the next heartbeat, thus confirming it
>> has received the member ID. This would not be a protocol change at all,
>> just
>> a change to the GC to keep a new member in the lobby until it’s comfirmed
>> it knows its member ID.
>>
>>
>> Thanks,
>> Andrew
>>
>>> On 13 Aug 2024, at 15:59, TengYao Chi <ki...@gmail.com> wrote:
>>>
>>> Hi Chia-Ping,
>>>
>>> Thanks for review and suggestions.
>>> I have updated the content of KIP accordingly.
>>> Please take a look.
>>>
>>> Best regards,
>>> TengYao
>>>
>>> Chia-Ping Tsai <ch...@apache.org> 於 2024年8月13日 週二 下午9:45寫道：
>>>
>>>> hi TengYao
>>>>
>>>> thanks for this KIP.
>>>>
>>>> 1) could you please describe the before/after behavior in the "Proposed
>>>> Changes" section? IIRC, current RPC allows HB having member id
>> generated by
>>>> client, right? If HB has no member ID, server will generate one and then
>>>> return. The new behavior will enforce HB "must" have member ID.
>>>>
>>>> 2) could you please write the version number explicitly in the KIP
>>>>
>>>> 3) how new client code handle the old HB? Does it always generate member
>>>> ID on client-side even though that is not restricted?
>>>>
>>>> Best,
>>>> Chia-Ping
>>>>
>>>> On 2024/08/13 06:20:42 TengYao Chi wrote:
>>>>> Hello everyone,
>>>>>
>>>>> I would like to start a discussion thread on KIP-1082, which proposes
>>>>> enabling id generation for clients over the ConsumerGroupHeartbeat RPC.
>>>>>
>>>>> Here is the KIP Link: KIP-1082
>>>>> <
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
>>>>>
>>>>> Please take a look and let me know what you think, and I would
>> appreciate
>>>>> any suggestions and feedback.
>>>>>
>>>>> Best regards,
>>>>> TengYao
>>>>>
>>>>
>>
>>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[TengYao Chi <ki...@gmail.com>]

Hello Andrew
Thank you for your thoughtful suggestions and getting the discussion going.

To AS1:
In the current scenario where the server generates the UUID, if the client
shuts down before receiving the memberId generated by the GC (regardless of
whether it’s a graceful shutdown or not), the GC will still have to wait
for the heartbeat timeout because the client doesn’t know its memberId.
This KIP indeed cannot completely resolve the idempotency issue, but it can
better handle shutdown scenarios under normal circumstances because the
client always knows its memberId. Even if the client shuts down immediately
after the initial heartbeat, as long as it performs a graceful shutdown and
sends a leave heartbeat, the GC can manage the situation and remove the
member. Therefore, the goal of this KIP is to address the issue where the
GC has to wait for the heartbeat timeout due to the client leaving without
knowing its memberId, which leads to reduced throughput and limited
scalability.

The solution you suggest has also been proposed by David. The concern with
this approach is that it introduces additional complexity for
compatibility, as the new server would not immediately add the member to
the group, while the old server would. This requires clients to
differentiate whether their memberId has been added to the group or not,
which could result in unexpected logs.

Best Regards,
TengYao

Andrew Schofield <an...@live.com> 於 2024年8月14日 週三 上午12:29寫道：

> Hi TengYao,
> Thanks for the KIP. I wonder if there’s a different way to close what
> is quite a small window.
>
> AS1: It is true that the initial heartbeat is not idempotent, but this
> remains
> true with this KIP. It’s just differently not idempotent. If the client
> makes its
> own member ID, sends a request and dies, the GC will still have added
> the member to the group and it will hang around until the session expires.
>
> I wonder if the GC could still generate the member ID in response to the
> first
> heartbeat, and put the member in a special PENDING state with no
> assignments until the client sends the next heartbeat, thus confirming it
> has received the member ID. This would not be a protocol change at all,
> just
> a change to the GC to keep a new member in the lobby until it’s comfirmed
> it knows its member ID.
>
>
> Thanks,
> Andrew
>
> > On 13 Aug 2024, at 15:59, TengYao Chi <ki...@gmail.com> wrote:
> >
> > Hi Chia-Ping,
> >
> > Thanks for review and suggestions.
> > I have updated the content of KIP accordingly.
> > Please take a look.
> >
> > Best regards,
> > TengYao
> >
> > Chia-Ping Tsai <ch...@apache.org> 於 2024年8月13日 週二 下午9:45寫道：
> >
> >> hi TengYao
> >>
> >> thanks for this KIP.
> >>
> >> 1) could you please describe the before/after behavior in the "Proposed
> >> Changes" section? IIRC, current RPC allows HB having member id
> generated by
> >> client, right? If HB has no member ID, server will generate one and then
> >> return. The new behavior will enforce HB "must" have member ID.
> >>
> >> 2) could you please write the version number explicitly in the KIP
> >>
> >> 3) how new client code handle the old HB? Does it always generate member
> >> ID on client-side even though that is not restricted?
> >>
> >> Best,
> >> Chia-Ping
> >>
> >> On 2024/08/13 06:20:42 TengYao Chi wrote:
> >>> Hello everyone,
> >>>
> >>> I would like to start a discussion thread on KIP-1082, which proposes
> >>> enabling id generation for clients over the ConsumerGroupHeartbeat RPC.
> >>>
> >>> Here is the KIP Link: KIP-1082
> >>> <
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
> >>>
> >>> Please take a look and let me know what you think, and I would
> appreciate
> >>> any suggestions and feedback.
> >>>
> >>> Best regards,
> >>> TengYao
> >>>
> >>
>
>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[Andrew Schofield <an...@live.com>]

Hi TengYao,
Thanks for the KIP. I wonder if there’s a different way to close what
is quite a small window.

AS1: It is true that the initial heartbeat is not idempotent, but this remains
true with this KIP. It’s just differently not idempotent. If the client makes its
own member ID, sends a request and dies, the GC will still have added
the member to the group and it will hang around until the session expires.

I wonder if the GC could still generate the member ID in response to the first
heartbeat, and put the member in a special PENDING state with no
assignments until the client sends the next heartbeat, thus confirming it
has received the member ID. This would not be a protocol change at all, just
a change to the GC to keep a new member in the lobby until it’s comfirmed
it knows its member ID.


Thanks,
Andrew

> On 13 Aug 2024, at 15:59, TengYao Chi <ki...@gmail.com> wrote:
>
> Hi Chia-Ping,
>
> Thanks for review and suggestions.
> I have updated the content of KIP accordingly.
> Please take a look.
>
> Best regards,
> TengYao
>
> Chia-Ping Tsai <ch...@apache.org> 於 2024年8月13日 週二 下午9:45寫道：
>
>> hi TengYao
>>
>> thanks for this KIP.
>>
>> 1) could you please describe the before/after behavior in the "Proposed
>> Changes" section? IIRC, current RPC allows HB having member id generated by
>> client, right? If HB has no member ID, server will generate one and then
>> return. The new behavior will enforce HB "must" have member ID.
>>
>> 2) could you please write the version number explicitly in the KIP
>>
>> 3) how new client code handle the old HB? Does it always generate member
>> ID on client-side even though that is not restricted?
>>
>> Best,
>> Chia-Ping
>>
>> On 2024/08/13 06:20:42 TengYao Chi wrote:
>>> Hello everyone,
>>>
>>> I would like to start a discussion thread on KIP-1082, which proposes
>>> enabling id generation for clients over the ConsumerGroupHeartbeat RPC.
>>>
>>> Here is the KIP Link: KIP-1082
>>> <
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
>>>
>>> Please take a look and let me know what you think, and I would appreciate
>>> any suggestions and feedback.
>>>
>>> Best regards,
>>> TengYao
>>>
>>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[TengYao Chi <ki...@gmail.com>]

Hi Chia-Ping,

Thanks for review and suggestions.
I have updated the content of KIP accordingly.
Please take a look.

Best regards,
TengYao

Chia-Ping Tsai <ch...@apache.org> 於 2024年8月13日 週二 下午9:45寫道：

> hi TengYao
>
> thanks for this KIP.
>
> 1) could you please describe the before/after behavior in the "Proposed
> Changes" section? IIRC, current RPC allows HB having member id generated by
> client, right? If HB has no member ID, server will generate one and then
> return. The new behavior will enforce HB "must" have member ID.
>
> 2) could you please write the version number explicitly in the KIP
>
> 3) how new client code handle the old HB? Does it always generate member
> ID on client-side even though that is not restricted?
>
> Best,
> Chia-Ping
>
> On 2024/08/13 06:20:42 TengYao Chi wrote:
> > Hello everyone,
> >
> > I would like to start a discussion thread on KIP-1082, which proposes
> > enabling id generation for clients over the ConsumerGroupHeartbeat RPC.
> >
> > Here is the KIP Link: KIP-1082
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC
> >
> > Please take a look and let me know what you think, and I would appreciate
> > any suggestions and feedback.
> >
> > Best regards,
> > TengYao
> >
>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[Chia-Ping Tsai <ch...@apache.org>]

hi TengYao

thanks for this KIP. 

1) could you please describe the before/after behavior in the "Proposed Changes" section? IIRC, current RPC allows HB having member id generated by client, right? If HB has no member ID, server will generate one and then return. The new behavior will enforce HB "must" have member ID.

2) could you please write the version number explicitly in the KIP

3) how new client code handle the old HB? Does it always generate member ID on client-side even though that is not restricted? 

Best,
Chia-Ping

On 2024/08/13 06:20:42 TengYao Chi wrote:
> Hello everyone,
> 
> I would like to start a discussion thread on KIP-1082, which proposes
> enabling id generation for clients over the ConsumerGroupHeartbeat RPC.
> 
> Here is the KIP Link: KIP-1082
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1082%3A+Enable+ID+Generation+for+Clients+over+the+ConsumerGroupHeartbeat+RPC>
> Please take a look and let me know what you think, and I would appreciate
> any suggestions and feedback.
> 
> Best regards,
> TengYao
>