You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Lucas Brutschy via dev <de...@kafka.apache.org> on 2025/12/18 19:59:56 UTC
[DISCUSS] KIP-1251: Assignment epochs for consumer groups
Hi all,
I posted a new KIP to add assignment epochs to KIP-848-style consumer
groups. The idea is to avoid failing offset commits in the new
consumer protocol by relaxing the member epoch validation slightly.
Please take a look and let me know what you think:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1251%3A+Assignment+epochs+for+consumer+groups
Cheers,
Lucas
Re: [DISCUSS] KIP-1251: Assignment epochs for consumer groups
Posted by Dongnuo Lyu via dev <de...@kafka.apache.org>.
Thanks Lucas for the update!
Re: [DISCUSS] KIP-1251: Assignment epochs for consumer groups
Posted by David Jacot via dev <de...@kafka.apache.org>.
Hi Lucas,
Thanks for the KIP! I have only a few minor comments:
dj01: In the description of the handling of static members, you state that
the assignment epoch will be set when the record is loaded. Did you mean
replayed? The record is not necessarily loaded during normal operation.
dj02: When replaying the record, it may be better to set the assigned
epochs to zero if the member epoch is negative. The negative value is only
used as a sentinel in the member epoch. I don't want it to be used in other
places.
dj03: We already have some code
in getOrMaybeSubscribeStaticConsumerGroupMember to reset some fields when a
static member is replaced. I wonder whether we should just follow this
pattern instead of nulling the field when the static member leaves the
group. This is an implementation detail but I wanted to raise it for
awareness.
dj04: It may be worth specifying that this KIP will completely eliminate
the transient STALE_MEMBER_EPOCH that we observe during rebalances with the
new protocol. So it will also improve the protocol's efficiency during
rebalances too.
Cheers,
David
On Tue, Jan 6, 2026 at 2:29 PM Lucas Brutschy via dev <de...@kafka.apache.org>
wrote:
> Hi Sean,
>
> sq01: Excellent point! I think the simplest approach would be to omit
> assignment epochs from current assignment records for static members
> that have left temporarily (memberEpoch = -2). When loading the
> record, treat it exactly like a legacy record; Assume all assignment
> epochs equal the member epoch (-2). Partitions would then be
> considered assigned to the new member ID "from the start."
>
> Cheers,
> Lucas
>
> On Sat, Jan 3, 2026 at 9:49 PM Sean Quah via dev <de...@kafka.apache.org>
> wrote:
> >
> > Hi Lucas,
> > Thanks for the KIP!
> >
> > sq01: Do you think we could describe what happens when a static member is
> > replaced? The new static member will inherit the previous member's
> > assignment, but will we also inherit the previous assignment epoch or
> reset
> > it to the new member's epoch?
> >
> > Thanks,
> > Sean
> >
> > On Thu, Dec 18, 2025 at 8:00 PM Lucas Brutschy via dev <
> dev@kafka.apache.org>
> > wrote:
> >
> > > Hi all,
> > >
> > > I posted a new KIP to add assignment epochs to KIP-848-style consumer
> > > groups. The idea is to avoid failing offset commits in the new
> > > consumer protocol by relaxing the member epoch validation slightly.
> > >
> > > Please take a look and let me know what you think:
> > >
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1251%3A+Assignment+epochs+for+consumer+groups
> > >
> > > Cheers,
> > > Lucas
> > >
>
Re: [DISCUSS] KIP-1251: Assignment epochs for consumer groups
Posted by Lucas Brutschy via dev <de...@kafka.apache.org>.
Hi Sean,
sq01: Excellent point! I think the simplest approach would be to omit
assignment epochs from current assignment records for static members
that have left temporarily (memberEpoch = -2). When loading the
record, treat it exactly like a legacy record; Assume all assignment
epochs equal the member epoch (-2). Partitions would then be
considered assigned to the new member ID "from the start."
Cheers,
Lucas
On Sat, Jan 3, 2026 at 9:49 PM Sean Quah via dev <de...@kafka.apache.org> wrote:
>
> Hi Lucas,
> Thanks for the KIP!
>
> sq01: Do you think we could describe what happens when a static member is
> replaced? The new static member will inherit the previous member's
> assignment, but will we also inherit the previous assignment epoch or reset
> it to the new member's epoch?
>
> Thanks,
> Sean
>
> On Thu, Dec 18, 2025 at 8:00 PM Lucas Brutschy via dev <de...@kafka.apache.org>
> wrote:
>
> > Hi all,
> >
> > I posted a new KIP to add assignment epochs to KIP-848-style consumer
> > groups. The idea is to avoid failing offset commits in the new
> > consumer protocol by relaxing the member epoch validation slightly.
> >
> > Please take a look and let me know what you think:
> >
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1251%3A+Assignment+epochs+for+consumer+groups
> >
> > Cheers,
> > Lucas
> >
Re: [DISCUSS] KIP-1251: Assignment epochs for consumer groups
Posted by Sean Quah via dev <de...@kafka.apache.org>.
Hi Lucas,
Thanks for the KIP!
sq01: Do you think we could describe what happens when a static member is
replaced? The new static member will inherit the previous member's
assignment, but will we also inherit the previous assignment epoch or reset
it to the new member's epoch?
Thanks,
Sean
On Thu, Dec 18, 2025 at 8:00 PM Lucas Brutschy via dev <de...@kafka.apache.org>
wrote:
> Hi all,
>
> I posted a new KIP to add assignment epochs to KIP-848-style consumer
> groups. The idea is to avoid failing offset commits in the new
> consumer protocol by relaxing the member epoch validation slightly.
>
> Please take a look and let me know what you think:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1251%3A+Assignment+epochs+for+consumer+groups
>
> Cheers,
> Lucas
>
Re: [DISCUSS] KIP-1251: Assignment epochs for consumer groups
Posted by Lucas Brutschy via dev <de...@kafka.apache.org>.
Hi everybody,
Sorry for the late reply.
dj01. This is what I meant, but I didn't use the precise terminology.
Updated in the KIP
dj02. That is fine with me, I updated the KIP to reflect that change.
dj03. Noted for the implementation.
dj04. Added to the KIP.
dl01. Good point, that's indeed a typo. I fixed it in the KIP.
dl02. Yes, added a small section on assignment epochs during group
protocol conversion to the KIP. Your suggestions make sense to me.
I will give this KIP a bit more time to be commented on, and then open
the vote thread.
Cheers,
Lucas
On Thu, Jan 8, 2026 at 9:38 PM Dongnuo Lyu via dev <de...@kafka.apache.org> wrote:
>
> Hi Lucas, thanks for the KIP! I've got 2 questions about the assignment
> epoch.
>
> dl01: To confirm I guess it should be a typo "reject all offset commit
> requests from a member that... includes any member epoch that is
> smaller or *equal
> to* the assignment epoch"?
>
> dl02: Should we add some discussion on assignment epochs during group
> protocol conversion? In the upgrade scenario, I think we can just set it to
> be the generation id as the classic protocol don't have such "fake zombie"
> issue; In the downgrade scenario, classic groups only check generation id
> in offset commit validation, so no need to keep the assignment epoch; In
> the mixed group scenario, there shouldn't be any difference because the
> classic members just don't have in flight commit during rebalance.
>
> Thanks,
> Dongnuo
>
> On Thu, Dec 18, 2025 at 3:00 PM Lucas Brutschy via dev <de...@kafka.apache.org>
> wrote:
>
> > Hi all,
> >
> > I posted a new KIP to add assignment epochs to KIP-848-style consumer
> > groups. The idea is to avoid failing offset commits in the new
> > consumer protocol by relaxing the member epoch validation slightly.
> >
> > Please take a look and let me know what you think:
> >
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1251%3A+Assignment+epochs+for+consumer+groups
> >
> > Cheers,
> > Lucas
> >
>
>
> --
>
> [image: Confluent] <https://www.confluent.io>
> Dongnuo Lyu
> Software Engineer
> +1 (919) 717-3048 <+1+(919)+717-3048>
> Follow us: [image: Blog]
> <https://www.confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog>[image:
> Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn]
> <https://www.linkedin.com/company/confluent/>[image: Slack]
> <https://slackpass.io/confluentcommunity>[image: YouTube]
> <https://youtube.com/confluent>
>
> [image: Try Confluent Cloud for Free]
> <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound&utm_source=gmail&utm_medium=organic>
Re: [DISCUSS] KIP-1251: Assignment epochs for consumer groups
Posted by Dongnuo Lyu via dev <de...@kafka.apache.org>.
Hi Lucas, thanks for the KIP! I've got 2 questions about the assignment
epoch.
dl01: To confirm I guess it should be a typo "reject all offset commit
requests from a member that... includes any member epoch that is
smaller or *equal
to* the assignment epoch"?
dl02: Should we add some discussion on assignment epochs during group
protocol conversion? In the upgrade scenario, I think we can just set it to
be the generation id as the classic protocol don't have such "fake zombie"
issue; In the downgrade scenario, classic groups only check generation id
in offset commit validation, so no need to keep the assignment epoch; In
the mixed group scenario, there shouldn't be any difference because the
classic members just don't have in flight commit during rebalance.
Thanks,
Dongnuo
On Thu, Dec 18, 2025 at 3:00 PM Lucas Brutschy via dev <de...@kafka.apache.org>
wrote:
> Hi all,
>
> I posted a new KIP to add assignment epochs to KIP-848-style consumer
> groups. The idea is to avoid failing offset commits in the new
> consumer protocol by relaxing the member epoch validation slightly.
>
> Please take a look and let me know what you think:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1251%3A+Assignment+epochs+for+consumer+groups
>
> Cheers,
> Lucas
>
--
[image: Confluent] <https://www.confluent.io>
Dongnuo Lyu
Software Engineer
+1 (919) 717-3048 <+1+(919)+717-3048>
Follow us: [image: Blog]
<https://www.confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog>[image:
Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn]
<https://www.linkedin.com/company/confluent/>[image: Slack]
<https://slackpass.io/confluentcommunity>[image: YouTube]
<https://youtube.com/confluent>
[image: Try Confluent Cloud for Free]
<https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound&utm_source=gmail&utm_medium=organic>
Re: [DISCUSS] KIP-1251: Assignment epochs for consumer groups
Posted by Chia-Ping Tsai <ch...@apache.org>.
> did not focus on it. If we want this, we should probably write a
> separate KIP and have a separate discussion.
I've opened https://issues.apache.org/jira/browse/KAFKA-20191
On 2026/02/16 09:57:17 Lucas Brutschy via dev wrote:
> Hi all,
>
> @Chia: That's an excellent point. I'll amend the KIP like this:
>
> "Where, for simplicity, we can assume the assignment epoch of a
> partition that is not assigned to that member to be the broker-side
> member epoch."
>
> This should make the new validation behave like the old validation in
> this corner case. Overall, we'll get a weaker validation than the
> current validation. Adding new guard-rails to prevent committing to
> unassigned partitions or even unsubscribed topics makes sense to me,
> but this wasn't really the motivation for this KIP and the discussion
> did not focus on it. If we want this, we should probably write a
> separate KIP and have a separate discussion.
>
> @Sean: Yes, self-assigned commits will not have a member epoch so the
> validation described in the KIP will not affect this logic. I can add
> a clarification to the KIP.
>
> Cheers,
> Lucas
>
> On Mon, Feb 16, 2026 at 9:42 AM Sean Quah via dev <de...@kafka.apache.org> wrote:
> >
> > > You cannot commit partitions that are not assigned to you.
> >
> > Don't we have to continue supporting clients that assign() themselves
> > partitions manually? I don't think the broker is aware of manual
> > assignments.
> >
> > Thanks,
> > Sean
> >
> > On Mon, Feb 16, 2026 at 2:17 AM Chia-Ping Tsai <ch...@apache.org> wrote:
> >
> > > > You cannot commit partitions that are not assigned to you.
> > >
> > > Thanks for the clarification. That sounds good to me, but this new
> > > protection implies a change to the existing behavior. Therefore, it would
> > > be good to explicitly mention this in the KIP
> > >
> > > For example, the following sendOffsetsToTransaction can succeed today, but
> > > will fail after this KIP.
> > >
> > > ```java
> > > consumer.subscribe(List.of("chia"));
> > > ...
> > > producer.sendOffsetsToTransaction(Map.of(
> > > new TopicPartition("chia", 0), new OffsetAndMetadata(100),
> > > new TopicPartition("ikea", 0), new OffsetAndMetadata(100)), // not
> > > assigned partitions
> > > consumer.groupMetadata());
> > > ```
> > >
> > > Best,
> > > Chia-Ping
> > >
>
Re: [DISCUSS] KIP-1251: Assignment epochs for consumer groups
Posted by Lucas Brutschy via dev <de...@kafka.apache.org>.
Hi all,
@Chia: That's an excellent point. I'll amend the KIP like this:
"Where, for simplicity, we can assume the assignment epoch of a
partition that is not assigned to that member to be the broker-side
member epoch."
This should make the new validation behave like the old validation in
this corner case. Overall, we'll get a weaker validation than the
current validation. Adding new guard-rails to prevent committing to
unassigned partitions or even unsubscribed topics makes sense to me,
but this wasn't really the motivation for this KIP and the discussion
did not focus on it. If we want this, we should probably write a
separate KIP and have a separate discussion.
@Sean: Yes, self-assigned commits will not have a member epoch so the
validation described in the KIP will not affect this logic. I can add
a clarification to the KIP.
Cheers,
Lucas
On Mon, Feb 16, 2026 at 9:42 AM Sean Quah via dev <de...@kafka.apache.org> wrote:
>
> > You cannot commit partitions that are not assigned to you.
>
> Don't we have to continue supporting clients that assign() themselves
> partitions manually? I don't think the broker is aware of manual
> assignments.
>
> Thanks,
> Sean
>
> On Mon, Feb 16, 2026 at 2:17 AM Chia-Ping Tsai <ch...@apache.org> wrote:
>
> > > You cannot commit partitions that are not assigned to you.
> >
> > Thanks for the clarification. That sounds good to me, but this new
> > protection implies a change to the existing behavior. Therefore, it would
> > be good to explicitly mention this in the KIP
> >
> > For example, the following sendOffsetsToTransaction can succeed today, but
> > will fail after this KIP.
> >
> > ```java
> > consumer.subscribe(List.of("chia"));
> > ...
> > producer.sendOffsetsToTransaction(Map.of(
> > new TopicPartition("chia", 0), new OffsetAndMetadata(100),
> > new TopicPartition("ikea", 0), new OffsetAndMetadata(100)), // not
> > assigned partitions
> > consumer.groupMetadata());
> > ```
> >
> > Best,
> > Chia-Ping
> >
Re: [DISCUSS] KIP-1251: Assignment epochs for consumer groups
Posted by Sean Quah via dev <de...@kafka.apache.org>.
> You cannot commit partitions that are not assigned to you.
Don't we have to continue supporting clients that assign() themselves
partitions manually? I don't think the broker is aware of manual
assignments.
Thanks,
Sean
On Mon, Feb 16, 2026 at 2:17 AM Chia-Ping Tsai <ch...@apache.org> wrote:
> > You cannot commit partitions that are not assigned to you.
>
> Thanks for the clarification. That sounds good to me, but this new
> protection implies a change to the existing behavior. Therefore, it would
> be good to explicitly mention this in the KIP
>
> For example, the following sendOffsetsToTransaction can succeed today, but
> will fail after this KIP.
>
> ```java
> consumer.subscribe(List.of("chia"));
> ...
> producer.sendOffsetsToTransaction(Map.of(
> new TopicPartition("chia", 0), new OffsetAndMetadata(100),
> new TopicPartition("ikea", 0), new OffsetAndMetadata(100)), // not
> assigned partitions
> consumer.groupMetadata());
> ```
>
> Best,
> Chia-Ping
>
Re: [DISCUSS] KIP-1251: Assignment epochs for consumer groups
Posted by Chia-Ping Tsai <ch...@apache.org>.
> You cannot commit partitions that are not assigned to you.
Thanks for the clarification. That sounds good to me, but this new protection implies a change to the existing behavior. Therefore, it would be good to explicitly mention this in the KIP
For example, the following sendOffsetsToTransaction can succeed today, but will fail after this KIP.
```java
consumer.subscribe(List.of("chia"));
...
producer.sendOffsetsToTransaction(Map.of(
new TopicPartition("chia", 0), new OffsetAndMetadata(100),
new TopicPartition("ikea", 0), new OffsetAndMetadata(100)), // not assigned partitions
consumer.groupMetadata());
```
Best,
Chia-Ping
Re: [DISCUSS] KIP-1251: Assignment epochs for consumer groups
Posted by Lucas Brutschy via dev <de...@kafka.apache.org>.
Hey Chia,
this is mentioned in the KIP, albeit well hidden: "Where, for simplicity,
we can assume the assignment epoch of a partition that is not assigned to
that member to be Integer.maxValue.". Although I realize now that this
explanation suffers from an off-by-one error, so let me put it this way:
You cannot commit partitions that are not assigned to you.
Hope that clarifies it.
Cheers
Lucas
Chia-Ping Tsai <ch...@apache.org> schrieb am Sa., 14. Feb. 2026, 10:57:
> hi Lucas
>
> thanks for this KIP. I have a small question.
>
> What happens if the request brings a partition which is not hosted by the
> member id? IIRC, we didn't check it before
>
> Best,
> Chia-Ping
>
> On 2025/12/18 19:59:56 Lucas Brutschy via dev wrote:
> > Hi all,
> >
> > I posted a new KIP to add assignment epochs to KIP-848-style consumer
> > groups. The idea is to avoid failing offset commits in the new
> > consumer protocol by relaxing the member epoch validation slightly.
> >
> > Please take a look and let me know what you think:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1251%3A+Assignment+epochs+for+consumer+groups
> >
> > Cheers,
> > Lucas
> >
>
Re: [DISCUSS] KIP-1251: Assignment epochs for consumer groups
Posted by Chia-Ping Tsai <ch...@apache.org>.
hi Lucas
thanks for this KIP. I have a small question.
What happens if the request brings a partition which is not hosted by the member id? IIRC, we didn't check it before
Best,
Chia-Ping
On 2025/12/18 19:59:56 Lucas Brutschy via dev wrote:
> Hi all,
>
> I posted a new KIP to add assignment epochs to KIP-848-style consumer
> groups. The idea is to avoid failing offset commits in the new
> consumer protocol by relaxing the member epoch validation slightly.
>
> Please take a look and let me know what you think:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1251%3A+Assignment+epochs+for+consumer+groups
>
> Cheers,
> Lucas
>