You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Lucas Brutschy via dev <de...@kafka.apache.org> on 2025/12/11 09:39:39 UTC
Re: [VOTE]: KIP-1071: Streams Rebalance Protocol
Hi,
we made one more minor change for 4.2. In the heartbeat response, we
added a new status code
`ASSIGNMENT_DELAYED` to indicate that the returned assignment was empty, because
the assignment was delayed by `group.streams.initial.rebalance.delay.ms`.
Best,
Lucas
On Mon, Nov 17, 2025 at 8:51 AM Lucas Brutschy <lb...@confluent.io> wrote:
>
> Hi,
>
> while implementing KIP-1071 in 4.1 and upcoming 4.2 releases, we found
> a couple of issues which required some tweaking of the KIP, I want to
> highlight all changes. The KIP was already updated accordingly:
>
> - Streams group description in Admin API now also exposes currentAssignmentEpoch
> - In the Interactive Query endpoint information, we differentiate
> active partitions from standby partitions, as in the classic protocol,
> and we introduced an additional versioning number
> (endpointInformationEpoch) to version it.
> - StreamsGroupDescribe requires DESCRIBE ACL on groups and topics, not READ
> - Adapted the changes from KIP-1101 (described there)
> - In the StreamsGroupMetadataValue record, we added fields to store
> the last assignment configuration, and the last validated topology
> epoch
> - In StreamsGroupCurrentMemberAssignmentValue, we track additionally
> the epoch each task was assigned
> - We added the configuration group.streams.initial.rebalance.delay.ms,
> the equivalent of group.initial.rebalance.delay.ms in the classic
> protocol.
> - kafka-streams-group.sh got the additional option --all-groups as an
> alternative to --group
>
> We are marking the interfaces, records and RPCs as stable in 4.2.
>
> Please let us know if there are any concerns.
>
> Thanks,
> Lucas
>
> On Fri, Dec 13, 2024 at 11:10 AM Lucas Brutschy <lb...@confluent.io> wrote:
> >
> > Thank you all for voting!
> >
> > The KIP is now accepted with:
> >
> > 3 binding +1 (Sophie, Andrew, Bill).
> > 2 non-binding +1 (Alieh, Colt).
> > No -1 or 0 votes.
> >
> > We will target 4.1 for a non-production opt-in implementation of this KIP.
> >
> > On Tue, Dec 10, 2024 at 1:18 AM Sophie Blee-Goldman
> > <so...@responsive.dev> wrote:
> > >
> > > +1 (binding)
> > >
> > > Thank you guys for the deep discussion and for carefully considering all
> > > the details of this new protocol. Looking forward to seeing the KIP in
> > > action!
> > >
> > > -Sophie
> > >
> > > On Fri, Dec 6, 2024 at 5:52 AM Andrew Schofield <
> > > andrew_schofield_jira@outlook.com> wrote:
> > >
> > > > Hi Bruno and Lucas,
> > > > Thanks for proposing this excellent KIP.
> > > >
> > > > +1 (binding)
> > > >
> > > > Thanks,
> > > > Andrew
> > > > ________________________________________
> > > > From: Alieh Saeedi <as...@confluent.io.INVALID>
> > > > Sent: 03 December 2024 16:07
> > > > To: dev@kafka.apache.org <de...@kafka.apache.org>
> > > > Subject: Re: [VOTE]: KIP-1071: Streams Rebalance Protocol
> > > >
> > > > Thanks for the effort. I really appreciate the time and dedication you put
> > > > into this.
> > > >
> > > > +1(non-binding)
> > > >
> > > > Cheers,
> > > > Alieh
> > > >
> > > > On Wed, Nov 27, 2024 at 11:31 AM Bruno Cadonna <ca...@apache.org> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > We (Lucas and myself) would like to call for a vote on KIP-1071: Streams
> > > > > Rebalance Protocol: https://cwiki.apache.org/confluence/x/2BCTEg
> > > > >
> > > > > Discussion thread:
> > > > > https://lists.apache.org/thread/h8lfcwm1ljq5d84194jfnqp9wh148r1v
> > > > >
> > > > > Best,
> > > > > Bruno
> > > > >
> > > > >
> > > > >
> > > >
Re: [VOTE]: KIP-1071: Streams Rebalance Protocol
Posted by "Matthias J. Sax" <mj...@apache.org>.
Hello,
while continuing the work on KIP-1071, we found two errors that need to
be fixed. I updated the KIP wiki page accordingly. Please take a look
and let me know if I got anything wrong, or if there are ant concerns
about these changes:
1) Implementation error in 4.2.0 release: By mistake, the
StreamsGroupHeartbeatResponse field `acceptable-recovery-lag`, which is
defined as `int64` on the KIP, was implemented as `int32`. To fix this
problem, we need to bump the RPC version to `1`, and change the type
accordingly. -- For 4.2/4.3 releases, this is not really a problem
because the field is not yet used client side. -- To get correct
backward compatibility, we also need to bump the
StreamsGroupHeartbeatRequest version to `1`, allowing the broker to
either keep the `acceptable-recovery-lag` field unset for 4.2/4.3
clients and sending a v0 response back, or setting the correct int64
field and send a v1+ response.
2) For rack-aware standby task assignment, we need a broker side config
`rack.aware.assignment.tags` which was missing on the KIP. We are adding
it as both group-coordinator and streams-group config. The broker will
perform a validation (cf KIP-708) for each member, if all required
`client.tags` are configured. If not, the client is still allowed to
join the group, but rack-aware standby task assignment might not be able
to give the expected result. To surface this issue client side, we add a
new `Status Code` (6) `MISSING_CLIENT_TAGS` that the
`StreamsGroupHeartbeatResponse` can carry. The client is supposed to log
a WARN about this misconfiguration, allowing application developers to
react and update their configs. -- As both StreamsGroupHeartbeatRequest
and StreamsGroupHeartbeatResponse are bumped to v1 already, this new
error code with only be sent for v1+ responses, to avoid that 4.2/4.3
clients, connecting to a a 4.4+ broker don't get an error code they
don't understand.
-Matthias
On 12/11/25 1:39 AM, Lucas Brutschy via dev wrote:
> Hi,
>
> we made one more minor change for 4.2. In the heartbeat response, we
> added a new status code
> `ASSIGNMENT_DELAYED` to indicate that the returned assignment was empty, because
> the assignment was delayed by `group.streams.initial.rebalance.delay.ms`.
>
> Best,
> Lucas
>
> On Mon, Nov 17, 2025 at 8:51 AM Lucas Brutschy <lb...@confluent.io> wrote:
>>
>> Hi,
>>
>> while implementing KIP-1071 in 4.1 and upcoming 4.2 releases, we found
>> a couple of issues which required some tweaking of the KIP, I want to
>> highlight all changes. The KIP was already updated accordingly:
>>
>> - Streams group description in Admin API now also exposes currentAssignmentEpoch
>> - In the Interactive Query endpoint information, we differentiate
>> active partitions from standby partitions, as in the classic protocol,
>> and we introduced an additional versioning number
>> (endpointInformationEpoch) to version it.
>> - StreamsGroupDescribe requires DESCRIBE ACL on groups and topics, not READ
>> - Adapted the changes from KIP-1101 (described there)
>> - In the StreamsGroupMetadataValue record, we added fields to store
>> the last assignment configuration, and the last validated topology
>> epoch
>> - In StreamsGroupCurrentMemberAssignmentValue, we track additionally
>> the epoch each task was assigned
>> - We added the configuration group.streams.initial.rebalance.delay.ms,
>> the equivalent of group.initial.rebalance.delay.ms in the classic
>> protocol.
>> - kafka-streams-group.sh got the additional option --all-groups as an
>> alternative to --group
>>
>> We are marking the interfaces, records and RPCs as stable in 4.2.
>>
>> Please let us know if there are any concerns.
>>
>> Thanks,
>> Lucas
>>
>> On Fri, Dec 13, 2024 at 11:10 AM Lucas Brutschy <lb...@confluent.io> wrote:
>>>
>>> Thank you all for voting!
>>>
>>> The KIP is now accepted with:
>>>
>>> 3 binding +1 (Sophie, Andrew, Bill).
>>> 2 non-binding +1 (Alieh, Colt).
>>> No -1 or 0 votes.
>>>
>>> We will target 4.1 for a non-production opt-in implementation of this KIP.
>>>
>>> On Tue, Dec 10, 2024 at 1:18 AM Sophie Blee-Goldman
>>> <so...@responsive.dev> wrote:
>>>>
>>>> +1 (binding)
>>>>
>>>> Thank you guys for the deep discussion and for carefully considering all
>>>> the details of this new protocol. Looking forward to seeing the KIP in
>>>> action!
>>>>
>>>> -Sophie
>>>>
>>>> On Fri, Dec 6, 2024 at 5:52 AM Andrew Schofield <
>>>> andrew_schofield_jira@outlook.com> wrote:
>>>>
>>>>> Hi Bruno and Lucas,
>>>>> Thanks for proposing this excellent KIP.
>>>>>
>>>>> +1 (binding)
>>>>>
>>>>> Thanks,
>>>>> Andrew
>>>>> ________________________________________
>>>>> From: Alieh Saeedi <as...@confluent.io.INVALID>
>>>>> Sent: 03 December 2024 16:07
>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
>>>>> Subject: Re: [VOTE]: KIP-1071: Streams Rebalance Protocol
>>>>>
>>>>> Thanks for the effort. I really appreciate the time and dedication you put
>>>>> into this.
>>>>>
>>>>> +1(non-binding)
>>>>>
>>>>> Cheers,
>>>>> Alieh
>>>>>
>>>>> On Wed, Nov 27, 2024 at 11:31 AM Bruno Cadonna <ca...@apache.org> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> We (Lucas and myself) would like to call for a vote on KIP-1071: Streams
>>>>>> Rebalance Protocol: https://cwiki.apache.org/confluence/x/2BCTEg
>>>>>>
>>>>>> Discussion thread:
>>>>>> https://lists.apache.org/thread/h8lfcwm1ljq5d84194jfnqp9wh148r1v
>>>>>>
>>>>>> Best,
>>>>>> Bruno
>>>>>>
>>>>>>
>>>>>>
>>>>>