You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Alieh Saeedi <as...@confluent.io.INVALID> on 2024/12/03 16:07:11 UTC
Re: [VOTE]: KIP-1071: Streams Rebalance Protocol
Thanks for the effort. I really appreciate the time and dedication you put
into this.
+1(non-binding)
Cheers,
Alieh
On Wed, Nov 27, 2024 at 11:31 AM Bruno Cadonna <ca...@apache.org> wrote:
> Hi all,
>
> We (Lucas and myself) would like to call for a vote on KIP-1071: Streams
> Rebalance Protocol: https://cwiki.apache.org/confluence/x/2BCTEg
>
> Discussion thread:
> https://lists.apache.org/thread/h8lfcwm1ljq5d84194jfnqp9wh148r1v
>
> Best,
> Bruno
>
>
>
Re: [VOTE]: KIP-1071: Streams Rebalance Protocol
Posted by "Matthias J. Sax" <mj...@apache.org>.
Hello,
while continuing the work on KIP-1071, we found two errors that need to
be fixed. I updated the KIP wiki page accordingly. Please take a look
and let me know if I got anything wrong, or if there are ant concerns
about these changes:
1) Implementation error in 4.2.0 release: By mistake, the
StreamsGroupHeartbeatResponse field `acceptable-recovery-lag`, which is
defined as `int64` on the KIP, was implemented as `int32`. To fix this
problem, we need to bump the RPC version to `1`, and change the type
accordingly. -- For 4.2/4.3 releases, this is not really a problem
because the field is not yet used client side. -- To get correct
backward compatibility, we also need to bump the
StreamsGroupHeartbeatRequest version to `1`, allowing the broker to
either keep the `acceptable-recovery-lag` field unset for 4.2/4.3
clients and sending a v0 response back, or setting the correct int64
field and send a v1+ response.
2) For rack-aware standby task assignment, we need a broker side config
`rack.aware.assignment.tags` which was missing on the KIP. We are adding
it as both group-coordinator and streams-group config. The broker will
perform a validation (cf KIP-708) for each member, if all required
`client.tags` are configured. If not, the client is still allowed to
join the group, but rack-aware standby task assignment might not be able
to give the expected result. To surface this issue client side, we add a
new `Status Code` (6) `MISSING_CLIENT_TAGS` that the
`StreamsGroupHeartbeatResponse` can carry. The client is supposed to log
a WARN about this misconfiguration, allowing application developers to
react and update their configs. -- As both StreamsGroupHeartbeatRequest
and StreamsGroupHeartbeatResponse are bumped to v1 already, this new
error code with only be sent for v1+ responses, to avoid that 4.2/4.3
clients, connecting to a a 4.4+ broker don't get an error code they
don't understand.
-Matthias
On 12/11/25 1:39 AM, Lucas Brutschy via dev wrote:
> Hi,
>
> we made one more minor change for 4.2. In the heartbeat response, we
> added a new status code
> `ASSIGNMENT_DELAYED` to indicate that the returned assignment was empty, because
> the assignment was delayed by `group.streams.initial.rebalance.delay.ms`.
>
> Best,
> Lucas
>
> On Mon, Nov 17, 2025 at 8:51 AM Lucas Brutschy <lb...@confluent.io> wrote:
>>
>> Hi,
>>
>> while implementing KIP-1071 in 4.1 and upcoming 4.2 releases, we found
>> a couple of issues which required some tweaking of the KIP, I want to
>> highlight all changes. The KIP was already updated accordingly:
>>
>> - Streams group description in Admin API now also exposes currentAssignmentEpoch
>> - In the Interactive Query endpoint information, we differentiate
>> active partitions from standby partitions, as in the classic protocol,
>> and we introduced an additional versioning number
>> (endpointInformationEpoch) to version it.
>> - StreamsGroupDescribe requires DESCRIBE ACL on groups and topics, not READ
>> - Adapted the changes from KIP-1101 (described there)
>> - In the StreamsGroupMetadataValue record, we added fields to store
>> the last assignment configuration, and the last validated topology
>> epoch
>> - In StreamsGroupCurrentMemberAssignmentValue, we track additionally
>> the epoch each task was assigned
>> - We added the configuration group.streams.initial.rebalance.delay.ms,
>> the equivalent of group.initial.rebalance.delay.ms in the classic
>> protocol.
>> - kafka-streams-group.sh got the additional option --all-groups as an
>> alternative to --group
>>
>> We are marking the interfaces, records and RPCs as stable in 4.2.
>>
>> Please let us know if there are any concerns.
>>
>> Thanks,
>> Lucas
>>
>> On Fri, Dec 13, 2024 at 11:10 AM Lucas Brutschy <lb...@confluent.io> wrote:
>>>
>>> Thank you all for voting!
>>>
>>> The KIP is now accepted with:
>>>
>>> 3 binding +1 (Sophie, Andrew, Bill).
>>> 2 non-binding +1 (Alieh, Colt).
>>> No -1 or 0 votes.
>>>
>>> We will target 4.1 for a non-production opt-in implementation of this KIP.
>>>
>>> On Tue, Dec 10, 2024 at 1:18 AM Sophie Blee-Goldman
>>> <so...@responsive.dev> wrote:
>>>>
>>>> +1 (binding)
>>>>
>>>> Thank you guys for the deep discussion and for carefully considering all
>>>> the details of this new protocol. Looking forward to seeing the KIP in
>>>> action!
>>>>
>>>> -Sophie
>>>>
>>>> On Fri, Dec 6, 2024 at 5:52 AM Andrew Schofield <
>>>> andrew_schofield_jira@outlook.com> wrote:
>>>>
>>>>> Hi Bruno and Lucas,
>>>>> Thanks for proposing this excellent KIP.
>>>>>
>>>>> +1 (binding)
>>>>>
>>>>> Thanks,
>>>>> Andrew
>>>>> ________________________________________
>>>>> From: Alieh Saeedi <as...@confluent.io.INVALID>
>>>>> Sent: 03 December 2024 16:07
>>>>> To: dev@kafka.apache.org <de...@kafka.apache.org>
>>>>> Subject: Re: [VOTE]: KIP-1071: Streams Rebalance Protocol
>>>>>
>>>>> Thanks for the effort. I really appreciate the time and dedication you put
>>>>> into this.
>>>>>
>>>>> +1(non-binding)
>>>>>
>>>>> Cheers,
>>>>> Alieh
>>>>>
>>>>> On Wed, Nov 27, 2024 at 11:31 AM Bruno Cadonna <ca...@apache.org> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> We (Lucas and myself) would like to call for a vote on KIP-1071: Streams
>>>>>> Rebalance Protocol: https://cwiki.apache.org/confluence/x/2BCTEg
>>>>>>
>>>>>> Discussion thread:
>>>>>> https://lists.apache.org/thread/h8lfcwm1ljq5d84194jfnqp9wh148r1v
>>>>>>
>>>>>> Best,
>>>>>> Bruno
>>>>>>
>>>>>>
>>>>>>
>>>>>
Re: [VOTE]: KIP-1071: Streams Rebalance Protocol
Posted by Lucas Brutschy via dev <de...@kafka.apache.org>.
Hi,
we made one more minor change for 4.2. In the heartbeat response, we
added a new status code
`ASSIGNMENT_DELAYED` to indicate that the returned assignment was empty, because
the assignment was delayed by `group.streams.initial.rebalance.delay.ms`.
Best,
Lucas
On Mon, Nov 17, 2025 at 8:51 AM Lucas Brutschy <lb...@confluent.io> wrote:
>
> Hi,
>
> while implementing KIP-1071 in 4.1 and upcoming 4.2 releases, we found
> a couple of issues which required some tweaking of the KIP, I want to
> highlight all changes. The KIP was already updated accordingly:
>
> - Streams group description in Admin API now also exposes currentAssignmentEpoch
> - In the Interactive Query endpoint information, we differentiate
> active partitions from standby partitions, as in the classic protocol,
> and we introduced an additional versioning number
> (endpointInformationEpoch) to version it.
> - StreamsGroupDescribe requires DESCRIBE ACL on groups and topics, not READ
> - Adapted the changes from KIP-1101 (described there)
> - In the StreamsGroupMetadataValue record, we added fields to store
> the last assignment configuration, and the last validated topology
> epoch
> - In StreamsGroupCurrentMemberAssignmentValue, we track additionally
> the epoch each task was assigned
> - We added the configuration group.streams.initial.rebalance.delay.ms,
> the equivalent of group.initial.rebalance.delay.ms in the classic
> protocol.
> - kafka-streams-group.sh got the additional option --all-groups as an
> alternative to --group
>
> We are marking the interfaces, records and RPCs as stable in 4.2.
>
> Please let us know if there are any concerns.
>
> Thanks,
> Lucas
>
> On Fri, Dec 13, 2024 at 11:10 AM Lucas Brutschy <lb...@confluent.io> wrote:
> >
> > Thank you all for voting!
> >
> > The KIP is now accepted with:
> >
> > 3 binding +1 (Sophie, Andrew, Bill).
> > 2 non-binding +1 (Alieh, Colt).
> > No -1 or 0 votes.
> >
> > We will target 4.1 for a non-production opt-in implementation of this KIP.
> >
> > On Tue, Dec 10, 2024 at 1:18 AM Sophie Blee-Goldman
> > <so...@responsive.dev> wrote:
> > >
> > > +1 (binding)
> > >
> > > Thank you guys for the deep discussion and for carefully considering all
> > > the details of this new protocol. Looking forward to seeing the KIP in
> > > action!
> > >
> > > -Sophie
> > >
> > > On Fri, Dec 6, 2024 at 5:52 AM Andrew Schofield <
> > > andrew_schofield_jira@outlook.com> wrote:
> > >
> > > > Hi Bruno and Lucas,
> > > > Thanks for proposing this excellent KIP.
> > > >
> > > > +1 (binding)
> > > >
> > > > Thanks,
> > > > Andrew
> > > > ________________________________________
> > > > From: Alieh Saeedi <as...@confluent.io.INVALID>
> > > > Sent: 03 December 2024 16:07
> > > > To: dev@kafka.apache.org <de...@kafka.apache.org>
> > > > Subject: Re: [VOTE]: KIP-1071: Streams Rebalance Protocol
> > > >
> > > > Thanks for the effort. I really appreciate the time and dedication you put
> > > > into this.
> > > >
> > > > +1(non-binding)
> > > >
> > > > Cheers,
> > > > Alieh
> > > >
> > > > On Wed, Nov 27, 2024 at 11:31 AM Bruno Cadonna <ca...@apache.org> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > We (Lucas and myself) would like to call for a vote on KIP-1071: Streams
> > > > > Rebalance Protocol: https://cwiki.apache.org/confluence/x/2BCTEg
> > > > >
> > > > > Discussion thread:
> > > > > https://lists.apache.org/thread/h8lfcwm1ljq5d84194jfnqp9wh148r1v
> > > > >
> > > > > Best,
> > > > > Bruno
> > > > >
> > > > >
> > > > >
> > > >
Re: [VOTE]: KIP-1071: Streams Rebalance Protocol
Posted by Lucas Brutschy <lb...@confluent.io.INVALID>.
Hi,
while implementing KIP-1071 in 4.1 and upcoming 4.2 releases, we found
a couple of issues which required some tweaking of the KIP, I want to
highlight all changes. The KIP was already updated accordingly:
- Streams group description in Admin API now also exposes currentAssignmentEpoch
- In the Interactive Query endpoint information, we differentiate
active partitions from standby partitions, as in the classic protocol,
and we introduced an additional versioning number
(endpointInformationEpoch) to version it.
- StreamsGroupDescribe requires DESCRIBE ACL on groups and topics, not READ
- Adapted the changes from KIP-1101 (described there)
- In the StreamsGroupMetadataValue record, we added fields to store
the last assignment configuration, and the last validated topology
epoch
- In StreamsGroupCurrentMemberAssignmentValue, we track additionally
the epoch each task was assigned
- We added the configuration group.streams.initial.rebalance.delay.ms,
the equivalent of group.initial.rebalance.delay.ms in the classic
protocol.
- kafka-streams-group.sh got the additional option --all-groups as an
alternative to --group
We are marking the interfaces, records and RPCs as stable in 4.2.
Please let us know if there are any concerns.
Thanks,
Lucas
On Fri, Dec 13, 2024 at 11:10 AM Lucas Brutschy <lb...@confluent.io> wrote:
>
> Thank you all for voting!
>
> The KIP is now accepted with:
>
> 3 binding +1 (Sophie, Andrew, Bill).
> 2 non-binding +1 (Alieh, Colt).
> No -1 or 0 votes.
>
> We will target 4.1 for a non-production opt-in implementation of this KIP.
>
> On Tue, Dec 10, 2024 at 1:18 AM Sophie Blee-Goldman
> <so...@responsive.dev> wrote:
> >
> > +1 (binding)
> >
> > Thank you guys for the deep discussion and for carefully considering all
> > the details of this new protocol. Looking forward to seeing the KIP in
> > action!
> >
> > -Sophie
> >
> > On Fri, Dec 6, 2024 at 5:52 AM Andrew Schofield <
> > andrew_schofield_jira@outlook.com> wrote:
> >
> > > Hi Bruno and Lucas,
> > > Thanks for proposing this excellent KIP.
> > >
> > > +1 (binding)
> > >
> > > Thanks,
> > > Andrew
> > > ________________________________________
> > > From: Alieh Saeedi <as...@confluent.io.INVALID>
> > > Sent: 03 December 2024 16:07
> > > To: dev@kafka.apache.org <de...@kafka.apache.org>
> > > Subject: Re: [VOTE]: KIP-1071: Streams Rebalance Protocol
> > >
> > > Thanks for the effort. I really appreciate the time and dedication you put
> > > into this.
> > >
> > > +1(non-binding)
> > >
> > > Cheers,
> > > Alieh
> > >
> > > On Wed, Nov 27, 2024 at 11:31 AM Bruno Cadonna <ca...@apache.org> wrote:
> > >
> > > > Hi all,
> > > >
> > > > We (Lucas and myself) would like to call for a vote on KIP-1071: Streams
> > > > Rebalance Protocol: https://cwiki.apache.org/confluence/x/2BCTEg
> > > >
> > > > Discussion thread:
> > > > https://lists.apache.org/thread/h8lfcwm1ljq5d84194jfnqp9wh148r1v
> > > >
> > > > Best,
> > > > Bruno
> > > >
> > > >
> > > >
> > >
Re: [VOTE]: KIP-1071: Streams Rebalance Protocol
Posted by Lucas Brutschy <lb...@confluent.io.INVALID>.
Thank you all for voting!
The KIP is now accepted with:
3 binding +1 (Sophie, Andrew, Bill).
2 non-binding +1 (Alieh, Colt).
No -1 or 0 votes.
We will target 4.1 for a non-production opt-in implementation of this KIP.
On Tue, Dec 10, 2024 at 1:18 AM Sophie Blee-Goldman
<so...@responsive.dev> wrote:
>
> +1 (binding)
>
> Thank you guys for the deep discussion and for carefully considering all
> the details of this new protocol. Looking forward to seeing the KIP in
> action!
>
> -Sophie
>
> On Fri, Dec 6, 2024 at 5:52 AM Andrew Schofield <
> andrew_schofield_jira@outlook.com> wrote:
>
> > Hi Bruno and Lucas,
> > Thanks for proposing this excellent KIP.
> >
> > +1 (binding)
> >
> > Thanks,
> > Andrew
> > ________________________________________
> > From: Alieh Saeedi <as...@confluent.io.INVALID>
> > Sent: 03 December 2024 16:07
> > To: dev@kafka.apache.org <de...@kafka.apache.org>
> > Subject: Re: [VOTE]: KIP-1071: Streams Rebalance Protocol
> >
> > Thanks for the effort. I really appreciate the time and dedication you put
> > into this.
> >
> > +1(non-binding)
> >
> > Cheers,
> > Alieh
> >
> > On Wed, Nov 27, 2024 at 11:31 AM Bruno Cadonna <ca...@apache.org> wrote:
> >
> > > Hi all,
> > >
> > > We (Lucas and myself) would like to call for a vote on KIP-1071: Streams
> > > Rebalance Protocol: https://cwiki.apache.org/confluence/x/2BCTEg
> > >
> > > Discussion thread:
> > > https://lists.apache.org/thread/h8lfcwm1ljq5d84194jfnqp9wh148r1v
> > >
> > > Best,
> > > Bruno
> > >
> > >
> > >
> >
Re: [VOTE]: KIP-1071: Streams Rebalance Protocol
Posted by Sophie Blee-Goldman <so...@responsive.dev>.
+1 (binding)
Thank you guys for the deep discussion and for carefully considering all
the details of this new protocol. Looking forward to seeing the KIP in
action!
-Sophie
On Fri, Dec 6, 2024 at 5:52 AM Andrew Schofield <
andrew_schofield_jira@outlook.com> wrote:
> Hi Bruno and Lucas,
> Thanks for proposing this excellent KIP.
>
> +1 (binding)
>
> Thanks,
> Andrew
> ________________________________________
> From: Alieh Saeedi <as...@confluent.io.INVALID>
> Sent: 03 December 2024 16:07
> To: dev@kafka.apache.org <de...@kafka.apache.org>
> Subject: Re: [VOTE]: KIP-1071: Streams Rebalance Protocol
>
> Thanks for the effort. I really appreciate the time and dedication you put
> into this.
>
> +1(non-binding)
>
> Cheers,
> Alieh
>
> On Wed, Nov 27, 2024 at 11:31 AM Bruno Cadonna <ca...@apache.org> wrote:
>
> > Hi all,
> >
> > We (Lucas and myself) would like to call for a vote on KIP-1071: Streams
> > Rebalance Protocol: https://cwiki.apache.org/confluence/x/2BCTEg
> >
> > Discussion thread:
> > https://lists.apache.org/thread/h8lfcwm1ljq5d84194jfnqp9wh148r1v
> >
> > Best,
> > Bruno
> >
> >
> >
>
Re: [VOTE]: KIP-1071: Streams Rebalance Protocol
Posted by Andrew Schofield <an...@outlook.com>.
Hi Bruno and Lucas,
Thanks for proposing this excellent KIP.
+1 (binding)
Thanks,
Andrew
________________________________________
From: Alieh Saeedi <as...@confluent.io.INVALID>
Sent: 03 December 2024 16:07
To: dev@kafka.apache.org <de...@kafka.apache.org>
Subject: Re: [VOTE]: KIP-1071: Streams Rebalance Protocol
Thanks for the effort. I really appreciate the time and dedication you put
into this.
+1(non-binding)
Cheers,
Alieh
On Wed, Nov 27, 2024 at 11:31 AM Bruno Cadonna <ca...@apache.org> wrote:
> Hi all,
>
> We (Lucas and myself) would like to call for a vote on KIP-1071: Streams
> Rebalance Protocol: https://cwiki.apache.org/confluence/x/2BCTEg
>
> Discussion thread:
> https://lists.apache.org/thread/h8lfcwm1ljq5d84194jfnqp9wh148r1v
>
> Best,
> Bruno
>
>
>