You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by jian fu <fu...@gmail.com> on 2026/04/04 07:44:29 UTC
[DISCUSS] KIP-1314: Support broker-side consumer group rebalance callback
Hi everyone,
I would like to start a discussion for this small KIP.
https://cwiki.apache.org/confluence/x/UZc8G
Rebalance is the most important consumer event, and Kafka is moving toward
server-side rebalance.
Today, client-side/sdk callbacks are useful but not always reliable
operationally (SDK dependency, Not easy for central management). In
incidents, key rebalance evidence is often missing on the client side.On
the broker side, logs and metrics are still limited for automation: logs
are hard to integrate reliably, and metrics do not include the most
actionable key (group.id).
This KIP proposes a lightweight broker-side callback to expose key
rebalance context (including group.id) for centralized observability and
automation, without forcing client upgrades and without adding
high-cardinality metric tags.
Your feedback is greatly appreciated. Thank you!
Regards
Jian
Re: [DISCUSS] KIP-1314: Support broker-side consumer group rebalance callback
Posted by jian fu <fu...@gmail.com>.
Hi Chia-Ping
Sorry for delay response！
Thank you very much for your feedback！
One concern with relying on periodic scanning is that it may miss some
rebalance event collection. In addition, a cluster can have hundreds of
thousands of consumer groups, while in practice there may be no rebalance
events in a whole day. This makes the scanning approach relatively
inefficient and potentially wasteful.
Ideally, a callback-based approach would be more suitable after all it is
weight-light. . I understand there might be concerns about callbacks
introducing blocking behavior into the system. However, this can be
mitigated with a well-designed asynchronous callback mechanism and proper
risk control by user.
Thanks again for joining this discussion.
Regards
Jian
Chia-Ping Tsai <ch...@apache.org> 于2026年4月4日周六 16:48写道：
> hi,
>
> If the goal is simply to collect the rebalance timing, using a broker-side
> plugin feels a bit overkill. Alternatively, could we just expose the
> assignmentTimestamp via the DescribeGroupsResponse? With this new field,
> users could set up a cron job or monitor to poll the timestamp
> periodically. While it might not cover 100% of real-time event tracking use
> cases, it is much simpler and poses a significantly lower risk to the
> server.
>
> Best,
> Chia-Ping
>
> On 2026/04/04 07:44:29 jian fu wrote:
> > Hi everyone,
> >
> > I would like to start a discussion for this small KIP.
> >
> > https://cwiki.apache.org/confluence/x/UZc8G
> >
> > Rebalance is the most important consumer event, and Kafka is moving
> toward
> > server-side rebalance.
> >
> > Today, client-side/sdk callbacks are useful but not always reliable
> > operationally (SDK dependency, Not easy for central management). In
> > incidents, key rebalance evidence is often missing on the client side.On
> > the broker side, logs and metrics are still limited for automation: logs
> > are hard to integrate reliably, and metrics do not include the most
> > actionable key (group.id).
> >
> > This KIP proposes a lightweight broker-side callback to expose key
> > rebalance context (including group.id) for centralized observability and
> > automation, without forcing client upgrades and without adding
> > high-cardinality metric tags.
> >
> > Your feedback is greatly appreciated. Thank you!
> >
> > Regards
> > Jian
> >
>
Re: [DISCUSS] KIP-1314: Support broker-side consumer group rebalance callback
Posted by Chia-Ping Tsai <ch...@apache.org>.
hi,
If the goal is simply to collect the rebalance timing, using a broker-side plugin feels a bit overkill. Alternatively, could we just expose the assignmentTimestamp via the DescribeGroupsResponse? With this new field, users could set up a cron job or monitor to poll the timestamp periodically. While it might not cover 100% of real-time event tracking use cases, it is much simpler and poses a significantly lower risk to the server.
Best,
Chia-Ping
On 2026/04/04 07:44:29 jian fu wrote:
> Hi everyone,
>
> I would like to start a discussion for this small KIP.
>
> https://cwiki.apache.org/confluence/x/UZc8G
>
> Rebalance is the most important consumer event, and Kafka is moving toward
> server-side rebalance.
>
> Today, client-side/sdk callbacks are useful but not always reliable
> operationally (SDK dependency, Not easy for central management). In
> incidents, key rebalance evidence is often missing on the client side.On
> the broker side, logs and metrics are still limited for automation: logs
> are hard to integrate reliably, and metrics do not include the most
> actionable key (group.id).
>
> This KIP proposes a lightweight broker-side callback to expose key
> rebalance context (including group.id) for centralized observability and
> automation, without forcing client upgrades and without adding
> high-cardinality metric tags.
>
> Your feedback is greatly appreciated. Thank you!
>
> Regards
> Jian
>