[DISCUSS] KIP-1263: Group Coordinator Assignment Batching and Offload

[Sean Quah via dev <de...@kafka.apache.org>]

Hi all,

I would like to start a discussion for KIP-1263. Under the new consumer,
share and streams protocols (KIP-848, KIP-932, KIP-1071) we calculate
assignments on the broker whereas previously we used to calculate
assignments on clients under the old classic protocol. The assignment
calculation can get expensive for large groups and impact group coordinator
performance. The KIP aims to reduce the impact of slow assignor runs
through two methods: enforcing a minimum interval between assignment
updates and moving the assignment calculation to a separate thread pool.

KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1263%3A+Group+Coordinator+Assignment+Batching+and+Offload

Best regards,
Sean

Re: [DISCUSS] KIP-1263: Group Coordinator Assignment Batching and Offload

[Sean Quah via dev <de...@kafka.apache.org>]

Hi all,
During the review of https://github.com/apache/kafka/pull/21555, we have
opted to rename the executor configs and metrics.

*Configs*

   - group.coordinator.executor.threads ->
   group.coordinator.background.threads

*Metrics*

   - executor-thread-idle-ratio-avg -> background-thread-idle-ratio-avg
   - executor-queue-time-ms-(p50|p95|p99|p999|max)
   -> background-queue-time-ms-(p50|p95|p99|p999|max)
   - executor-processing-time-ms-(p50|p95|p99|p999|max)
   -> background-processing-time-ms-(p50|p95|p99|p999|max)

Thanks,
Sean

On Fri, Feb 13, 2026 at 8:00 AM David Jacot via dev <de...@kafka.apache.org>
wrote:

> Thanks for the update, Sean. The KIP looks good to me.
>
> Best,
> David
>
> On Fri, Feb 13, 2026 at 8:27 AM Sean Quah via dev <de...@kafka.apache.org>
> wrote:
>
> > Thanks for the feedback.
> >
> > dj08: I wonder whether we should use 1s for the interval combined with
> > > offloading enabled by default. The offloading preserves the coordinator
> > > threads while providing back-pressure because a group can have only one
> > > assignment inflight. Going from 5s to 1s would ensure that groups
> remain
> > > more dynamic.
> >
> > I've updated the default to 1s.
> >
> > dj09: Should we have min/max configs on the server for the new interval,
> > > similar to other intervals?
> >
> > I've added those too. I've renamed min.assignment.interval.ms to
> > assignment.interval.ms, otherwise the names look weird.
> >
> > dj10: What's the reason for using -1 by default for the Timestamp field?
> It
> >
> > seems we can just use zero.
> >
> > I wanted a value to indicate "no timestamp". 0 works just as well as -1.
> > I've updated the KIP.
> >
> >
> > > dj11: I wonder whether we should have a plan to sunset
> > > the *.assignor.offload.enable=false too. I believe that this will hurt
> > > users more than anything else. We could perhaps mention this in a
> future
> > > work section.
> >
> > Updated the Compatibility, Deprecation, and Migration Plan to mention
> this.
> >
> >  dj12: In the migration section, we should also mention the offloading
> >
> > enabled by default.
> >
> > Updated the Compatibility, Deprecation, and Migration Plan section
> >
> > Thanks,
> > Sean
> >
> > On Thu, Feb 12, 2026 at 1:08 PM David Jacot via dev <
> dev@kafka.apache.org>
> > wrote:
> >
> > > Hi Sean,
> > >
> > > Thanks for the update. I have a few more comments to consider:
> > >
> > > dj08: I wonder whether we should use 1s for the interval combined with
> > > offloading enabled by default. The offloading preserves the coordinator
> > > threads while providing back-pressure because a group can have only one
> > > assignment inflight. Going from 5s to 1s would ensure that groups
> remain
> > > more dynamic.
> > > dj09: Should we have min/max configs on the server for the new
> interval,
> > > similar to other intervals?
> > > dj10: What's the reason for using -1 by default for the Timestamp
> field?
> > It
> > > seems we can just use zero.
> > > dj11: I wonder whether we should have a plan to sunset
> > > the *.assignor.offload.enable=false too. I believe that this will hurt
> > > users more than anything else. We could perhaps mention this in a
> future
> > > work section.
> > > dj12: In the migration section, we should also mention the offloading
> > > enabled by default.
> > >
> > > Best,
> > > David
> > >
> > > On Tue, Feb 10, 2026 at 10:14 PM Sean Quah via dev <
> dev@kafka.apache.org
> > >
> > > wrote:
> > >
> > > > Thanks all for the feedback so far. I will start a vote within the
> next
> > > few
> > > > days unless anyone raises further feedback.
> > > >
> > > > Thanks,
> > > > Sean
> > > >
> > > > On Mon, Jan 26, 2026 at 10:42 PM Dongnuo Lyu via dev <
> > > dev@kafka.apache.org
> > > > >
> > > > wrote:
> > > >
> > > > > Thanks Sean for the update!
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1263: Group Coordinator Assignment Batching and Offload

[David Jacot via dev <de...@kafka.apache.org>]

Thanks for the update, Sean. The KIP looks good to me.

Best,
David

On Fri, Feb 13, 2026 at 8:27 AM Sean Quah via dev <de...@kafka.apache.org>
wrote:

> Thanks for the feedback.
>
> dj08: I wonder whether we should use 1s for the interval combined with
> > offloading enabled by default. The offloading preserves the coordinator
> > threads while providing back-pressure because a group can have only one
> > assignment inflight. Going from 5s to 1s would ensure that groups remain
> > more dynamic.
>
> I've updated the default to 1s.
>
> dj09: Should we have min/max configs on the server for the new interval,
> > similar to other intervals?
>
> I've added those too. I've renamed min.assignment.interval.ms to
> assignment.interval.ms, otherwise the names look weird.
>
> dj10: What's the reason for using -1 by default for the Timestamp field? It
>
> seems we can just use zero.
>
> I wanted a value to indicate "no timestamp". 0 works just as well as -1.
> I've updated the KIP.
>
>
> > dj11: I wonder whether we should have a plan to sunset
> > the *.assignor.offload.enable=false too. I believe that this will hurt
> > users more than anything else. We could perhaps mention this in a future
> > work section.
>
> Updated the Compatibility, Deprecation, and Migration Plan to mention this.
>
>  dj12: In the migration section, we should also mention the offloading
>
> enabled by default.
>
> Updated the Compatibility, Deprecation, and Migration Plan section
>
> Thanks,
> Sean
>
> On Thu, Feb 12, 2026 at 1:08 PM David Jacot via dev <de...@kafka.apache.org>
> wrote:
>
> > Hi Sean,
> >
> > Thanks for the update. I have a few more comments to consider:
> >
> > dj08: I wonder whether we should use 1s for the interval combined with
> > offloading enabled by default. The offloading preserves the coordinator
> > threads while providing back-pressure because a group can have only one
> > assignment inflight. Going from 5s to 1s would ensure that groups remain
> > more dynamic.
> > dj09: Should we have min/max configs on the server for the new interval,
> > similar to other intervals?
> > dj10: What's the reason for using -1 by default for the Timestamp field?
> It
> > seems we can just use zero.
> > dj11: I wonder whether we should have a plan to sunset
> > the *.assignor.offload.enable=false too. I believe that this will hurt
> > users more than anything else. We could perhaps mention this in a future
> > work section.
> > dj12: In the migration section, we should also mention the offloading
> > enabled by default.
> >
> > Best,
> > David
> >
> > On Tue, Feb 10, 2026 at 10:14 PM Sean Quah via dev <dev@kafka.apache.org
> >
> > wrote:
> >
> > > Thanks all for the feedback so far. I will start a vote within the next
> > few
> > > days unless anyone raises further feedback.
> > >
> > > Thanks,
> > > Sean
> > >
> > > On Mon, Jan 26, 2026 at 10:42 PM Dongnuo Lyu via dev <
> > dev@kafka.apache.org
> > > >
> > > wrote:
> > >
> > > > Thanks Sean for the update!
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1263: Group Coordinator Assignment Batching and Offload

[Sean Quah via dev <de...@kafka.apache.org>]

Thanks for the feedback.

dj08: I wonder whether we should use 1s for the interval combined with
> offloading enabled by default. The offloading preserves the coordinator
> threads while providing back-pressure because a group can have only one
> assignment inflight. Going from 5s to 1s would ensure that groups remain
> more dynamic.

I've updated the default to 1s.

dj09: Should we have min/max configs on the server for the new interval,
> similar to other intervals?

I've added those too. I've renamed min.assignment.interval.ms to
assignment.interval.ms, otherwise the names look weird.

dj10: What's the reason for using -1 by default for the Timestamp field? It

seems we can just use zero.

I wanted a value to indicate "no timestamp". 0 works just as well as -1.
I've updated the KIP.


> dj11: I wonder whether we should have a plan to sunset
> the *.assignor.offload.enable=false too. I believe that this will hurt
> users more than anything else. We could perhaps mention this in a future
> work section.

Updated the Compatibility, Deprecation, and Migration Plan to mention this.

 dj12: In the migration section, we should also mention the offloading

enabled by default.

Updated the Compatibility, Deprecation, and Migration Plan section

Thanks,
Sean

On Thu, Feb 12, 2026 at 1:08 PM David Jacot via dev <de...@kafka.apache.org>
wrote:

> Hi Sean,
>
> Thanks for the update. I have a few more comments to consider:
>
> dj08: I wonder whether we should use 1s for the interval combined with
> offloading enabled by default. The offloading preserves the coordinator
> threads while providing back-pressure because a group can have only one
> assignment inflight. Going from 5s to 1s would ensure that groups remain
> more dynamic.
> dj09: Should we have min/max configs on the server for the new interval,
> similar to other intervals?
> dj10: What's the reason for using -1 by default for the Timestamp field? It
> seems we can just use zero.
> dj11: I wonder whether we should have a plan to sunset
> the *.assignor.offload.enable=false too. I believe that this will hurt
> users more than anything else. We could perhaps mention this in a future
> work section.
> dj12: In the migration section, we should also mention the offloading
> enabled by default.
>
> Best,
> David
>
> On Tue, Feb 10, 2026 at 10:14 PM Sean Quah via dev <de...@kafka.apache.org>
> wrote:
>
> > Thanks all for the feedback so far. I will start a vote within the next
> few
> > days unless anyone raises further feedback.
> >
> > Thanks,
> > Sean
> >
> > On Mon, Jan 26, 2026 at 10:42 PM Dongnuo Lyu via dev <
> dev@kafka.apache.org
> > >
> > wrote:
> >
> > > Thanks Sean for the update!
> > >
> >
>

Re: [DISCUSS] KIP-1263: Group Coordinator Assignment Batching and Offload

[David Jacot via dev <de...@kafka.apache.org>]

Hi Sean,

Thanks for the update. I have a few more comments to consider:

dj08: I wonder whether we should use 1s for the interval combined with
offloading enabled by default. The offloading preserves the coordinator
threads while providing back-pressure because a group can have only one
assignment inflight. Going from 5s to 1s would ensure that groups remain
more dynamic.
dj09: Should we have min/max configs on the server for the new interval,
similar to other intervals?
dj10: What's the reason for using -1 by default for the Timestamp field? It
seems we can just use zero.
dj11: I wonder whether we should have a plan to sunset
the *.assignor.offload.enable=false too. I believe that this will hurt
users more than anything else. We could perhaps mention this in a future
work section.
dj12: In the migration section, we should also mention the offloading
enabled by default.

Best,
David

On Tue, Feb 10, 2026 at 10:14 PM Sean Quah via dev <de...@kafka.apache.org>
wrote:

> Thanks all for the feedback so far. I will start a vote within the next few
> days unless anyone raises further feedback.
>
> Thanks,
> Sean
>
> On Mon, Jan 26, 2026 at 10:42 PM Dongnuo Lyu via dev <dev@kafka.apache.org
> >
> wrote:
>
> > Thanks Sean for the update!
> >
>

Re: [DISCUSS] KIP-1263: Group Coordinator Assignment Batching and Offload

[Sean Quah via dev <de...@kafka.apache.org>]

Thanks all for the feedback so far. I will start a vote within the next few
days unless anyone raises further feedback.

Thanks,
Sean

On Mon, Jan 26, 2026 at 10:42 PM Dongnuo Lyu via dev <de...@kafka.apache.org>
wrote:

> Thanks Sean for the update!
>

Re: [DISCUSS] KIP-1263: Group Coordinator Assignment Batching and Offload

[Dongnuo Lyu via dev <de...@kafka.apache.org>]

Thanks Sean for the update!

Re: [DISCUSS] KIP-1263: Group Coordinator Assignment Batching and Offload

[Lianet Magrans <li...@apache.org>]

Thanks for the answers and updates Sean!

On Mon, Jan 26, 2026 at 7:43 AM Lucas Brutschy via dev <de...@kafka.apache.org>
wrote:

> Hi Sean,
>
> that makes a lot of sense, thanks for the explanation!
>
> Cheers,
> Lucas
>
> On Sat, Jan 24, 2026 at 11:17 AM Sean Quah via dev <de...@kafka.apache.org>
> wrote:
> >
> > Hi Lucas,
> >
> > LB01: I'm just wondering if it would have been an option to instead
> > > update the target assignment to remove the partitions from the member
> > > immediately when the member unsubscribes?
> >
> > Good question. I can't recall my thought process when implementing
> > KAFKA-19431. Three points spring to mind with updating the target
> > assignment on subscription changes:
> > 1. I think I wanted to preserve the property that the target assignment
> is
> > only updated by assignment runs and immutable for a given epoch. Though
> it
> > turns out that's not actually the case since we do patch the target
> > assignment when members leave or static members are replaced.
> > 2. An offloaded assignment based on older subscriptions can complete
> right
> > after we patch the target assignment to remove unsubscribed topics so we
> > would need to do some extra filtering on assignor completion.
> > 3. Even if we update the target assignment, we would need to touch the
> > reconciliation process anyway, since it wouldn't do anything when there
> is
> > no epoch bump.
> >
> > There is certainly nothing stopping us from updating the target
> assignment.
> > I think it seemed cleaner at the time to keep it all in the
> reconciliation
> > process.
> >
> > Thanks,
> > Sean
> >
> > On Fri, Jan 23, 2026 at 12:32 PM Lucas Brutschy <lb...@confluent.io>
> > wrote:
> >
> > > Hey Sean,
> > >
> > > thanks for the KIP! This makes a lot of sense to me. I don't really
> > > have anything I want you to change about the KIP.
> > >
> > > > We modify reconciliation to revoke any partitions the member is no
> > > longer subscribed to, since the target assignment may lag behind member
> > > subscriptions.
> > >
> > > LB01: I'm just wondering if it would have been an option to instead
> > > update the target assignment to remove the partitions from the member
> > > immediately when the member unsubscribes?
> > >
> > > Cheers,
> > > Lucas
> > >
> > > On Thu, Jan 22, 2026 at 11:58 AM Sean Quah via dev <
> dev@kafka.apache.org>
> > > wrote:
> > > >
> > > > >
> > > > > LM1: About group.initial.rebalance.delay.ms, I expect the
> interaction
> > > > > with the interval is just as described for the streams initial
> delay
> > > and
> > > > > interval, correct? Should we clarify that in the KIP (it only
> mentions
> > > the
> > > > > streams case)
> > > >
> > > > We haven't added a consumer or share group
> initial.rebalance.delay.ms
> > > > config yet. It only exists for streams right now.
> > > >
> > > > LM2: The KIP refers to batching assignment re-calculations triggered
> by
> > > > > member subscriptions changes, but I expect the batching mechanism
> > > applies
> > > > > the same when the assignment re-calculation is triggered by
> metadata
> > > > > changes (i.e topic/partition created or deleted), without any HB
> > > changing
> > > > > subscriptions. Is my understanding correct?
> > > >
> > > > Yes, that's right. Topic metadata changes also bump the group epoch
> and
> > > > triggers the same assignment flow.
> > > >
> > > > LM3: About this section: "When there is an in-flight assignor run
> for the
> > > > > group, there is no new target assignment. We will trigger the next
> > > assignor
> > > > > run on a future heartbeat.". I expect that the next assignor run
> will
> > > be
> > > > > triggered on the next HB from this or from any other member of the
> > > group,
> > > > > received after the interval expires (without the members
> re-sending the
> > > > > subscription change). Is my expectation correct? If so, it may be
> worth
> > > > > clarifying in the KIP to avoid confusion with client-side
> > > implementations.
> > > >
> > > > I tried to clarify in the KIP. Let me know your thoughts!
> > > >
> > > > On Thu, Jan 22, 2026 at 10:56 AM Sean Quah <sq...@confluent.io>
> wrote:
> > > >
> > > > > dl01: Could we mention the handling when the group metadata or
> > > > >> topic partition metadata is changed or deleted during the async
> > > assignor
> > > > >> run?
> > > > >
> > > > > Thanks! I've added a paragraph to the Assignment Offload section
> > > > > describing the handling of group metadata changes. Topic metadata
> > > changes
> > > > > already bump the group epoch and we don't need to handle them
> > > specially.
> > > > >
> > > > > dl02: This might be a question for the overall coordinator
> executor -
> > > do
> > > > >> we have plans to apply an explicit size limit to the executor
> queue?
> > > If
> > > > >> many groups trigger offloaded assignments simultaneously, should
> we
> > > apply
> > > > >> some backpressure for protection?
> > > > >
> > > > > There aren't any plans for that right now. We actually don't have a
> > > size
> > > > > limit for the event processor queue either.
> > > > >
> > > > > On Thu, Jan 22, 2026 at 10:56 AM Sean Quah <sq...@confluent.io>
> wrote:
> > > > >
> > > > >> Hi all, thanks for the feedback so far.
> > > > >>
> > > > >> dj01: In the proposed changes section, you state that the
> timestamp of
> > > > >>> the last assignment is not persisted. How do you plan to
> bookkeep it
> > > if it
> > > > >>> is not stored with the assignment? Intuitively, I would add a
> > > timestamp to
> > > > >>> the assignment record.
> > > > >>
> > > > >> Thinking about it, it's easier to add it to the assignment
> record. I
> > > will
> > > > >> update the KIP. One thing to note is that the timestamp will be
> > > subject to
> > > > >> rollbacks when writing to the log fails, so we can allow extra
> > > assignment
> > > > >> runs when that happens.
> > > > >>
> > > > >> dj02: I wonder whether we should also add a "thread idle ratio"
> metric
> > > > >>> for the group coordinator executor. What do you think?
> > > > >>
> > > > >> I think it could be useful so I've added it to the KIP. The
> > > > >> implementation will have to be different to the event processor,
> > > since we
> > > > >> currently use an ExecutorService.
> > > > >>
> > > > >> dj03: If the executor is not used by the share coordinator, it
> should
> > > not
> > > > >>> expose any metrics about it. Is it possible to remove them?
> > > > >>
> > > > >> I've removed them from the KIP. We can add a parameter to the
> > > coordinator
> > > > >> metrics class to control whether they are visible.
> > > > >>
> > > > >> dj04: Is having one group coordinator executor thread sufficient
> by
> > > > >>> default for common workloads?
> > > > >>
> > > > >> Yes and no. I expect it will be very difficult to overload an
> entire
> > > > >> thread, ie. submit work faster than it can complete it. But
> updating
> > > the
> > > > >> default to two threads could be good for reducing delays due to
> > > > >> simultaneous assignor runs. I've raised the default to 2 threads.
> > > > >>
> > > > >> dj05: It seems you propose enabling the minimum assignor interval
> > > with a
> > > > >>> default of 5 seconds. However, the offloading is not enabled by
> > > default. Is
> > > > >>> the first one enough to guarantee the stability of the group
> > > coordinator?
> > > > >>> How do you foresee enabling the second one in the future? It
> would
> > > be great
> > > > >>> if you could address this in the KIP. We need a clear motivation
> for
> > > > >>> changing the default behavior and a plan for the future.
> > > > >>
> > > > >> I initially thought that offloading would increase rebalance
> times by
> > > 1
> > > > >> heartbeat and so didn't propose turning it on by default. But
> after
> > > some
> > > > >> more thinking, I believe both features will increase rebalance
> times
> > > by 1
> > > > >> heartbeat interval and the increase shouldn't stack. The minimum
> > > assignor
> > > > >> interval only impacts groups with more than 2 members, while
> > > offloading
> > > > >> only impacts groups with a single member. This is because in the
> other
> > > > >> cases, the extra delays are folded into existing revocation +
> > > heartbeat
> > > > >> delays. Note that share groups have no revocation so always see
> > > increased
> > > > >> rebalance times. I've updated the KIP to add the analysis of
> rebalance
> > > > >> times and propose turning both features on by default.
> > > > >>
> > > > >> dj06: Based on its description, I wonder whether `
> > > > >>> consumer.min.assignor.interval.ms` should be called `
> > > > >>> consumer.min.assignment.interval.ms`. What do you think?
> > > > >>
> > > > >> Thanks, I've renamed the config options in the KIP. What about the
> > > > >> assignor.offload.enable configs?
> > > > >>
> > > > >> dj07: It is not possible to enable/disable the offloading at the
> group
> > > > >>> level. This makes sense to me but it would be great to explain
> the
> > > > >>> rationale for it in the KIP.
> > > > >>
> > > > >> Thinking about it, there's nothing stopping us from configuring
> > > > >> offloading at the group level. In fact it might be desirable for
> some
> > > users
> > > > >> to disable offloading at the group coordinator level to keep
> > > rebalances
> > > > >> fast and only enable it for problematic large groups. I've added a
> > > > >> group-level override to the KIP.
> > > > >>
> > > > >> On Tue, Jan 20, 2026 at 1:38 PM Lianet Magrans <
> lianetm@apache.org>
> > > > >> wrote:
> > > > >>
> > > > >>> Hi Sean, thanks for the KIP.
> > > > >>>
> > > > >>> LM1: About group.initial.rebalance.delay.ms, I expect the
> > > interaction
> > > > >>> with the interval is just as described for the streams initial
> delay
> > > and
> > > > >>> interval, correct? Should we clarify that in the KIP (it only
> > > mentions the
> > > > >>> streams case)
> > > > >>>
> > > > >>> LM2: The KIP refers to batching assignment re-calculations
> triggered
> > > by
> > > > >>> member subscriptions changes, but I expect the batching mechanism
> > > applies
> > > > >>> the same when the assignment re-calculation is triggered by
> metadata
> > > > >>> changes (i.e topic/partition created or deleted), without any HB
> > > changing
> > > > >>> subscriptions. Is my understanding correct?
> > > > >>>
> > > > >>> LM3: About this section: "*When there is an in-flight assignor
> run
> > > for
> > > > >>> the group, there is no new target assignment. We will trigger the
> > > next
> > > > >>> assignor run on a future heartbeat.*". I expect that the next
> > > assignor
> > > > >>> run will be triggered on the next HB from this or from any other
> > > member of
> > > > >>> the group, received after the interval expires (without the
> members
> > > > >>> re-sending the subscription change). Is my expectation correct?
> If
> > > so,
> > > > >>> it may be worth clarifying in the KIP to avoid confusion with
> > > client-side
> > > > >>> implementations.
> > > > >>>
> > > > >>> Thanks!
> > > > >>> Lianet
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> On Tue, Jan 13, 2026 at 1:23 AM Sean Quah via dev <
> > > dev@kafka.apache.org>
> > > > >>> wrote:
> > > > >>>
> > > > >>>> sq01: We also have to update the SyncGroup request handling to
> only
> > > > >>>> return
> > > > >>>> REBALANCE_IN_PROGRESS when the member's epoch is behind the
> target
> > > > >>>> assignment epoch, not the group epoch. Thanks to Dongnuo for
> > > pointing
> > > > >>>> this
> > > > >>>> out.
> > > > >>>>
> > > > >>>> On Thu, Jan 8, 2026 at 5:40 PM Dongnuo Lyu via dev <
> > > > >>>> dev@kafka.apache.org>
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>> > Hi Sean, thanks for the KIP! I have a few questions as
> follows.
> > > > >>>> >
> > > > >>>> > dl01: Could we mention the handling when the group metadata or
> > > topic
> > > > >>>> > partition metadata is changed or deleted during the async
> assignor
> > > > >>>> run?
> > > > >>>> >
> > > > >>>> > dl02: This might be a question for the overall coordinator
> > > executor -
> > > > >>>> do we
> > > > >>>> > have plans to apply an explicit size limit to the executor
> queue?
> > > If
> > > > >>>> many
> > > > >>>> > groups trigger offloaded assignments simultaneously, should we
> > > apply
> > > > >>>> some
> > > > >>>> > backpressure for protection?
> > > > >>>> >
> > > > >>>> > Also resonate with dj05, for small groups default `
> > > > >>>> > min.assignor.interval.ms`
> > > > >>>> > to 5s might not be necessary, so not sure if we want to make
> the
> > > batch
> > > > >>>> > assignment default. Or it might be good to have a per group
> > > > >>>> enablement.
> > > > >>>> >
> > > > >>>> > Thanks
> > > > >>>> > Dongnuo
> > > > >>>> >
> > > > >>>>
> > > > >>>
> > >
>

Re: [DISCUSS] KIP-1263: Group Coordinator Assignment Batching and Offload

[Lucas Brutschy via dev <de...@kafka.apache.org>]

Hi Sean,

that makes a lot of sense, thanks for the explanation!

Cheers,
Lucas

On Sat, Jan 24, 2026 at 11:17 AM Sean Quah via dev <de...@kafka.apache.org> wrote:
>
> Hi Lucas,
>
> LB01: I'm just wondering if it would have been an option to instead
> > update the target assignment to remove the partitions from the member
> > immediately when the member unsubscribes?
>
> Good question. I can't recall my thought process when implementing
> KAFKA-19431. Three points spring to mind with updating the target
> assignment on subscription changes:
> 1. I think I wanted to preserve the property that the target assignment is
> only updated by assignment runs and immutable for a given epoch. Though it
> turns out that's not actually the case since we do patch the target
> assignment when members leave or static members are replaced.
> 2. An offloaded assignment based on older subscriptions can complete right
> after we patch the target assignment to remove unsubscribed topics so we
> would need to do some extra filtering on assignor completion.
> 3. Even if we update the target assignment, we would need to touch the
> reconciliation process anyway, since it wouldn't do anything when there is
> no epoch bump.
>
> There is certainly nothing stopping us from updating the target assignment.
> I think it seemed cleaner at the time to keep it all in the reconciliation
> process.
>
> Thanks,
> Sean
>
> On Fri, Jan 23, 2026 at 12:32 PM Lucas Brutschy <lb...@confluent.io>
> wrote:
>
> > Hey Sean,
> >
> > thanks for the KIP! This makes a lot of sense to me. I don't really
> > have anything I want you to change about the KIP.
> >
> > > We modify reconciliation to revoke any partitions the member is no
> > longer subscribed to, since the target assignment may lag behind member
> > subscriptions.
> >
> > LB01: I'm just wondering if it would have been an option to instead
> > update the target assignment to remove the partitions from the member
> > immediately when the member unsubscribes?
> >
> > Cheers,
> > Lucas
> >
> > On Thu, Jan 22, 2026 at 11:58 AM Sean Quah via dev <de...@kafka.apache.org>
> > wrote:
> > >
> > > >
> > > > LM1: About group.initial.rebalance.delay.ms, I expect the interaction
> > > > with the interval is just as described for the streams initial delay
> > and
> > > > interval, correct? Should we clarify that in the KIP (it only mentions
> > the
> > > > streams case)
> > >
> > > We haven't added a consumer or share group initial.rebalance.delay.ms
> > > config yet. It only exists for streams right now.
> > >
> > > LM2: The KIP refers to batching assignment re-calculations triggered by
> > > > member subscriptions changes, but I expect the batching mechanism
> > applies
> > > > the same when the assignment re-calculation is triggered by metadata
> > > > changes (i.e topic/partition created or deleted), without any HB
> > changing
> > > > subscriptions. Is my understanding correct?
> > >
> > > Yes, that's right. Topic metadata changes also bump the group epoch and
> > > triggers the same assignment flow.
> > >
> > > LM3: About this section: "When there is an in-flight assignor run for the
> > > > group, there is no new target assignment. We will trigger the next
> > assignor
> > > > run on a future heartbeat.". I expect that the next assignor run will
> > be
> > > > triggered on the next HB from this or from any other member of the
> > group,
> > > > received after the interval expires (without the members re-sending the
> > > > subscription change). Is my expectation correct? If so, it may be worth
> > > > clarifying in the KIP to avoid confusion with client-side
> > implementations.
> > >
> > > I tried to clarify in the KIP. Let me know your thoughts!
> > >
> > > On Thu, Jan 22, 2026 at 10:56 AM Sean Quah <sq...@confluent.io> wrote:
> > >
> > > > dl01: Could we mention the handling when the group metadata or
> > > >> topic partition metadata is changed or deleted during the async
> > assignor
> > > >> run?
> > > >
> > > > Thanks! I've added a paragraph to the Assignment Offload section
> > > > describing the handling of group metadata changes. Topic metadata
> > changes
> > > > already bump the group epoch and we don't need to handle them
> > specially.
> > > >
> > > > dl02: This might be a question for the overall coordinator executor -
> > do
> > > >> we have plans to apply an explicit size limit to the executor queue?
> > If
> > > >> many groups trigger offloaded assignments simultaneously, should we
> > apply
> > > >> some backpressure for protection?
> > > >
> > > > There aren't any plans for that right now. We actually don't have a
> > size
> > > > limit for the event processor queue either.
> > > >
> > > > On Thu, Jan 22, 2026 at 10:56 AM Sean Quah <sq...@confluent.io> wrote:
> > > >
> > > >> Hi all, thanks for the feedback so far.
> > > >>
> > > >> dj01: In the proposed changes section, you state that the timestamp of
> > > >>> the last assignment is not persisted. How do you plan to bookkeep it
> > if it
> > > >>> is not stored with the assignment? Intuitively, I would add a
> > timestamp to
> > > >>> the assignment record.
> > > >>
> > > >> Thinking about it, it's easier to add it to the assignment record. I
> > will
> > > >> update the KIP. One thing to note is that the timestamp will be
> > subject to
> > > >> rollbacks when writing to the log fails, so we can allow extra
> > assignment
> > > >> runs when that happens.
> > > >>
> > > >> dj02: I wonder whether we should also add a "thread idle ratio" metric
> > > >>> for the group coordinator executor. What do you think?
> > > >>
> > > >> I think it could be useful so I've added it to the KIP. The
> > > >> implementation will have to be different to the event processor,
> > since we
> > > >> currently use an ExecutorService.
> > > >>
> > > >> dj03: If the executor is not used by the share coordinator, it should
> > not
> > > >>> expose any metrics about it. Is it possible to remove them?
> > > >>
> > > >> I've removed them from the KIP. We can add a parameter to the
> > coordinator
> > > >> metrics class to control whether they are visible.
> > > >>
> > > >> dj04: Is having one group coordinator executor thread sufficient by
> > > >>> default for common workloads?
> > > >>
> > > >> Yes and no. I expect it will be very difficult to overload an entire
> > > >> thread, ie. submit work faster than it can complete it. But updating
> > the
> > > >> default to two threads could be good for reducing delays due to
> > > >> simultaneous assignor runs. I've raised the default to 2 threads.
> > > >>
> > > >> dj05: It seems you propose enabling the minimum assignor interval
> > with a
> > > >>> default of 5 seconds. However, the offloading is not enabled by
> > default. Is
> > > >>> the first one enough to guarantee the stability of the group
> > coordinator?
> > > >>> How do you foresee enabling the second one in the future? It would
> > be great
> > > >>> if you could address this in the KIP. We need a clear motivation for
> > > >>> changing the default behavior and a plan for the future.
> > > >>
> > > >> I initially thought that offloading would increase rebalance times by
> > 1
> > > >> heartbeat and so didn't propose turning it on by default. But after
> > some
> > > >> more thinking, I believe both features will increase rebalance times
> > by 1
> > > >> heartbeat interval and the increase shouldn't stack. The minimum
> > assignor
> > > >> interval only impacts groups with more than 2 members, while
> > offloading
> > > >> only impacts groups with a single member. This is because in the other
> > > >> cases, the extra delays are folded into existing revocation +
> > heartbeat
> > > >> delays. Note that share groups have no revocation so always see
> > increased
> > > >> rebalance times. I've updated the KIP to add the analysis of rebalance
> > > >> times and propose turning both features on by default.
> > > >>
> > > >> dj06: Based on its description, I wonder whether `
> > > >>> consumer.min.assignor.interval.ms` should be called `
> > > >>> consumer.min.assignment.interval.ms`. What do you think?
> > > >>
> > > >> Thanks, I've renamed the config options in the KIP. What about the
> > > >> assignor.offload.enable configs?
> > > >>
> > > >> dj07: It is not possible to enable/disable the offloading at the group
> > > >>> level. This makes sense to me but it would be great to explain the
> > > >>> rationale for it in the KIP.
> > > >>
> > > >> Thinking about it, there's nothing stopping us from configuring
> > > >> offloading at the group level. In fact it might be desirable for some
> > users
> > > >> to disable offloading at the group coordinator level to keep
> > rebalances
> > > >> fast and only enable it for problematic large groups. I've added a
> > > >> group-level override to the KIP.
> > > >>
> > > >> On Tue, Jan 20, 2026 at 1:38 PM Lianet Magrans <li...@apache.org>
> > > >> wrote:
> > > >>
> > > >>> Hi Sean, thanks for the KIP.
> > > >>>
> > > >>> LM1: About group.initial.rebalance.delay.ms, I expect the
> > interaction
> > > >>> with the interval is just as described for the streams initial delay
> > and
> > > >>> interval, correct? Should we clarify that in the KIP (it only
> > mentions the
> > > >>> streams case)
> > > >>>
> > > >>> LM2: The KIP refers to batching assignment re-calculations triggered
> > by
> > > >>> member subscriptions changes, but I expect the batching mechanism
> > applies
> > > >>> the same when the assignment re-calculation is triggered by metadata
> > > >>> changes (i.e topic/partition created or deleted), without any HB
> > changing
> > > >>> subscriptions. Is my understanding correct?
> > > >>>
> > > >>> LM3: About this section: "*When there is an in-flight assignor run
> > for
> > > >>> the group, there is no new target assignment. We will trigger the
> > next
> > > >>> assignor run on a future heartbeat.*". I expect that the next
> > assignor
> > > >>> run will be triggered on the next HB from this or from any other
> > member of
> > > >>> the group, received after the interval expires (without the members
> > > >>> re-sending the subscription change). Is my expectation correct? If
> > so,
> > > >>> it may be worth clarifying in the KIP to avoid confusion with
> > client-side
> > > >>> implementations.
> > > >>>
> > > >>> Thanks!
> > > >>> Lianet
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Tue, Jan 13, 2026 at 1:23 AM Sean Quah via dev <
> > dev@kafka.apache.org>
> > > >>> wrote:
> > > >>>
> > > >>>> sq01: We also have to update the SyncGroup request handling to only
> > > >>>> return
> > > >>>> REBALANCE_IN_PROGRESS when the member's epoch is behind the target
> > > >>>> assignment epoch, not the group epoch. Thanks to Dongnuo for
> > pointing
> > > >>>> this
> > > >>>> out.
> > > >>>>
> > > >>>> On Thu, Jan 8, 2026 at 5:40 PM Dongnuo Lyu via dev <
> > > >>>> dev@kafka.apache.org>
> > > >>>> wrote:
> > > >>>>
> > > >>>> > Hi Sean, thanks for the KIP! I have a few questions as follows.
> > > >>>> >
> > > >>>> > dl01: Could we mention the handling when the group metadata or
> > topic
> > > >>>> > partition metadata is changed or deleted during the async assignor
> > > >>>> run?
> > > >>>> >
> > > >>>> > dl02: This might be a question for the overall coordinator
> > executor -
> > > >>>> do we
> > > >>>> > have plans to apply an explicit size limit to the executor queue?
> > If
> > > >>>> many
> > > >>>> > groups trigger offloaded assignments simultaneously, should we
> > apply
> > > >>>> some
> > > >>>> > backpressure for protection?
> > > >>>> >
> > > >>>> > Also resonate with dj05, for small groups default `
> > > >>>> > min.assignor.interval.ms`
> > > >>>> > to 5s might not be necessary, so not sure if we want to make the
> > batch
> > > >>>> > assignment default. Or it might be good to have a per group
> > > >>>> enablement.
> > > >>>> >
> > > >>>> > Thanks
> > > >>>> > Dongnuo
> > > >>>> >
> > > >>>>
> > > >>>
> >

Re: [DISCUSS] KIP-1263: Group Coordinator Assignment Batching and Offload

[Sean Quah via dev <de...@kafka.apache.org>]

Hi Lucas,

LB01: I'm just wondering if it would have been an option to instead
> update the target assignment to remove the partitions from the member
> immediately when the member unsubscribes?

Good question. I can't recall my thought process when implementing
KAFKA-19431. Three points spring to mind with updating the target
assignment on subscription changes:
1. I think I wanted to preserve the property that the target assignment is
only updated by assignment runs and immutable for a given epoch. Though it
turns out that's not actually the case since we do patch the target
assignment when members leave or static members are replaced.
2. An offloaded assignment based on older subscriptions can complete right
after we patch the target assignment to remove unsubscribed topics so we
would need to do some extra filtering on assignor completion.
3. Even if we update the target assignment, we would need to touch the
reconciliation process anyway, since it wouldn't do anything when there is
no epoch bump.

There is certainly nothing stopping us from updating the target assignment.
I think it seemed cleaner at the time to keep it all in the reconciliation
process.

Thanks,
Sean

On Fri, Jan 23, 2026 at 12:32 PM Lucas Brutschy <lb...@confluent.io>
wrote:

> Hey Sean,
>
> thanks for the KIP! This makes a lot of sense to me. I don't really
> have anything I want you to change about the KIP.
>
> > We modify reconciliation to revoke any partitions the member is no
> longer subscribed to, since the target assignment may lag behind member
> subscriptions.
>
> LB01: I'm just wondering if it would have been an option to instead
> update the target assignment to remove the partitions from the member
> immediately when the member unsubscribes?
>
> Cheers,
> Lucas
>
> On Thu, Jan 22, 2026 at 11:58 AM Sean Quah via dev <de...@kafka.apache.org>
> wrote:
> >
> > >
> > > LM1: About group.initial.rebalance.delay.ms, I expect the interaction
> > > with the interval is just as described for the streams initial delay
> and
> > > interval, correct? Should we clarify that in the KIP (it only mentions
> the
> > > streams case)
> >
> > We haven't added a consumer or share group initial.rebalance.delay.ms
> > config yet. It only exists for streams right now.
> >
> > LM2: The KIP refers to batching assignment re-calculations triggered by
> > > member subscriptions changes, but I expect the batching mechanism
> applies
> > > the same when the assignment re-calculation is triggered by metadata
> > > changes (i.e topic/partition created or deleted), without any HB
> changing
> > > subscriptions. Is my understanding correct?
> >
> > Yes, that's right. Topic metadata changes also bump the group epoch and
> > triggers the same assignment flow.
> >
> > LM3: About this section: "When there is an in-flight assignor run for the
> > > group, there is no new target assignment. We will trigger the next
> assignor
> > > run on a future heartbeat.". I expect that the next assignor run will
> be
> > > triggered on the next HB from this or from any other member of the
> group,
> > > received after the interval expires (without the members re-sending the
> > > subscription change). Is my expectation correct? If so, it may be worth
> > > clarifying in the KIP to avoid confusion with client-side
> implementations.
> >
> > I tried to clarify in the KIP. Let me know your thoughts!
> >
> > On Thu, Jan 22, 2026 at 10:56 AM Sean Quah <sq...@confluent.io> wrote:
> >
> > > dl01: Could we mention the handling when the group metadata or
> > >> topic partition metadata is changed or deleted during the async
> assignor
> > >> run?
> > >
> > > Thanks! I've added a paragraph to the Assignment Offload section
> > > describing the handling of group metadata changes. Topic metadata
> changes
> > > already bump the group epoch and we don't need to handle them
> specially.
> > >
> > > dl02: This might be a question for the overall coordinator executor -
> do
> > >> we have plans to apply an explicit size limit to the executor queue?
> If
> > >> many groups trigger offloaded assignments simultaneously, should we
> apply
> > >> some backpressure for protection?
> > >
> > > There aren't any plans for that right now. We actually don't have a
> size
> > > limit for the event processor queue either.
> > >
> > > On Thu, Jan 22, 2026 at 10:56 AM Sean Quah <sq...@confluent.io> wrote:
> > >
> > >> Hi all, thanks for the feedback so far.
> > >>
> > >> dj01: In the proposed changes section, you state that the timestamp of
> > >>> the last assignment is not persisted. How do you plan to bookkeep it
> if it
> > >>> is not stored with the assignment? Intuitively, I would add a
> timestamp to
> > >>> the assignment record.
> > >>
> > >> Thinking about it, it's easier to add it to the assignment record. I
> will
> > >> update the KIP. One thing to note is that the timestamp will be
> subject to
> > >> rollbacks when writing to the log fails, so we can allow extra
> assignment
> > >> runs when that happens.
> > >>
> > >> dj02: I wonder whether we should also add a "thread idle ratio" metric
> > >>> for the group coordinator executor. What do you think?
> > >>
> > >> I think it could be useful so I've added it to the KIP. The
> > >> implementation will have to be different to the event processor,
> since we
> > >> currently use an ExecutorService.
> > >>
> > >> dj03: If the executor is not used by the share coordinator, it should
> not
> > >>> expose any metrics about it. Is it possible to remove them?
> > >>
> > >> I've removed them from the KIP. We can add a parameter to the
> coordinator
> > >> metrics class to control whether they are visible.
> > >>
> > >> dj04: Is having one group coordinator executor thread sufficient by
> > >>> default for common workloads?
> > >>
> > >> Yes and no. I expect it will be very difficult to overload an entire
> > >> thread, ie. submit work faster than it can complete it. But updating
> the
> > >> default to two threads could be good for reducing delays due to
> > >> simultaneous assignor runs. I've raised the default to 2 threads.
> > >>
> > >> dj05: It seems you propose enabling the minimum assignor interval
> with a
> > >>> default of 5 seconds. However, the offloading is not enabled by
> default. Is
> > >>> the first one enough to guarantee the stability of the group
> coordinator?
> > >>> How do you foresee enabling the second one in the future? It would
> be great
> > >>> if you could address this in the KIP. We need a clear motivation for
> > >>> changing the default behavior and a plan for the future.
> > >>
> > >> I initially thought that offloading would increase rebalance times by
> 1
> > >> heartbeat and so didn't propose turning it on by default. But after
> some
> > >> more thinking, I believe both features will increase rebalance times
> by 1
> > >> heartbeat interval and the increase shouldn't stack. The minimum
> assignor
> > >> interval only impacts groups with more than 2 members, while
> offloading
> > >> only impacts groups with a single member. This is because in the other
> > >> cases, the extra delays are folded into existing revocation +
> heartbeat
> > >> delays. Note that share groups have no revocation so always see
> increased
> > >> rebalance times. I've updated the KIP to add the analysis of rebalance
> > >> times and propose turning both features on by default.
> > >>
> > >> dj06: Based on its description, I wonder whether `
> > >>> consumer.min.assignor.interval.ms` should be called `
> > >>> consumer.min.assignment.interval.ms`. What do you think?
> > >>
> > >> Thanks, I've renamed the config options in the KIP. What about the
> > >> assignor.offload.enable configs?
> > >>
> > >> dj07: It is not possible to enable/disable the offloading at the group
> > >>> level. This makes sense to me but it would be great to explain the
> > >>> rationale for it in the KIP.
> > >>
> > >> Thinking about it, there's nothing stopping us from configuring
> > >> offloading at the group level. In fact it might be desirable for some
> users
> > >> to disable offloading at the group coordinator level to keep
> rebalances
> > >> fast and only enable it for problematic large groups. I've added a
> > >> group-level override to the KIP.
> > >>
> > >> On Tue, Jan 20, 2026 at 1:38 PM Lianet Magrans <li...@apache.org>
> > >> wrote:
> > >>
> > >>> Hi Sean, thanks for the KIP.
> > >>>
> > >>> LM1: About group.initial.rebalance.delay.ms, I expect the
> interaction
> > >>> with the interval is just as described for the streams initial delay
> and
> > >>> interval, correct? Should we clarify that in the KIP (it only
> mentions the
> > >>> streams case)
> > >>>
> > >>> LM2: The KIP refers to batching assignment re-calculations triggered
> by
> > >>> member subscriptions changes, but I expect the batching mechanism
> applies
> > >>> the same when the assignment re-calculation is triggered by metadata
> > >>> changes (i.e topic/partition created or deleted), without any HB
> changing
> > >>> subscriptions. Is my understanding correct?
> > >>>
> > >>> LM3: About this section: "*When there is an in-flight assignor run
> for
> > >>> the group, there is no new target assignment. We will trigger the
> next
> > >>> assignor run on a future heartbeat.*". I expect that the next
> assignor
> > >>> run will be triggered on the next HB from this or from any other
> member of
> > >>> the group, received after the interval expires (without the members
> > >>> re-sending the subscription change). Is my expectation correct? If
> so,
> > >>> it may be worth clarifying in the KIP to avoid confusion with
> client-side
> > >>> implementations.
> > >>>
> > >>> Thanks!
> > >>> Lianet
> > >>>
> > >>>
> > >>>
> > >>> On Tue, Jan 13, 2026 at 1:23 AM Sean Quah via dev <
> dev@kafka.apache.org>
> > >>> wrote:
> > >>>
> > >>>> sq01: We also have to update the SyncGroup request handling to only
> > >>>> return
> > >>>> REBALANCE_IN_PROGRESS when the member's epoch is behind the target
> > >>>> assignment epoch, not the group epoch. Thanks to Dongnuo for
> pointing
> > >>>> this
> > >>>> out.
> > >>>>
> > >>>> On Thu, Jan 8, 2026 at 5:40 PM Dongnuo Lyu via dev <
> > >>>> dev@kafka.apache.org>
> > >>>> wrote:
> > >>>>
> > >>>> > Hi Sean, thanks for the KIP! I have a few questions as follows.
> > >>>> >
> > >>>> > dl01: Could we mention the handling when the group metadata or
> topic
> > >>>> > partition metadata is changed or deleted during the async assignor
> > >>>> run?
> > >>>> >
> > >>>> > dl02: This might be a question for the overall coordinator
> executor -
> > >>>> do we
> > >>>> > have plans to apply an explicit size limit to the executor queue?
> If
> > >>>> many
> > >>>> > groups trigger offloaded assignments simultaneously, should we
> apply
> > >>>> some
> > >>>> > backpressure for protection?
> > >>>> >
> > >>>> > Also resonate with dj05, for small groups default `
> > >>>> > min.assignor.interval.ms`
> > >>>> > to 5s might not be necessary, so not sure if we want to make the
> batch
> > >>>> > assignment default. Or it might be good to have a per group
> > >>>> enablement.
> > >>>> >
> > >>>> > Thanks
> > >>>> > Dongnuo
> > >>>> >
> > >>>>
> > >>>
>

Re: [DISCUSS] KIP-1263: Group Coordinator Assignment Batching and Offload

[Lucas Brutschy via dev <de...@kafka.apache.org>]

Hey Sean,

thanks for the KIP! This makes a lot of sense to me. I don't really
have anything I want you to change about the KIP.

> We modify reconciliation to revoke any partitions the member is no longer subscribed to, since the target assignment may lag behind member subscriptions.

LB01: I'm just wondering if it would have been an option to instead
update the target assignment to remove the partitions from the member
immediately when the member unsubscribes?

Cheers,
Lucas

On Thu, Jan 22, 2026 at 11:58 AM Sean Quah via dev <de...@kafka.apache.org> wrote:
>
> >
> > LM1: About group.initial.rebalance.delay.ms, I expect the interaction
> > with the interval is just as described for the streams initial delay and
> > interval, correct? Should we clarify that in the KIP (it only mentions the
> > streams case)
>
> We haven't added a consumer or share group initial.rebalance.delay.ms
> config yet. It only exists for streams right now.
>
> LM2: The KIP refers to batching assignment re-calculations triggered by
> > member subscriptions changes, but I expect the batching mechanism applies
> > the same when the assignment re-calculation is triggered by metadata
> > changes (i.e topic/partition created or deleted), without any HB changing
> > subscriptions. Is my understanding correct?
>
> Yes, that's right. Topic metadata changes also bump the group epoch and
> triggers the same assignment flow.
>
> LM3: About this section: "When there is an in-flight assignor run for the
> > group, there is no new target assignment. We will trigger the next assignor
> > run on a future heartbeat.". I expect that the next assignor run will be
> > triggered on the next HB from this or from any other member of the group,
> > received after the interval expires (without the members re-sending the
> > subscription change). Is my expectation correct? If so, it may be worth
> > clarifying in the KIP to avoid confusion with client-side implementations.
>
> I tried to clarify in the KIP. Let me know your thoughts!
>
> On Thu, Jan 22, 2026 at 10:56 AM Sean Quah <sq...@confluent.io> wrote:
>
> > dl01: Could we mention the handling when the group metadata or
> >> topic partition metadata is changed or deleted during the async assignor
> >> run?
> >
> > Thanks! I've added a paragraph to the Assignment Offload section
> > describing the handling of group metadata changes. Topic metadata changes
> > already bump the group epoch and we don't need to handle them specially.
> >
> > dl02: This might be a question for the overall coordinator executor - do
> >> we have plans to apply an explicit size limit to the executor queue? If
> >> many groups trigger offloaded assignments simultaneously, should we apply
> >> some backpressure for protection?
> >
> > There aren't any plans for that right now. We actually don't have a size
> > limit for the event processor queue either.
> >
> > On Thu, Jan 22, 2026 at 10:56 AM Sean Quah <sq...@confluent.io> wrote:
> >
> >> Hi all, thanks for the feedback so far.
> >>
> >> dj01: In the proposed changes section, you state that the timestamp of
> >>> the last assignment is not persisted. How do you plan to bookkeep it if it
> >>> is not stored with the assignment? Intuitively, I would add a timestamp to
> >>> the assignment record.
> >>
> >> Thinking about it, it's easier to add it to the assignment record. I will
> >> update the KIP. One thing to note is that the timestamp will be subject to
> >> rollbacks when writing to the log fails, so we can allow extra assignment
> >> runs when that happens.
> >>
> >> dj02: I wonder whether we should also add a "thread idle ratio" metric
> >>> for the group coordinator executor. What do you think?
> >>
> >> I think it could be useful so I've added it to the KIP. The
> >> implementation will have to be different to the event processor, since we
> >> currently use an ExecutorService.
> >>
> >> dj03: If the executor is not used by the share coordinator, it should not
> >>> expose any metrics about it. Is it possible to remove them?
> >>
> >> I've removed them from the KIP. We can add a parameter to the coordinator
> >> metrics class to control whether they are visible.
> >>
> >> dj04: Is having one group coordinator executor thread sufficient by
> >>> default for common workloads?
> >>
> >> Yes and no. I expect it will be very difficult to overload an entire
> >> thread, ie. submit work faster than it can complete it. But updating the
> >> default to two threads could be good for reducing delays due to
> >> simultaneous assignor runs. I've raised the default to 2 threads.
> >>
> >> dj05: It seems you propose enabling the minimum assignor interval with a
> >>> default of 5 seconds. However, the offloading is not enabled by default. Is
> >>> the first one enough to guarantee the stability of the group coordinator?
> >>> How do you foresee enabling the second one in the future? It would be great
> >>> if you could address this in the KIP. We need a clear motivation for
> >>> changing the default behavior and a plan for the future.
> >>
> >> I initially thought that offloading would increase rebalance times by 1
> >> heartbeat and so didn't propose turning it on by default. But after some
> >> more thinking, I believe both features will increase rebalance times by 1
> >> heartbeat interval and the increase shouldn't stack. The minimum assignor
> >> interval only impacts groups with more than 2 members, while offloading
> >> only impacts groups with a single member. This is because in the other
> >> cases, the extra delays are folded into existing revocation + heartbeat
> >> delays. Note that share groups have no revocation so always see increased
> >> rebalance times. I've updated the KIP to add the analysis of rebalance
> >> times and propose turning both features on by default.
> >>
> >> dj06: Based on its description, I wonder whether `
> >>> consumer.min.assignor.interval.ms` should be called `
> >>> consumer.min.assignment.interval.ms`. What do you think?
> >>
> >> Thanks, I've renamed the config options in the KIP. What about the
> >> assignor.offload.enable configs?
> >>
> >> dj07: It is not possible to enable/disable the offloading at the group
> >>> level. This makes sense to me but it would be great to explain the
> >>> rationale for it in the KIP.
> >>
> >> Thinking about it, there's nothing stopping us from configuring
> >> offloading at the group level. In fact it might be desirable for some users
> >> to disable offloading at the group coordinator level to keep rebalances
> >> fast and only enable it for problematic large groups. I've added a
> >> group-level override to the KIP.
> >>
> >> On Tue, Jan 20, 2026 at 1:38 PM Lianet Magrans <li...@apache.org>
> >> wrote:
> >>
> >>> Hi Sean, thanks for the KIP.
> >>>
> >>> LM1: About group.initial.rebalance.delay.ms, I expect the interaction
> >>> with the interval is just as described for the streams initial delay and
> >>> interval, correct? Should we clarify that in the KIP (it only mentions the
> >>> streams case)
> >>>
> >>> LM2: The KIP refers to batching assignment re-calculations triggered by
> >>> member subscriptions changes, but I expect the batching mechanism applies
> >>> the same when the assignment re-calculation is triggered by metadata
> >>> changes (i.e topic/partition created or deleted), without any HB changing
> >>> subscriptions. Is my understanding correct?
> >>>
> >>> LM3: About this section: "*When there is an in-flight assignor run for
> >>> the group, there is no new target assignment. We will trigger the next
> >>> assignor run on a future heartbeat.*". I expect that the next assignor
> >>> run will be triggered on the next HB from this or from any other member of
> >>> the group, received after the interval expires (without the members
> >>> re-sending the subscription change). Is my expectation correct? If so,
> >>> it may be worth clarifying in the KIP to avoid confusion with client-side
> >>> implementations.
> >>>
> >>> Thanks!
> >>> Lianet
> >>>
> >>>
> >>>
> >>> On Tue, Jan 13, 2026 at 1:23 AM Sean Quah via dev <de...@kafka.apache.org>
> >>> wrote:
> >>>
> >>>> sq01: We also have to update the SyncGroup request handling to only
> >>>> return
> >>>> REBALANCE_IN_PROGRESS when the member's epoch is behind the target
> >>>> assignment epoch, not the group epoch. Thanks to Dongnuo for pointing
> >>>> this
> >>>> out.
> >>>>
> >>>> On Thu, Jan 8, 2026 at 5:40 PM Dongnuo Lyu via dev <
> >>>> dev@kafka.apache.org>
> >>>> wrote:
> >>>>
> >>>> > Hi Sean, thanks for the KIP! I have a few questions as follows.
> >>>> >
> >>>> > dl01: Could we mention the handling when the group metadata or topic
> >>>> > partition metadata is changed or deleted during the async assignor
> >>>> run?
> >>>> >
> >>>> > dl02: This might be a question for the overall coordinator executor -
> >>>> do we
> >>>> > have plans to apply an explicit size limit to the executor queue? If
> >>>> many
> >>>> > groups trigger offloaded assignments simultaneously, should we apply
> >>>> some
> >>>> > backpressure for protection?
> >>>> >
> >>>> > Also resonate with dj05, for small groups default `
> >>>> > min.assignor.interval.ms`
> >>>> > to 5s might not be necessary, so not sure if we want to make the batch
> >>>> > assignment default. Or it might be good to have a per group
> >>>> enablement.
> >>>> >
> >>>> > Thanks
> >>>> > Dongnuo
> >>>> >
> >>>>
> >>>

Re: [DISCUSS] KIP-1263: Group Coordinator Assignment Batching and Offload

[Sean Quah via dev <de...@kafka.apache.org>]

>
> LM1: About group.initial.rebalance.delay.ms, I expect the interaction
> with the interval is just as described for the streams initial delay and
> interval, correct? Should we clarify that in the KIP (it only mentions the
> streams case)

We haven't added a consumer or share group initial.rebalance.delay.ms
config yet. It only exists for streams right now.

LM2: The KIP refers to batching assignment re-calculations triggered by
> member subscriptions changes, but I expect the batching mechanism applies
> the same when the assignment re-calculation is triggered by metadata
> changes (i.e topic/partition created or deleted), without any HB changing
> subscriptions. Is my understanding correct?

Yes, that's right. Topic metadata changes also bump the group epoch and
triggers the same assignment flow.

LM3: About this section: "When there is an in-flight assignor run for the
> group, there is no new target assignment. We will trigger the next assignor
> run on a future heartbeat.". I expect that the next assignor run will be
> triggered on the next HB from this or from any other member of the group,
> received after the interval expires (without the members re-sending the
> subscription change). Is my expectation correct? If so, it may be worth
> clarifying in the KIP to avoid confusion with client-side implementations.

I tried to clarify in the KIP. Let me know your thoughts!

On Thu, Jan 22, 2026 at 10:56 AM Sean Quah <sq...@confluent.io> wrote:

> dl01: Could we mention the handling when the group metadata or
>> topic partition metadata is changed or deleted during the async assignor
>> run?
>
> Thanks! I've added a paragraph to the Assignment Offload section
> describing the handling of group metadata changes. Topic metadata changes
> already bump the group epoch and we don't need to handle them specially.
>
> dl02: This might be a question for the overall coordinator executor - do
>> we have plans to apply an explicit size limit to the executor queue? If
>> many groups trigger offloaded assignments simultaneously, should we apply
>> some backpressure for protection?
>
> There aren't any plans for that right now. We actually don't have a size
> limit for the event processor queue either.
>
> On Thu, Jan 22, 2026 at 10:56 AM Sean Quah <sq...@confluent.io> wrote:
>
>> Hi all, thanks for the feedback so far.
>>
>> dj01: In the proposed changes section, you state that the timestamp of
>>> the last assignment is not persisted. How do you plan to bookkeep it if it
>>> is not stored with the assignment? Intuitively, I would add a timestamp to
>>> the assignment record.
>>
>> Thinking about it, it's easier to add it to the assignment record. I will
>> update the KIP. One thing to note is that the timestamp will be subject to
>> rollbacks when writing to the log fails, so we can allow extra assignment
>> runs when that happens.
>>
>> dj02: I wonder whether we should also add a "thread idle ratio" metric
>>> for the group coordinator executor. What do you think?
>>
>> I think it could be useful so I've added it to the KIP. The
>> implementation will have to be different to the event processor, since we
>> currently use an ExecutorService.
>>
>> dj03: If the executor is not used by the share coordinator, it should not
>>> expose any metrics about it. Is it possible to remove them?
>>
>> I've removed them from the KIP. We can add a parameter to the coordinator
>> metrics class to control whether they are visible.
>>
>> dj04: Is having one group coordinator executor thread sufficient by
>>> default for common workloads?
>>
>> Yes and no. I expect it will be very difficult to overload an entire
>> thread, ie. submit work faster than it can complete it. But updating the
>> default to two threads could be good for reducing delays due to
>> simultaneous assignor runs. I've raised the default to 2 threads.
>>
>> dj05: It seems you propose enabling the minimum assignor interval with a
>>> default of 5 seconds. However, the offloading is not enabled by default. Is
>>> the first one enough to guarantee the stability of the group coordinator?
>>> How do you foresee enabling the second one in the future? It would be great
>>> if you could address this in the KIP. We need a clear motivation for
>>> changing the default behavior and a plan for the future.
>>
>> I initially thought that offloading would increase rebalance times by 1
>> heartbeat and so didn't propose turning it on by default. But after some
>> more thinking, I believe both features will increase rebalance times by 1
>> heartbeat interval and the increase shouldn't stack. The minimum assignor
>> interval only impacts groups with more than 2 members, while offloading
>> only impacts groups with a single member. This is because in the other
>> cases, the extra delays are folded into existing revocation + heartbeat
>> delays. Note that share groups have no revocation so always see increased
>> rebalance times. I've updated the KIP to add the analysis of rebalance
>> times and propose turning both features on by default.
>>
>> dj06: Based on its description, I wonder whether `
>>> consumer.min.assignor.interval.ms` should be called `
>>> consumer.min.assignment.interval.ms`. What do you think?
>>
>> Thanks, I've renamed the config options in the KIP. What about the
>> assignor.offload.enable configs?
>>
>> dj07: It is not possible to enable/disable the offloading at the group
>>> level. This makes sense to me but it would be great to explain the
>>> rationale for it in the KIP.
>>
>> Thinking about it, there's nothing stopping us from configuring
>> offloading at the group level. In fact it might be desirable for some users
>> to disable offloading at the group coordinator level to keep rebalances
>> fast and only enable it for problematic large groups. I've added a
>> group-level override to the KIP.
>>
>> On Tue, Jan 20, 2026 at 1:38 PM Lianet Magrans <li...@apache.org>
>> wrote:
>>
>>> Hi Sean, thanks for the KIP.
>>>
>>> LM1: About group.initial.rebalance.delay.ms, I expect the interaction
>>> with the interval is just as described for the streams initial delay and
>>> interval, correct? Should we clarify that in the KIP (it only mentions the
>>> streams case)
>>>
>>> LM2: The KIP refers to batching assignment re-calculations triggered by
>>> member subscriptions changes, but I expect the batching mechanism applies
>>> the same when the assignment re-calculation is triggered by metadata
>>> changes (i.e topic/partition created or deleted), without any HB changing
>>> subscriptions. Is my understanding correct?
>>>
>>> LM3: About this section: "*When there is an in-flight assignor run for
>>> the group, there is no new target assignment. We will trigger the next
>>> assignor run on a future heartbeat.*". I expect that the next assignor
>>> run will be triggered on the next HB from this or from any other member of
>>> the group, received after the interval expires (without the members
>>> re-sending the subscription change). Is my expectation correct? If so,
>>> it may be worth clarifying in the KIP to avoid confusion with client-side
>>> implementations.
>>>
>>> Thanks!
>>> Lianet
>>>
>>>
>>>
>>> On Tue, Jan 13, 2026 at 1:23 AM Sean Quah via dev <de...@kafka.apache.org>
>>> wrote:
>>>
>>>> sq01: We also have to update the SyncGroup request handling to only
>>>> return
>>>> REBALANCE_IN_PROGRESS when the member's epoch is behind the target
>>>> assignment epoch, not the group epoch. Thanks to Dongnuo for pointing
>>>> this
>>>> out.
>>>>
>>>> On Thu, Jan 8, 2026 at 5:40 PM Dongnuo Lyu via dev <
>>>> dev@kafka.apache.org>
>>>> wrote:
>>>>
>>>> > Hi Sean, thanks for the KIP! I have a few questions as follows.
>>>> >
>>>> > dl01: Could we mention the handling when the group metadata or topic
>>>> > partition metadata is changed or deleted during the async assignor
>>>> run?
>>>> >
>>>> > dl02: This might be a question for the overall coordinator executor -
>>>> do we
>>>> > have plans to apply an explicit size limit to the executor queue? If
>>>> many
>>>> > groups trigger offloaded assignments simultaneously, should we apply
>>>> some
>>>> > backpressure for protection?
>>>> >
>>>> > Also resonate with dj05, for small groups default `
>>>> > min.assignor.interval.ms`
>>>> > to 5s might not be necessary, so not sure if we want to make the batch
>>>> > assignment default. Or it might be good to have a per group
>>>> enablement.
>>>> >
>>>> > Thanks
>>>> > Dongnuo
>>>> >
>>>>
>>>

Re: [DISCUSS] KIP-1263: Group Coordinator Assignment Batching and Offload

[Sean Quah via dev <de...@kafka.apache.org>]

>
> dl01: Could we mention the handling when the group metadata or
> topic partition metadata is changed or deleted during the async assignor
> run?

Thanks! I've added a paragraph to the Assignment Offload section describing
the handling of group metadata changes. Topic metadata changes already bump
the group epoch and we don't need to handle them specially.

dl02: This might be a question for the overall coordinator executor - do we
> have plans to apply an explicit size limit to the executor queue? If many
> groups trigger offloaded assignments simultaneously, should we apply some
> backpressure for protection?

There aren't any plans for that right now. We actually don't have a size
limit for the event processor queue either.

On Thu, Jan 22, 2026 at 10:56 AM Sean Quah <sq...@confluent.io> wrote:

> Hi all, thanks for the feedback so far.
>
> dj01: In the proposed changes section, you state that the timestamp of the
>> last assignment is not persisted. How do you plan to bookkeep it if it is
>> not stored with the assignment? Intuitively, I would add a timestamp to the
>> assignment record.
>
> Thinking about it, it's easier to add it to the assignment record. I will
> update the KIP. One thing to note is that the timestamp will be subject to
> rollbacks when writing to the log fails, so we can allow extra assignment
> runs when that happens.
>
> dj02: I wonder whether we should also add a "thread idle ratio" metric for
>> the group coordinator executor. What do you think?
>
> I think it could be useful so I've added it to the KIP. The implementation
> will have to be different to the event processor, since we currently use an
> ExecutorService.
>
> dj03: If the executor is not used by the share coordinator, it should not
>> expose any metrics about it. Is it possible to remove them?
>
> I've removed them from the KIP. We can add a parameter to the coordinator
> metrics class to control whether they are visible.
>
> dj04: Is having one group coordinator executor thread sufficient by
>> default for common workloads?
>
> Yes and no. I expect it will be very difficult to overload an entire
> thread, ie. submit work faster than it can complete it. But updating the
> default to two threads could be good for reducing delays due to
> simultaneous assignor runs. I've raised the default to 2 threads.
>
> dj05: It seems you propose enabling the minimum assignor interval with a
>> default of 5 seconds. However, the offloading is not enabled by default. Is
>> the first one enough to guarantee the stability of the group coordinator?
>> How do you foresee enabling the second one in the future? It would be great
>> if you could address this in the KIP. We need a clear motivation for
>> changing the default behavior and a plan for the future.
>
> I initially thought that offloading would increase rebalance times by 1
> heartbeat and so didn't propose turning it on by default. But after some
> more thinking, I believe both features will increase rebalance times by 1
> heartbeat interval and the increase shouldn't stack. The minimum assignor
> interval only impacts groups with more than 2 members, while offloading
> only impacts groups with a single member. This is because in the other
> cases, the extra delays are folded into existing revocation + heartbeat
> delays. Note that share groups have no revocation so always see increased
> rebalance times. I've updated the KIP to add the analysis of rebalance
> times and propose turning both features on by default.
>
> dj06: Based on its description, I wonder whether `
>> consumer.min.assignor.interval.ms` should be called `
>> consumer.min.assignment.interval.ms`. What do you think?
>
> Thanks, I've renamed the config options in the KIP. What about the
> assignor.offload.enable configs?
>
> dj07: It is not possible to enable/disable the offloading at the group
>> level. This makes sense to me but it would be great to explain the
>> rationale for it in the KIP.
>
> Thinking about it, there's nothing stopping us from configuring offloading
> at the group level. In fact it might be desirable for some users to disable
> offloading at the group coordinator level to keep rebalances fast and only
> enable it for problematic large groups. I've added a group-level override
> to the KIP.
>
> On Tue, Jan 20, 2026 at 1:38 PM Lianet Magrans <li...@apache.org> wrote:
>
>> Hi Sean, thanks for the KIP.
>>
>> LM1: About group.initial.rebalance.delay.ms, I expect the interaction
>> with the interval is just as described for the streams initial delay and
>> interval, correct? Should we clarify that in the KIP (it only mentions the
>> streams case)
>>
>> LM2: The KIP refers to batching assignment re-calculations triggered by
>> member subscriptions changes, but I expect the batching mechanism applies
>> the same when the assignment re-calculation is triggered by metadata
>> changes (i.e topic/partition created or deleted), without any HB changing
>> subscriptions. Is my understanding correct?
>>
>> LM3: About this section: "*When there is an in-flight assignor run for
>> the group, there is no new target assignment. We will trigger the next
>> assignor run on a future heartbeat.*". I expect that the next assignor
>> run will be triggered on the next HB from this or from any other member of
>> the group, received after the interval expires (without the members
>> re-sending the subscription change). Is my expectation correct? If so,
>> it may be worth clarifying in the KIP to avoid confusion with client-side
>> implementations.
>>
>> Thanks!
>> Lianet
>>
>>
>>
>> On Tue, Jan 13, 2026 at 1:23 AM Sean Quah via dev <de...@kafka.apache.org>
>> wrote:
>>
>>> sq01: We also have to update the SyncGroup request handling to only
>>> return
>>> REBALANCE_IN_PROGRESS when the member's epoch is behind the target
>>> assignment epoch, not the group epoch. Thanks to Dongnuo for pointing
>>> this
>>> out.
>>>
>>> On Thu, Jan 8, 2026 at 5:40 PM Dongnuo Lyu via dev <dev@kafka.apache.org
>>> >
>>> wrote:
>>>
>>> > Hi Sean, thanks for the KIP! I have a few questions as follows.
>>> >
>>> > dl01: Could we mention the handling when the group metadata or topic
>>> > partition metadata is changed or deleted during the async assignor run?
>>> >
>>> > dl02: This might be a question for the overall coordinator executor -
>>> do we
>>> > have plans to apply an explicit size limit to the executor queue? If
>>> many
>>> > groups trigger offloaded assignments simultaneously, should we apply
>>> some
>>> > backpressure for protection?
>>> >
>>> > Also resonate with dj05, for small groups default `
>>> > min.assignor.interval.ms`
>>> > to 5s might not be necessary, so not sure if we want to make the batch
>>> > assignment default. Or it might be good to have a per group enablement.
>>> >
>>> > Thanks
>>> > Dongnuo
>>> >
>>>
>>

Re: [DISCUSS] KIP-1263: Group Coordinator Assignment Batching and Offload

[Sean Quah via dev <de...@kafka.apache.org>]

Hi all, thanks for the feedback so far.

dj01: In the proposed changes section, you state that the timestamp of the
> last assignment is not persisted. How do you plan to bookkeep it if it is
> not stored with the assignment? Intuitively, I would add a timestamp to the
> assignment record.

Thinking about it, it's easier to add it to the assignment record. I will
update the KIP. One thing to note is that the timestamp will be subject to
rollbacks when writing to the log fails, so we can allow extra assignment
runs when that happens.

dj02: I wonder whether we should also add a "thread idle ratio" metric for
> the group coordinator executor. What do you think?

I think it could be useful so I've added it to the KIP. The implementation
will have to be different to the event processor, since we currently use an
ExecutorService.

dj03: If the executor is not used by the share coordinator, it should not
> expose any metrics about it. Is it possible to remove them?

I've removed them from the KIP. We can add a parameter to the coordinator
metrics class to control whether they are visible.

dj04: Is having one group coordinator executor thread sufficient by default
> for common workloads?

Yes and no. I expect it will be very difficult to overload an entire
thread, ie. submit work faster than it can complete it. But updating the
default to two threads could be good for reducing delays due to
simultaneous assignor runs. I've raised the default to 2 threads.

dj05: It seems you propose enabling the minimum assignor interval with a
> default of 5 seconds. However, the offloading is not enabled by default. Is
> the first one enough to guarantee the stability of the group coordinator?
> How do you foresee enabling the second one in the future? It would be great
> if you could address this in the KIP. We need a clear motivation for
> changing the default behavior and a plan for the future.

I initially thought that offloading would increase rebalance times by 1
heartbeat and so didn't propose turning it on by default. But after some
more thinking, I believe both features will increase rebalance times by 1
heartbeat interval and the increase shouldn't stack. The minimum assignor
interval only impacts groups with more than 2 members, while offloading
only impacts groups with a single member. This is because in the other
cases, the extra delays are folded into existing revocation + heartbeat
delays. Note that share groups have no revocation so always see increased
rebalance times. I've updated the KIP to add the analysis of rebalance
times and propose turning both features on by default.

dj06: Based on its description, I wonder whether `
> consumer.min.assignor.interval.ms` should be called `
> consumer.min.assignment.interval.ms`. What do you think?

Thanks, I've renamed the config options in the KIP. What about the
assignor.offload.enable configs?

dj07: It is not possible to enable/disable the offloading at the group
> level. This makes sense to me but it would be great to explain the
> rationale for it in the KIP.

Thinking about it, there's nothing stopping us from configuring offloading
at the group level. In fact it might be desirable for some users to disable
offloading at the group coordinator level to keep rebalances fast and only
enable it for problematic large groups. I've added a group-level override
to the KIP.

On Tue, Jan 20, 2026 at 1:38 PM Lianet Magrans <li...@apache.org> wrote:

> Hi Sean, thanks for the KIP.
>
> LM1: About group.initial.rebalance.delay.ms, I expect the interaction
> with the interval is just as described for the streams initial delay and
> interval, correct? Should we clarify that in the KIP (it only mentions the
> streams case)
>
> LM2: The KIP refers to batching assignment re-calculations triggered by
> member subscriptions changes, but I expect the batching mechanism applies
> the same when the assignment re-calculation is triggered by metadata
> changes (i.e topic/partition created or deleted), without any HB changing
> subscriptions. Is my understanding correct?
>
> LM3: About this section: "*When there is an in-flight assignor run for
> the group, there is no new target assignment. We will trigger the next
> assignor run on a future heartbeat.*". I expect that the next assignor
> run will be triggered on the next HB from this or from any other member of
> the group, received after the interval expires (without the members
> re-sending the subscription change). Is my expectation correct? If so,
> it may be worth clarifying in the KIP to avoid confusion with client-side
> implementations.
>
> Thanks!
> Lianet
>
>
>
> On Tue, Jan 13, 2026 at 1:23 AM Sean Quah via dev <de...@kafka.apache.org>
> wrote:
>
>> sq01: We also have to update the SyncGroup request handling to only return
>> REBALANCE_IN_PROGRESS when the member's epoch is behind the target
>> assignment epoch, not the group epoch. Thanks to Dongnuo for pointing this
>> out.
>>
>> On Thu, Jan 8, 2026 at 5:40 PM Dongnuo Lyu via dev <de...@kafka.apache.org>
>> wrote:
>>
>> > Hi Sean, thanks for the KIP! I have a few questions as follows.
>> >
>> > dl01: Could we mention the handling when the group metadata or topic
>> > partition metadata is changed or deleted during the async assignor run?
>> >
>> > dl02: This might be a question for the overall coordinator executor -
>> do we
>> > have plans to apply an explicit size limit to the executor queue? If
>> many
>> > groups trigger offloaded assignments simultaneously, should we apply
>> some
>> > backpressure for protection?
>> >
>> > Also resonate with dj05, for small groups default `
>> > min.assignor.interval.ms`
>> > to 5s might not be necessary, so not sure if we want to make the batch
>> > assignment default. Or it might be good to have a per group enablement.
>> >
>> > Thanks
>> > Dongnuo
>> >
>>
>

Re: [DISCUSS] KIP-1263: Group Coordinator Assignment Batching and Offload

[Lianet Magrans <li...@apache.org>]

Hi Sean, thanks for the KIP.

LM1: About group.initial.rebalance.delay.ms, I expect the interaction with
the interval is just as described for the streams initial delay and
interval, correct? Should we clarify that in the KIP (it only mentions the
streams case)

LM2: The KIP refers to batching assignment re-calculations triggered by
member subscriptions changes, but I expect the batching mechanism applies
the same when the assignment re-calculation is triggered by metadata
changes (i.e topic/partition created or deleted), without any HB changing
subscriptions. Is my understanding correct?

LM3: About this section: "*When there is an in-flight assignor run for the
group, there is no new target assignment. We will trigger the next assignor
run on a future heartbeat.*". I expect that the next assignor run will be
triggered on the next HB from this or from any other member of the group,
received after the interval expires (without the members re-sending the
subscription change). Is my expectation correct? If so, it may be worth
clarifying in the KIP to avoid confusion with client-side implementations.

Thanks!
Lianet



On Tue, Jan 13, 2026 at 1:23 AM Sean Quah via dev <de...@kafka.apache.org>
wrote:

> sq01: We also have to update the SyncGroup request handling to only return
> REBALANCE_IN_PROGRESS when the member's epoch is behind the target
> assignment epoch, not the group epoch. Thanks to Dongnuo for pointing this
> out.
>
> On Thu, Jan 8, 2026 at 5:40 PM Dongnuo Lyu via dev <de...@kafka.apache.org>
> wrote:
>
> > Hi Sean, thanks for the KIP! I have a few questions as follows.
> >
> > dl01: Could we mention the handling when the group metadata or topic
> > partition metadata is changed or deleted during the async assignor run?
> >
> > dl02: This might be a question for the overall coordinator executor - do
> we
> > have plans to apply an explicit size limit to the executor queue? If many
> > groups trigger offloaded assignments simultaneously, should we apply some
> > backpressure for protection?
> >
> > Also resonate with dj05, for small groups default `
> > min.assignor.interval.ms`
> > to 5s might not be necessary, so not sure if we want to make the batch
> > assignment default. Or it might be good to have a per group enablement.
> >
> > Thanks
> > Dongnuo
> >
>

Re: [DISCUSS] KIP-1263: Group Coordinator Assignment Batching and Offload

[Sean Quah via dev <de...@kafka.apache.org>]

sq01: We also have to update the SyncGroup request handling to only return
REBALANCE_IN_PROGRESS when the member's epoch is behind the target
assignment epoch, not the group epoch. Thanks to Dongnuo for pointing this
out.

On Thu, Jan 8, 2026 at 5:40 PM Dongnuo Lyu via dev <de...@kafka.apache.org>
wrote:

> Hi Sean, thanks for the KIP! I have a few questions as follows.
>
> dl01: Could we mention the handling when the group metadata or topic
> partition metadata is changed or deleted during the async assignor run?
>
> dl02: This might be a question for the overall coordinator executor - do we
> have plans to apply an explicit size limit to the executor queue? If many
> groups trigger offloaded assignments simultaneously, should we apply some
> backpressure for protection?
>
> Also resonate with dj05, for small groups default `
> min.assignor.interval.ms`
> to 5s might not be necessary, so not sure if we want to make the batch
> assignment default. Or it might be good to have a per group enablement.
>
> Thanks
> Dongnuo
>

Re: [DISCUSS] KIP-1263: Group Coordinator Assignment Batching and Offload

[Dongnuo Lyu via dev <de...@kafka.apache.org>]

Hi Sean, thanks for the KIP! I have a few questions as follows.

dl01: Could we mention the handling when the group metadata or topic
partition metadata is changed or deleted during the async assignor run?

dl02: This might be a question for the overall coordinator executor - do we
have plans to apply an explicit size limit to the executor queue? If many
groups trigger offloaded assignments simultaneously, should we apply some
backpressure for protection?

Also resonate with dj05, for small groups default `min.assignor.interval.ms`
to 5s might not be necessary, so not sure if we want to make the batch
assignment default. Or it might be good to have a per group enablement.

Thanks
Dongnuo

Re: [DISCUSS] KIP-1263: Group Coordinator Assignment Batching and Offload

[David Jacot via dev <de...@kafka.apache.org>]

Hi Sean,

Thanks for the KIP! I have a some comments and questions:

dj01: In the proposed changes section, you state that the timestamp of the
last assignment is not persisted. How do you plan to bookkeep it if it is
not stored with the assignment? Intuitively, I would add a timestamp to the
assignment record.
dj02: I wonder whether we should also add a "thread idle ratio" metric for
the group coordinator executor. What do you think?
dj03: If the executor is not used by the share coordinator, it should not
expose any metrics about it. Is it possible to remove them?
dj04: Is having one group coordinator executor thread sufficient by default
for common workloads?
dj05: It seems you propose enabling the minimum assignor interval with a
default of 5 seconds. However, the offloading is not enabled by default. Is
the first one enough to guarantee the stability of the group coordinator?
How do you foresee enabling the second one in the future? It would be great
if you could address this in the KIP. We need a clear motivation for
changing the default behavior and a plan for the future.
dj06: Based on its description, I wonder whether `
consumer.min.assignor.interval.ms` should be called `
consumer.min.assignment.interval.ms`. What do you think?
dj07: It is not possible to enable/disable the offloading at the group
level. This makes sense to me but it would be great to explain the
rationale for it in the KIP.

Cheers,
David


On Tue, Jan 6, 2026 at 4:23 PM Sean Quah via dev <de...@kafka.apache.org>
wrote:

> Hi all,
>
> I would like to start a discussion for KIP-1263. Under the new consumer,
> share and streams protocols (KIP-848, KIP-932, KIP-1071) we calculate
> assignments on the broker whereas previously we used to calculate
> assignments on clients under the old classic protocol. The assignment
> calculation can get expensive for large groups and impact group coordinator
> performance. The KIP aims to reduce the impact of slow assignor runs
> through two methods: enforcing a minimum interval between assignment
> updates and moving the assignment calculation to a separate thread pool.
>
> KIP:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1263%3A+Group+Coordinator+Assignment+Batching+and+Offload
>
> Best regards,
> Sean
>