You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Levani Kokhreidze <le...@gmail.com> on 2025/11/11 12:59:36 UTC
More advanced broker side consumer group management
Hi all,
I'd like to get some feedback from the community on the following idea.
Currently, to safely perform a consumer group offset reset, the consumer group must be inactive. In practice, for a production service, this means the entire service must be stopped to perform the reset. Depending on the service's criticality, this isn't always possible.
The alternative is to build custom tooling for our Kafka clients that allows them to be shut down on request. However, developing this client-side tooling is expensive, and that cost multiplies when you need to support multiple programming languages.
I understand why resetting offsets on an active consumer group is problematic. This leads me to wonder: what is the community's opinion on developing more advanced broker-side management for the consumer groups?
In short, what if we could use the AdminClient to mark a specific consumer group as "inactive"? This state would then enable us to perform an offset reset safely, without ever needing to make changes or take actions on the client side.
I appreciate that "marking a consumer group as inactive" is an abstract term, and there are many nuances and details to think through. At this point, I'm just looking for super-early feedback from more experienced Kafka developers on whether this general idea makes sense.
If it does, my team and I would be very interested in helping to deliver this feature. We can start working on a KIP to share with the community. We would also appreciate any early pointers on how to approach this if you think it's a viable idea.
Thanks,
Levani
Re: More advanced broker side consumer group management
Posted by Lucas Brutschy <lb...@confluent.io.INVALID>.
Hi,
I am also interested in improving the Kafka Streams reset
functionality based on KIP-1071. Pausing partitions sounds like a good
approach to me, and could be implemented similarly in KIP-848 and
KIP-1071. For KS, this would be enough to reset stateless topologies.
Stateful topologies need an additional function to reset the local
state, but this could be added as a follow-up KIP. Looking forward to
reading the KIP!
Cheers,
Lucas
On Wed, Nov 19, 2025 at 5:22 PM David Jacot <dj...@confluent.io.invalid> wrote:
>
> Thanks, Levani!
>
> On Wed, Nov 19, 2025 at 4:53 PM Levani Kokhreidze <le...@gmail.com>
> wrote:
>
> > Hi David,
> >
> > Thank you for your reply and sharing your thoughts.
> >
> > We were already considering that this functionality would be limited to a
> > new Consumer Group (CG) protocol.
> >
> > I also agree that using a separate API for pausing instead of delegating
> > everything to the OffsetCommit API seems like a more solid approach.
> >
> > We will start working on the KIP and initiate the discussion thread
> > shortly.
> >
> > Best
> >
> > Levani
> >
> >
> > > On 19. Nov 2025, at 14:59, David Jacot <dj...@confluent.io.INVALID>
> > wrote:
> > >
> > > Hi Levani,
> > >
> > > Thanks for sharing your thoughts! Coincidentally, I was thinking about
> > > something similar. I agree that requiring the entire group to shut down
> > to
> > > reset offsets is an annoying limitation—especially when you only want to
> > > reset the offsets of a few partitions that are stuck for whatever reason.
> > > With the new consumer rebalance protocol, I think we have an opportunity
> > to
> > > improve this since the server is now in charge of the assignment. I don’t
> > > think we can do much when the classic protocol is used, though.
> > >
> > > Building on your idea, I would suggest adding the ability to pause
> > > partitions within the group. When a partition is paused, the group
> > > coordinator would unassign it and keep it aside. We could then relax the
> > > OffsetCommit API checks to allow committing offsets for paused
> > partitions.
> > > This part is actually quite easy to implement. However, we should
> > carefully
> > > consider the UX (both API and tooling).
> > >
> > > An alternative design would skip the pause step and delegate everything
> > to
> > > the OffsetCommit API. I’m not a fan of this approach because it feels
> > > error-prone—it becomes too easy to reset offsets by mistake. Requiring an
> > > explicit pause makes the process clearer and safer.
> > >
> > > If you want to proceed, it would be great to draft a KIP for this. If
> > > possible, we should also take Streams and KIP-1071 into account.
> > >
> > > Best,
> > > David
> > >
> > > On Tue, Nov 11, 2025 at 3:23 PM Levani Kokhreidze <
> > levani.codes@gmail.com>
> > > wrote:
> > >
> > >> Hi Andrew,
> > >>
> > >> Thanks for your thoughts! To be clear, at the moment our thinking is to
> > >> enable stopping/pausing consumer group from broker side via the Admin
> > API.
> > >> Which in return makes consumer group inactive.
> > >> After that one could use the existing consumer group offset reset
> > >> functionality that already exists.
> > >>
> > >> Hope this makes sense.
> > >>
> > >> Best,
> > >> Levani
> > >>
> > >>> On 11. Nov 2025, at 16:04, Andrew Mills <am...@gmail.com> wrote:
> > >>>
> > >>> I could see a situation where you pass a status to Kafka for a specific
> > >>> Consumer Group. That status could be resetLatest or resetEarliest and
> > >> Kafka
> > >>> handles the signal similar to a rebalance but sets the offset based on
> > >> the
> > >>> earliest/latest. I think anything more, like passing a map of
> > partitions
> > >>> and offsets, could over complicate things.
> > >>>
> > >>> -- Andrew
> > >>>
> > >>> On Tue, Nov 11, 2025, 6:59 AM Levani Kokhreidze <
> > levani.codes@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> Hi all,
> > >>>>
> > >>>>
> > >>>>
> > >>>> I'd like to get some feedback from the community on the following
> > idea.
> > >>>>
> > >>>> Currently, to safely perform a consumer group offset reset, the
> > consumer
> > >>>> group must be inactive. In practice, for a production service, this
> > >> means
> > >>>> the entire service must be stopped to perform the reset. Depending on
> > >> the
> > >>>> service's criticality, this isn't always possible.
> > >>>>
> > >>>> The alternative is to build custom tooling for our Kafka clients that
> > >>>> allows them to be shut down on request. However, developing this
> > >>>> client-side tooling is expensive, and that cost multiplies when you
> > >> need to
> > >>>> support multiple programming languages.
> > >>>>
> > >>>> I understand why resetting offsets on an active consumer group is
> > >>>> problematic. This leads me to wonder: what is the community's opinion
> > on
> > >>>> developing more advanced broker-side management for the consumer
> > groups?
> > >>>>
> > >>>> In short, what if we could use the AdminClient to mark a specific
> > >> consumer
> > >>>> group as "inactive"? This state would then enable us to perform an
> > >> offset
> > >>>> reset safely, without ever needing to make changes or take actions on
> > >> the
> > >>>> client side.
> > >>>>
> > >>>> I appreciate that "marking a consumer group as inactive" is an
> > abstract
> > >>>> term, and there are many nuances and details to think through. At this
> > >>>> point, I'm just looking for super-early feedback from more experienced
> > >>>> Kafka developers on whether this general idea makes sense.
> > >>>>
> > >>>> If it does, my team and I would be very interested in helping to
> > deliver
> > >>>> this feature. We can start working on a KIP to share with the
> > >> community. We
> > >>>> would also appreciate any early pointers on how to approach this if
> > you
> > >>>> think it's a viable idea.
> > >>>>
> > >>>>
> > >>>>
> > >>>> Thanks,
> > >>>>
> > >>>> Levani
> > >>
> > >>
> >
> >
Re: More advanced broker side consumer group management
Posted by David Jacot <dj...@confluent.io.INVALID>.
Thanks, Levani!
On Wed, Nov 19, 2025 at 4:53 PM Levani Kokhreidze <le...@gmail.com>
wrote:
> Hi David,
>
> Thank you for your reply and sharing your thoughts.
>
> We were already considering that this functionality would be limited to a
> new Consumer Group (CG) protocol.
>
> I also agree that using a separate API for pausing instead of delegating
> everything to the OffsetCommit API seems like a more solid approach.
>
> We will start working on the KIP and initiate the discussion thread
> shortly.
>
> Best
>
> Levani
>
>
> > On 19. Nov 2025, at 14:59, David Jacot <dj...@confluent.io.INVALID>
> wrote:
> >
> > Hi Levani,
> >
> > Thanks for sharing your thoughts! Coincidentally, I was thinking about
> > something similar. I agree that requiring the entire group to shut down
> to
> > reset offsets is an annoying limitation—especially when you only want to
> > reset the offsets of a few partitions that are stuck for whatever reason.
> > With the new consumer rebalance protocol, I think we have an opportunity
> to
> > improve this since the server is now in charge of the assignment. I don’t
> > think we can do much when the classic protocol is used, though.
> >
> > Building on your idea, I would suggest adding the ability to pause
> > partitions within the group. When a partition is paused, the group
> > coordinator would unassign it and keep it aside. We could then relax the
> > OffsetCommit API checks to allow committing offsets for paused
> partitions.
> > This part is actually quite easy to implement. However, we should
> carefully
> > consider the UX (both API and tooling).
> >
> > An alternative design would skip the pause step and delegate everything
> to
> > the OffsetCommit API. I’m not a fan of this approach because it feels
> > error-prone—it becomes too easy to reset offsets by mistake. Requiring an
> > explicit pause makes the process clearer and safer.
> >
> > If you want to proceed, it would be great to draft a KIP for this. If
> > possible, we should also take Streams and KIP-1071 into account.
> >
> > Best,
> > David
> >
> > On Tue, Nov 11, 2025 at 3:23 PM Levani Kokhreidze <
> levani.codes@gmail.com>
> > wrote:
> >
> >> Hi Andrew,
> >>
> >> Thanks for your thoughts! To be clear, at the moment our thinking is to
> >> enable stopping/pausing consumer group from broker side via the Admin
> API.
> >> Which in return makes consumer group inactive.
> >> After that one could use the existing consumer group offset reset
> >> functionality that already exists.
> >>
> >> Hope this makes sense.
> >>
> >> Best,
> >> Levani
> >>
> >>> On 11. Nov 2025, at 16:04, Andrew Mills <am...@gmail.com> wrote:
> >>>
> >>> I could see a situation where you pass a status to Kafka for a specific
> >>> Consumer Group. That status could be resetLatest or resetEarliest and
> >> Kafka
> >>> handles the signal similar to a rebalance but sets the offset based on
> >> the
> >>> earliest/latest. I think anything more, like passing a map of
> partitions
> >>> and offsets, could over complicate things.
> >>>
> >>> -- Andrew
> >>>
> >>> On Tue, Nov 11, 2025, 6:59 AM Levani Kokhreidze <
> levani.codes@gmail.com>
> >>> wrote:
> >>>
> >>>> Hi all,
> >>>>
> >>>>
> >>>>
> >>>> I'd like to get some feedback from the community on the following
> idea.
> >>>>
> >>>> Currently, to safely perform a consumer group offset reset, the
> consumer
> >>>> group must be inactive. In practice, for a production service, this
> >> means
> >>>> the entire service must be stopped to perform the reset. Depending on
> >> the
> >>>> service's criticality, this isn't always possible.
> >>>>
> >>>> The alternative is to build custom tooling for our Kafka clients that
> >>>> allows them to be shut down on request. However, developing this
> >>>> client-side tooling is expensive, and that cost multiplies when you
> >> need to
> >>>> support multiple programming languages.
> >>>>
> >>>> I understand why resetting offsets on an active consumer group is
> >>>> problematic. This leads me to wonder: what is the community's opinion
> on
> >>>> developing more advanced broker-side management for the consumer
> groups?
> >>>>
> >>>> In short, what if we could use the AdminClient to mark a specific
> >> consumer
> >>>> group as "inactive"? This state would then enable us to perform an
> >> offset
> >>>> reset safely, without ever needing to make changes or take actions on
> >> the
> >>>> client side.
> >>>>
> >>>> I appreciate that "marking a consumer group as inactive" is an
> abstract
> >>>> term, and there are many nuances and details to think through. At this
> >>>> point, I'm just looking for super-early feedback from more experienced
> >>>> Kafka developers on whether this general idea makes sense.
> >>>>
> >>>> If it does, my team and I would be very interested in helping to
> deliver
> >>>> this feature. We can start working on a KIP to share with the
> >> community. We
> >>>> would also appreciate any early pointers on how to approach this if
> you
> >>>> think it's a viable idea.
> >>>>
> >>>>
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Levani
> >>
> >>
>
>
Re: More advanced broker side consumer group management
Posted by Levani Kokhreidze <le...@gmail.com>.
Hi David,
Thank you for your reply and sharing your thoughts.
We were already considering that this functionality would be limited to a new Consumer Group (CG) protocol.
I also agree that using a separate API for pausing instead of delegating everything to the OffsetCommit API seems like a more solid approach.
We will start working on the KIP and initiate the discussion thread shortly.
Best
Levani
> On 19. Nov 2025, at 14:59, David Jacot <dj...@confluent.io.INVALID> wrote:
>
> Hi Levani,
>
> Thanks for sharing your thoughts! Coincidentally, I was thinking about
> something similar. I agree that requiring the entire group to shut down to
> reset offsets is an annoying limitation—especially when you only want to
> reset the offsets of a few partitions that are stuck for whatever reason.
> With the new consumer rebalance protocol, I think we have an opportunity to
> improve this since the server is now in charge of the assignment. I don’t
> think we can do much when the classic protocol is used, though.
>
> Building on your idea, I would suggest adding the ability to pause
> partitions within the group. When a partition is paused, the group
> coordinator would unassign it and keep it aside. We could then relax the
> OffsetCommit API checks to allow committing offsets for paused partitions.
> This part is actually quite easy to implement. However, we should carefully
> consider the UX (both API and tooling).
>
> An alternative design would skip the pause step and delegate everything to
> the OffsetCommit API. I’m not a fan of this approach because it feels
> error-prone—it becomes too easy to reset offsets by mistake. Requiring an
> explicit pause makes the process clearer and safer.
>
> If you want to proceed, it would be great to draft a KIP for this. If
> possible, we should also take Streams and KIP-1071 into account.
>
> Best,
> David
>
> On Tue, Nov 11, 2025 at 3:23 PM Levani Kokhreidze <le...@gmail.com>
> wrote:
>
>> Hi Andrew,
>>
>> Thanks for your thoughts! To be clear, at the moment our thinking is to
>> enable stopping/pausing consumer group from broker side via the Admin API.
>> Which in return makes consumer group inactive.
>> After that one could use the existing consumer group offset reset
>> functionality that already exists.
>>
>> Hope this makes sense.
>>
>> Best,
>> Levani
>>
>>> On 11. Nov 2025, at 16:04, Andrew Mills <am...@gmail.com> wrote:
>>>
>>> I could see a situation where you pass a status to Kafka for a specific
>>> Consumer Group. That status could be resetLatest or resetEarliest and
>> Kafka
>>> handles the signal similar to a rebalance but sets the offset based on
>> the
>>> earliest/latest. I think anything more, like passing a map of partitions
>>> and offsets, could over complicate things.
>>>
>>> -- Andrew
>>>
>>> On Tue, Nov 11, 2025, 6:59 AM Levani Kokhreidze <le...@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>>
>>>>
>>>> I'd like to get some feedback from the community on the following idea.
>>>>
>>>> Currently, to safely perform a consumer group offset reset, the consumer
>>>> group must be inactive. In practice, for a production service, this
>> means
>>>> the entire service must be stopped to perform the reset. Depending on
>> the
>>>> service's criticality, this isn't always possible.
>>>>
>>>> The alternative is to build custom tooling for our Kafka clients that
>>>> allows them to be shut down on request. However, developing this
>>>> client-side tooling is expensive, and that cost multiplies when you
>> need to
>>>> support multiple programming languages.
>>>>
>>>> I understand why resetting offsets on an active consumer group is
>>>> problematic. This leads me to wonder: what is the community's opinion on
>>>> developing more advanced broker-side management for the consumer groups?
>>>>
>>>> In short, what if we could use the AdminClient to mark a specific
>> consumer
>>>> group as "inactive"? This state would then enable us to perform an
>> offset
>>>> reset safely, without ever needing to make changes or take actions on
>> the
>>>> client side.
>>>>
>>>> I appreciate that "marking a consumer group as inactive" is an abstract
>>>> term, and there are many nuances and details to think through. At this
>>>> point, I'm just looking for super-early feedback from more experienced
>>>> Kafka developers on whether this general idea makes sense.
>>>>
>>>> If it does, my team and I would be very interested in helping to deliver
>>>> this feature. We can start working on a KIP to share with the
>> community. We
>>>> would also appreciate any early pointers on how to approach this if you
>>>> think it's a viable idea.
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Levani
>>
>>
Re: More advanced broker side consumer group management
Posted by David Jacot <dj...@confluent.io.INVALID>.
Hi Levani,
Thanks for sharing your thoughts! Coincidentally, I was thinking about
something similar. I agree that requiring the entire group to shut down to
reset offsets is an annoying limitation—especially when you only want to
reset the offsets of a few partitions that are stuck for whatever reason.
With the new consumer rebalance protocol, I think we have an opportunity to
improve this since the server is now in charge of the assignment. I don’t
think we can do much when the classic protocol is used, though.
Building on your idea, I would suggest adding the ability to pause
partitions within the group. When a partition is paused, the group
coordinator would unassign it and keep it aside. We could then relax the
OffsetCommit API checks to allow committing offsets for paused partitions.
This part is actually quite easy to implement. However, we should carefully
consider the UX (both API and tooling).
An alternative design would skip the pause step and delegate everything to
the OffsetCommit API. I’m not a fan of this approach because it feels
error-prone—it becomes too easy to reset offsets by mistake. Requiring an
explicit pause makes the process clearer and safer.
If you want to proceed, it would be great to draft a KIP for this. If
possible, we should also take Streams and KIP-1071 into account.
Best,
David
On Tue, Nov 11, 2025 at 3:23 PM Levani Kokhreidze <le...@gmail.com>
wrote:
> Hi Andrew,
>
> Thanks for your thoughts! To be clear, at the moment our thinking is to
> enable stopping/pausing consumer group from broker side via the Admin API.
> Which in return makes consumer group inactive.
> After that one could use the existing consumer group offset reset
> functionality that already exists.
>
> Hope this makes sense.
>
> Best,
> Levani
>
> > On 11. Nov 2025, at 16:04, Andrew Mills <am...@gmail.com> wrote:
> >
> > I could see a situation where you pass a status to Kafka for a specific
> > Consumer Group. That status could be resetLatest or resetEarliest and
> Kafka
> > handles the signal similar to a rebalance but sets the offset based on
> the
> > earliest/latest. I think anything more, like passing a map of partitions
> > and offsets, could over complicate things.
> >
> > -- Andrew
> >
> > On Tue, Nov 11, 2025, 6:59 AM Levani Kokhreidze <le...@gmail.com>
> > wrote:
> >
> >> Hi all,
> >>
> >>
> >>
> >> I'd like to get some feedback from the community on the following idea.
> >>
> >> Currently, to safely perform a consumer group offset reset, the consumer
> >> group must be inactive. In practice, for a production service, this
> means
> >> the entire service must be stopped to perform the reset. Depending on
> the
> >> service's criticality, this isn't always possible.
> >>
> >> The alternative is to build custom tooling for our Kafka clients that
> >> allows them to be shut down on request. However, developing this
> >> client-side tooling is expensive, and that cost multiplies when you
> need to
> >> support multiple programming languages.
> >>
> >> I understand why resetting offsets on an active consumer group is
> >> problematic. This leads me to wonder: what is the community's opinion on
> >> developing more advanced broker-side management for the consumer groups?
> >>
> >> In short, what if we could use the AdminClient to mark a specific
> consumer
> >> group as "inactive"? This state would then enable us to perform an
> offset
> >> reset safely, without ever needing to make changes or take actions on
> the
> >> client side.
> >>
> >> I appreciate that "marking a consumer group as inactive" is an abstract
> >> term, and there are many nuances and details to think through. At this
> >> point, I'm just looking for super-early feedback from more experienced
> >> Kafka developers on whether this general idea makes sense.
> >>
> >> If it does, my team and I would be very interested in helping to deliver
> >> this feature. We can start working on a KIP to share with the
> community. We
> >> would also appreciate any early pointers on how to approach this if you
> >> think it's a viable idea.
> >>
> >>
> >>
> >> Thanks,
> >>
> >> Levani
>
>
Re: More advanced broker side consumer group management
Posted by Levani Kokhreidze <le...@gmail.com>.
Hi Andrew,
Thanks for your thoughts! To be clear, at the moment our thinking is to enable stopping/pausing consumer group from broker side via the Admin API.
Which in return makes consumer group inactive.
After that one could use the existing consumer group offset reset functionality that already exists.
Hope this makes sense.
Best,
Levani
> On 11. Nov 2025, at 16:04, Andrew Mills <am...@gmail.com> wrote:
>
> I could see a situation where you pass a status to Kafka for a specific
> Consumer Group. That status could be resetLatest or resetEarliest and Kafka
> handles the signal similar to a rebalance but sets the offset based on the
> earliest/latest. I think anything more, like passing a map of partitions
> and offsets, could over complicate things.
>
> -- Andrew
>
> On Tue, Nov 11, 2025, 6:59 AM Levani Kokhreidze <le...@gmail.com>
> wrote:
>
>> Hi all,
>>
>>
>>
>> I'd like to get some feedback from the community on the following idea.
>>
>> Currently, to safely perform a consumer group offset reset, the consumer
>> group must be inactive. In practice, for a production service, this means
>> the entire service must be stopped to perform the reset. Depending on the
>> service's criticality, this isn't always possible.
>>
>> The alternative is to build custom tooling for our Kafka clients that
>> allows them to be shut down on request. However, developing this
>> client-side tooling is expensive, and that cost multiplies when you need to
>> support multiple programming languages.
>>
>> I understand why resetting offsets on an active consumer group is
>> problematic. This leads me to wonder: what is the community's opinion on
>> developing more advanced broker-side management for the consumer groups?
>>
>> In short, what if we could use the AdminClient to mark a specific consumer
>> group as "inactive"? This state would then enable us to perform an offset
>> reset safely, without ever needing to make changes or take actions on the
>> client side.
>>
>> I appreciate that "marking a consumer group as inactive" is an abstract
>> term, and there are many nuances and details to think through. At this
>> point, I'm just looking for super-early feedback from more experienced
>> Kafka developers on whether this general idea makes sense.
>>
>> If it does, my team and I would be very interested in helping to deliver
>> this feature. We can start working on a KIP to share with the community. We
>> would also appreciate any early pointers on how to approach this if you
>> think it's a viable idea.
>>
>>
>>
>> Thanks,
>>
>> Levani
Re: More advanced broker side consumer group management
Posted by Andrew Mills <am...@gmail.com>.
I could see a situation where you pass a status to Kafka for a specific
Consumer Group. That status could be resetLatest or resetEarliest and Kafka
handles the signal similar to a rebalance but sets the offset based on the
earliest/latest. I think anything more, like passing a map of partitions
and offsets, could over complicate things.
-- Andrew
On Tue, Nov 11, 2025, 6:59 AM Levani Kokhreidze <le...@gmail.com>
wrote:
> Hi all,
>
>
>
> I'd like to get some feedback from the community on the following idea.
>
> Currently, to safely perform a consumer group offset reset, the consumer
> group must be inactive. In practice, for a production service, this means
> the entire service must be stopped to perform the reset. Depending on the
> service's criticality, this isn't always possible.
>
> The alternative is to build custom tooling for our Kafka clients that
> allows them to be shut down on request. However, developing this
> client-side tooling is expensive, and that cost multiplies when you need to
> support multiple programming languages.
>
> I understand why resetting offsets on an active consumer group is
> problematic. This leads me to wonder: what is the community's opinion on
> developing more advanced broker-side management for the consumer groups?
>
> In short, what if we could use the AdminClient to mark a specific consumer
> group as "inactive"? This state would then enable us to perform an offset
> reset safely, without ever needing to make changes or take actions on the
> client side.
>
> I appreciate that "marking a consumer group as inactive" is an abstract
> term, and there are many nuances and details to think through. At this
> point, I'm just looking for super-early feedback from more experienced
> Kafka developers on whether this general idea makes sense.
>
> If it does, my team and I would be very interested in helping to deliver
> this feature. We can start working on a KIP to share with the community. We
> would also appreciate any early pointers on how to approach this if you
> think it's a viable idea.
>
>
>
> Thanks,
>
> Levani