You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Andrew Schofield <sc...@apache.org> on 2026/08/12 13:39:08 UTC

Re: [DISCUSS] KIP-1359: Improve usability of resetting group offsets

Hi Aditya,
Thanks for your reply. Sorry for the delay.

AK1: One of the advantages of the file is that it's possible to export from one group type and import into another. You can also export offsets for multiple groups, and this changes the file format (4 columns instead of 3). Sadly, this is all existing behaviour that I only learnt by trial and error. Personally, I think that export-import is a relatively infrequent operation, and the KIP makes enough of a step forward. I wonder if you could elaborate a bit more on your idea. Feel free to push back if you disagree.

AK2: The KIP does not mention this and I wasn't quite sure where to add it. You are correct. Should we improve the javadoc for the admin client?

AK3: Yes, Admin is better. KIP updated.

AK4: The DescribeGroupsGeneric RPC is intended as a way of getting a minimal set of group information for groups of any group types. It is used in Admin.describeGroups and this would be the initial Admin call from bin/kafka-XXXX-groups.sh when exporting or resetting offsets. It gives a way to check the group type and existence unambiguously. By contrast, if you use Admin.describeXXXGroups() for an existing group ID of the wrong type, the error code is GROUP_ID_NOT_FOUND.

AK4.1: I didn't include the members intentionally. This RPC is agnostic to group type. I think the common subset of group attributes is group ID, group type, group state and coordinator. If you dig deeper, you find that the group types differ. For example, assignments in consumer groups and share groups are partitions, while in streams groups they are tasks. I expect we'll add connect groups one day, and until we have the KIP, I wouldn't want to assume the details of membership.

AK4.2: This is following existing convention I'm afraid and I want to be consistent. If the broker does not send back a list of ACL operations, then this will be null. This is different than an empty set which means "you are not authorized", as opposed to "you don't know anything about authorized operations".

Thanks,
Andrew

On 2026/07/27 05:45:27 Aditya Kousik wrote:
> A small correction.
> 
> I meant to add a statement before the numbered comments.
> 
> “Since the KIP motivation revolves around usability of the script, would you be willing to consider the following:”
> 
> Also, typo on AK2. “…As long as there aren’t any active members in the group subscription”.
> 
> -Aditya
> 
> > On Jul 25, 2026, at 23:53, Aditya Kousik <ad...@icloud.com> wrote:
> > 
> > Hi Andrew,
> > 
> > Thanks for this KIP, I’ve personally faced more than one scenario where I’ve had to use the admin client to update the offsets.
> > 
> > The main uses cases teams face ( and TIL about the various options the script offers):
> > Teams migrating to a new consumer group from an existing one but don’t want to commit to either (or duration for newer clients) auto offset reset policy on the new group.
> > Teams want to seek past certain offsets in partitions because they have poison pill messages/corrupted deserialization story and they have no control over the data. In this case, they actually want latest offset strategy but most don’t even realise that it applies to streams out of range and are surprised/frustrated that the property doesn’t move their position in the queues. (I see we can do this already in the script today)
> > When I migrated our clients from 2.x directly to 4.x, out of abundance of caution, I wrote an hourly cron job that dumps the snapshot of group_id,topic,partition,offset into csv files so that if I had to revert a few affected partitions I can trim the csv down to those partitions and alter them with the admin client.
> > 
> > AK1. A single command line with args —source-group-id —target-group-id —export so that the intermediate CSV file can be skipped.
> > AK2. Minor: The KIP does not mention the alter/list consumer group offsets Admin call. alter only works as long as there are active members in the group subscription. I was bitten by this in the past; the documentation was sparse around that expectation but it does make sense.
> > AK3. KIP references AdminClient but I believe this is deprecated in favour of just Admin which is the interface that has the methods?
> > AK4. How do we leverage the describe groups RPC when using the script/admin client? I’m trying to imagine flows other than the export group offsets.
> > AK4.1 Can we list active members/offset state in a given group in addition to the GroupState? This has helped identify zombie processes to hunt down. The broker’s view would be useful here. (related to AK2)
> > AK4.2 Could we return empty set of acl operations instead of null?
> > 
> > Best,
> > Aditya
> > 
> > 
> >> On 2026/06/17 06:34:11 Andrew Schofield wrote:
> >> Hi,
> >> I’d like to start discussion on a small KIP to improve the usability of the command-line tools for resetting group offsets. When implementing KIP-1323 and reviewing the PRs, it became clear that the usability of this area of Kafka isn’t great. This KIP improves the usability of these tools.
> >> 
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1359%3A+Improve+usability+of+resetting+group+offsets
> >> 
> >> Thanks,
> >> Andrew
> >> 
> 

Re: [DISCUSS] KIP-1359: Improve usability of resetting group offsets

Posted by Aditya Kousik <ad...@icloud.com>.
Yup. LGTM! “- —topic” is going to be fun to parse out from the kube logs but I believe we can work with this. In the future if we port it to a -o option, it might break but that’s for later. When I think about it, the args for the command in the PodSpec will be a dance of hyphens: 
- —from-file
- -
- —topic

but this is YAML quirkiness and I can probably switch to JSON array [“—from-file”, “-“, “—topic”, …]

On AK2: for the list/alter group offsets documentation, looking through the KIP - I agree that that there isn’t a particular section to mention both RPCs (or the subscription state for alter). 

No other questions from me.

Thanks,
Aditya

> On Aug 13, 2026, at 05:21, Andrew Schofield <sc...@apache.org> wrote:
> Hi Aditya,
> Thanks for your reply. I'm not dismissing your concern, but I'm not entirely happy with a way forward yet.
> 
> AK1: I wonder whether we could use a pipe here to avoid the intermediate file. After all, export uses stdout as the destination, so if we could use stdin as the source. For example, I've used '-' with --from-file below to indicate stdin. It could be --from-stdin or similar too.
> 
> $ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --export-offsets --group CG1 --topic T1 --to-current | bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --reset-offsets --group CG2 --from-file - --topic T1 --execute
> 
> This would still allow resetting between different group types, which is actually the use case that got me into this, taking offsets from a consumer group to initialise a share group.
> 
> Thanks,
> Andrew
> 
> On 2026/08/12 18:12:08 Aditya Kousik wrote:
>> Hi Andrew.
>> 
>> Thanks for addressing AK2-4: lgtm
>> 
>> On AK1: I’m thinking from a user experience standpoint. The teams managing the kafka clusters and the ones using the clients diverge - but the Admin role may be exposed via self-service typically via API requests and a K8S CRD. I’m thinking of the latter case where Kafka SRE allow a thin wrapper for client teams to manage their own groups and topics and also isolating the environment where the scripts can be run. In such a case, it becomes cumbersome to fetch/write the csv file from a Pod that runs the migration ; it may not have any volume to persist it or running kubectl to scp the file is usually denied. So in such a case, the migrate offsets script could just be a single command that fetches offsets from one group and applies it to another (or several) without the middleman CSV.
>> 
>> Thanks,
>> Aditya
>> 
>>> On Aug 12, 2026, at 06:39, Andrew Schofield <sc...@apache.org> wrote:
>>> Hi Aditya,
>>> Thanks for your reply. Sorry for the delay.
>>> AK1: One of the advantages of the file is that it's possible to export from one group type and import into another. You can also export offsets for multiple groups, and this changes the file format (4 columns instead of 3). Sadly, this is all existing behaviour that I only learnt by trial and error. Personally, I think that export-import is a relatively infrequent operation, and the KIP makes enough of a step forward. I wonder if you could elaborate a bit more on your idea. Feel free to push back if you disagree.
>>> AK2: The KIP does not mention this and I wasn't quite sure where to add it. You are correct. Should we improve the javadoc for the admin client?
>>> AK3: Yes, Admin is better. KIP updated.
>>> AK4: The DescribeGroupsGeneric RPC is intended as a way of getting a minimal set of group information for groups of any group types. It is used in Admin.describeGroups and this would be the initial Admin call from bin/kafka-XXXX-groups.sh when exporting or resetting offsets. It gives a way to check the group type and existence unambiguously. By contrast, if you use Admin.describeXXXGroups() for an existing group ID of the wrong type, the error code is GROUP_ID_NOT_FOUND.
>>> AK4.1: I didn't include the members intentionally. This RPC is agnostic to group type. I think the common subset of group attributes is group ID, group type, group state and coordinator. If you dig deeper, you find that the group types differ. For example, assignments in consumer groups and share groups are partitions, while in streams groups they are tasks. I expect we'll add connect groups one day, and until we have the KIP, I wouldn't want to assume the details of membership.
>>> AK4.2: This is following existing convention I'm afraid and I want to be consistent. If the broker does not send back a list of ACL operations, then this will be null. This is different than an empty set which means "you are not authorized", as opposed to "you don't know anything about authorized operations".
>>> Thanks,
>>> Andrew
>>>> On 2026/07/27 05:45:27 Aditya Kousik wrote:
>>>> A small correction.
>>>> I meant to add a statement before the numbered comments.
>>>> “Since the KIP motivation revolves around usability of the script, would you be willing to consider the following:”
>>>> Also, typo on AK2. “…As long as there aren’t any active members in the group subscription”.
>>>> -Aditya
>>>>>> On Jul 25, 2026, at 23:53, Aditya Kousik <ad...@icloud.com> wrote:
>>>>> Hi Andrew,
>>>>> Thanks for this KIP, I’ve personally faced more than one scenario where I’ve had to use the admin client to update the offsets.
>>>>> The main uses cases teams face ( and TIL about the various options the script offers):
>>>>> Teams migrating to a new consumer group from an existing one but don’t want to commit to either (or duration for newer clients) auto offset reset policy on the new group.
>>>>> Teams want to seek past certain offsets in partitions because they have poison pill messages/corrupted deserialization story and they have no control over the data. In this case, they actually want latest offset strategy but most don’t even realise that it applies to streams out of range and are surprised/frustrated that the property doesn’t move their position in the queues. (I see we can do this already in the script today)
>>>>> When I migrated our clients from 2.x directly to 4.x, out of abundance of caution, I wrote an hourly cron job that dumps the snapshot of group_id,topic,partition,offset into csv files so that if I had to revert a few affected partitions I can trim the csv down to those partitions and alter them with the admin client.
>>>>> AK1. A single command line with args —source-group-id —target-group-id —export so that the intermediate CSV file can be skipped.
>>>>> AK2. Minor: The KIP does not mention the alter/list consumer group offsets Admin call. alter only works as long as there are active members in the group subscription. I was bitten by this in the past; the documentation was sparse around that expectation but it does make sense.
>>>>> AK3. KIP references AdminClient but I believe this is deprecated in favour of just Admin which is the interface that has the methods?
>>>>> AK4. How do we leverage the describe groups RPC when using the script/admin client? I’m trying to imagine flows other than the export group offsets.
>>>>> AK4.1 Can we list active members/offset state in a given group in addition to the GroupState? This has helped identify zombie processes to hunt down. The broker’s view would be useful here. (related to AK2)
>>>>> AK4.2 Could we return empty set of acl operations instead of null?
>>>>> Best,
>>>>> Aditya
>>>>>> On 2026/06/17 06:34:11 Andrew Schofield wrote:
>>>>>> Hi,
>>>>>> I’d like to start discussion on a small KIP to improve the usability of the command-line tools for resetting group offsets. When implementing KIP-1323 and reviewing the PRs, it became clear that the usability of this area of Kafka isn’t great. This KIP improves the usability of these tools.
>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1359%3A+Improve+usability+of+resetting+group+offsets
>>>>>> Thanks,
>>>>>> Andrew

Re: [DISCUSS] KIP-1359: Improve usability of resetting group offsets

Posted by Andrew Schofield <sc...@apache.org>.
Hi Apoorv,
Thanks for your response.

AM1: With --by-duration, the tool obtains the current time, subtracts the duration (such as 3 hours in your example), and then calculates the offset on the topic-partition for the resulting time. It then resets the offset accordingly. It does not depend upon the share.auto.offset.reset config.

Thanks,
Andrew

On 2026/08/13 23:33:40 Apoorv Mittal wrote:
> Hi Andrew,
> Thanks for the KIP. I have one question.
> 
> AM1: Since this KIP adds --by-duration to kafka-share-groups.sh while share
> groups also accept share.auto.offset.reset=by_duration:… as a group config. I
> can find share.auto.offset.reset=by_duration:PT3H (config,
> kafka-configs.sh): an init-time policy the broker applies only when no SPSO
> exists. I can set the config while there are active members but the config
> update is not honoured. It's only honoured when a new group is specified.
> Can you please clarify the expected behaviour when using
> `kafka-share-groups.sh --by-duration` reset option?
> 
> Regards,
> Apoorv Mittal
> 
> 
> On Thu, Aug 13, 2026 at 1:21 PM Andrew Schofield <sc...@apache.org>
> wrote:
> 
> > Hi Aditya,
> > Thanks for your reply. I'm not dismissing your concern, but I'm not
> > entirely happy with a way forward yet.
> >
> > AK1: I wonder whether we could use a pipe here to avoid the intermediate
> > file. After all, export uses stdout as the destination, so if we could use
> > stdin as the source. For example, I've used '-' with --from-file below to
> > indicate stdin. It could be --from-stdin or similar too.
> >
> > $ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092
> > --export-offsets --group CG1 --topic T1 --to-current |
> > bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092
> > --reset-offsets --group CG2 --from-file - --topic T1 --execute
> >
> > This would still allow resetting between different group types, which is
> > actually the use case that got me into this, taking offsets from a consumer
> > group to initialise a share group.
> >
> > Thanks,
> > Andrew
> >
> > On 2026/08/12 18:12:08 Aditya Kousik wrote:
> > > Hi Andrew.
> > >
> > > Thanks for addressing AK2-4: lgtm
> > >
> > > On AK1: I’m thinking from a user experience standpoint. The teams
> > managing the kafka clusters and the ones using the clients diverge - but
> > the Admin role may be exposed via self-service typically via API requests
> > and a K8S CRD. I’m thinking of the latter case where Kafka SRE allow a thin
> > wrapper for client teams to manage their own groups and topics and also
> > isolating the environment where the scripts can be run. In such a case, it
> > becomes cumbersome to fetch/write the csv file from a Pod that runs the
> > migration ; it may not have any volume to persist it or running kubectl to
> > scp the file is usually denied. So in such a case, the migrate offsets
> > script could just be a single command that fetches offsets from one group
> > and applies it to another (or several) without the middleman CSV.
> > >
> > > Thanks,
> > > Aditya
> > >
> > > > On Aug 12, 2026, at 06:39, Andrew Schofield <sc...@apache.org>
> > wrote:
> > > >
> > > > Hi Aditya,
> > > > Thanks for your reply. Sorry for the delay.
> > > >
> > > > AK1: One of the advantages of the file is that it's possible to export
> > from one group type and import into another. You can also export offsets
> > for multiple groups, and this changes the file format (4 columns instead of
> > 3). Sadly, this is all existing behaviour that I only learnt by trial and
> > error. Personally, I think that export-import is a relatively infrequent
> > operation, and the KIP makes enough of a step forward. I wonder if you
> > could elaborate a bit more on your idea. Feel free to push back if you
> > disagree.
> > > >
> > > > AK2: The KIP does not mention this and I wasn't quite sure where to
> > add it. You are correct. Should we improve the javadoc for the admin client?
> > > >
> > > > AK3: Yes, Admin is better. KIP updated.
> > > >
> > > > AK4: The DescribeGroupsGeneric RPC is intended as a way of getting a
> > minimal set of group information for groups of any group types. It is used
> > in Admin.describeGroups and this would be the initial Admin call from
> > bin/kafka-XXXX-groups.sh when exporting or resetting offsets. It gives a
> > way to check the group type and existence unambiguously. By contrast, if
> > you use Admin.describeXXXGroups() for an existing group ID of the wrong
> > type, the error code is GROUP_ID_NOT_FOUND.
> > > >
> > > > AK4.1: I didn't include the members intentionally. This RPC is
> > agnostic to group type. I think the common subset of group attributes is
> > group ID, group type, group state and coordinator. If you dig deeper, you
> > find that the group types differ. For example, assignments in consumer
> > groups and share groups are partitions, while in streams groups they are
> > tasks. I expect we'll add connect groups one day, and until we have the
> > KIP, I wouldn't want to assume the details of membership.
> > > >
> > > > AK4.2: This is following existing convention I'm afraid and I want to
> > be consistent. If the broker does not send back a list of ACL operations,
> > then this will be null. This is different than an empty set which means
> > "you are not authorized", as opposed to "you don't know anything about
> > authorized operations".
> > > >
> > > > Thanks,
> > > > Andrew
> > > >
> > > >> On 2026/07/27 05:45:27 Aditya Kousik wrote:
> > > >> A small correction.
> > > >>
> > > >> I meant to add a statement before the numbered comments.
> > > >>
> > > >> “Since the KIP motivation revolves around usability of the script,
> > would you be willing to consider the following:”
> > > >>
> > > >> Also, typo on AK2. “…As long as there aren’t any active members in
> > the group subscription”.
> > > >>
> > > >> -Aditya
> > > >>
> > > >>>> On Jul 25, 2026, at 23:53, Aditya Kousik <ad...@icloud.com>
> > wrote:
> > > >>>
> > > >>> Hi Andrew,
> > > >>>
> > > >>> Thanks for this KIP, I’ve personally faced more than one scenario
> > where I’ve had to use the admin client to update the offsets.
> > > >>>
> > > >>> The main uses cases teams face ( and TIL about the various options
> > the script offers):
> > > >>> Teams migrating to a new consumer group from an existing one but
> > don’t want to commit to either (or duration for newer clients) auto offset
> > reset policy on the new group.
> > > >>> Teams want to seek past certain offsets in partitions because they
> > have poison pill messages/corrupted deserialization story and they have no
> > control over the data. In this case, they actually want latest offset
> > strategy but most don’t even realise that it applies to streams out of
> > range and are surprised/frustrated that the property doesn’t move their
> > position in the queues. (I see we can do this already in the script today)
> > > >>> When I migrated our clients from 2.x directly to 4.x, out of
> > abundance of caution, I wrote an hourly cron job that dumps the snapshot of
> > group_id,topic,partition,offset into csv files so that if I had to revert a
> > few affected partitions I can trim the csv down to those partitions and
> > alter them with the admin client.
> > > >>>
> > > >>> AK1. A single command line with args —source-group-id
> > —target-group-id —export so that the intermediate CSV file can be skipped.
> > > >>> AK2. Minor: The KIP does not mention the alter/list consumer group
> > offsets Admin call. alter only works as long as there are active members in
> > the group subscription. I was bitten by this in the past; the documentation
> > was sparse around that expectation but it does make sense.
> > > >>> AK3. KIP references AdminClient but I believe this is deprecated in
> > favour of just Admin which is the interface that has the methods?
> > > >>> AK4. How do we leverage the describe groups RPC when using the
> > script/admin client? I’m trying to imagine flows other than the export
> > group offsets.
> > > >>> AK4.1 Can we list active members/offset state in a given group in
> > addition to the GroupState? This has helped identify zombie processes to
> > hunt down. The broker’s view would be useful here. (related to AK2)
> > > >>> AK4.2 Could we return empty set of acl operations instead of null?
> > > >>>
> > > >>> Best,
> > > >>> Aditya
> > > >>>
> > > >>>
> > > >>>> On 2026/06/17 06:34:11 Andrew Schofield wrote:
> > > >>>> Hi,
> > > >>>> I’d like to start discussion on a small KIP to improve the
> > usability of the command-line tools for resetting group offsets. When
> > implementing KIP-1323 and reviewing the PRs, it became clear that the
> > usability of this area of Kafka isn’t great. This KIP improves the
> > usability of these tools.
> > > >>>>
> > > >>>>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1359%3A+Improve+usability+of+resetting+group+offsets
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Andrew
> > > >>>>
> > > >>
> > >
> >
> 

Re: [DISCUSS] KIP-1359: Improve usability of resetting group offsets

Posted by Apoorv Mittal <ap...@gmail.com>.
Hi Andrew,
Thanks for the KIP. I have one question.

AM1: Since this KIP adds --by-duration to kafka-share-groups.sh while share
groups also accept share.auto.offset.reset=by_duration:… as a group config. I
can find share.auto.offset.reset=by_duration:PT3H (config,
kafka-configs.sh): an init-time policy the broker applies only when no SPSO
exists. I can set the config while there are active members but the config
update is not honoured. It's only honoured when a new group is specified.
Can you please clarify the expected behaviour when using
`kafka-share-groups.sh --by-duration` reset option?

Regards,
Apoorv Mittal


On Thu, Aug 13, 2026 at 1:21 PM Andrew Schofield <sc...@apache.org>
wrote:

> Hi Aditya,
> Thanks for your reply. I'm not dismissing your concern, but I'm not
> entirely happy with a way forward yet.
>
> AK1: I wonder whether we could use a pipe here to avoid the intermediate
> file. After all, export uses stdout as the destination, so if we could use
> stdin as the source. For example, I've used '-' with --from-file below to
> indicate stdin. It could be --from-stdin or similar too.
>
> $ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092
> --export-offsets --group CG1 --topic T1 --to-current |
> bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092
> --reset-offsets --group CG2 --from-file - --topic T1 --execute
>
> This would still allow resetting between different group types, which is
> actually the use case that got me into this, taking offsets from a consumer
> group to initialise a share group.
>
> Thanks,
> Andrew
>
> On 2026/08/12 18:12:08 Aditya Kousik wrote:
> > Hi Andrew.
> >
> > Thanks for addressing AK2-4: lgtm
> >
> > On AK1: I’m thinking from a user experience standpoint. The teams
> managing the kafka clusters and the ones using the clients diverge - but
> the Admin role may be exposed via self-service typically via API requests
> and a K8S CRD. I’m thinking of the latter case where Kafka SRE allow a thin
> wrapper for client teams to manage their own groups and topics and also
> isolating the environment where the scripts can be run. In such a case, it
> becomes cumbersome to fetch/write the csv file from a Pod that runs the
> migration ; it may not have any volume to persist it or running kubectl to
> scp the file is usually denied. So in such a case, the migrate offsets
> script could just be a single command that fetches offsets from one group
> and applies it to another (or several) without the middleman CSV.
> >
> > Thanks,
> > Aditya
> >
> > > On Aug 12, 2026, at 06:39, Andrew Schofield <sc...@apache.org>
> wrote:
> > >
> > > Hi Aditya,
> > > Thanks for your reply. Sorry for the delay.
> > >
> > > AK1: One of the advantages of the file is that it's possible to export
> from one group type and import into another. You can also export offsets
> for multiple groups, and this changes the file format (4 columns instead of
> 3). Sadly, this is all existing behaviour that I only learnt by trial and
> error. Personally, I think that export-import is a relatively infrequent
> operation, and the KIP makes enough of a step forward. I wonder if you
> could elaborate a bit more on your idea. Feel free to push back if you
> disagree.
> > >
> > > AK2: The KIP does not mention this and I wasn't quite sure where to
> add it. You are correct. Should we improve the javadoc for the admin client?
> > >
> > > AK3: Yes, Admin is better. KIP updated.
> > >
> > > AK4: The DescribeGroupsGeneric RPC is intended as a way of getting a
> minimal set of group information for groups of any group types. It is used
> in Admin.describeGroups and this would be the initial Admin call from
> bin/kafka-XXXX-groups.sh when exporting or resetting offsets. It gives a
> way to check the group type and existence unambiguously. By contrast, if
> you use Admin.describeXXXGroups() for an existing group ID of the wrong
> type, the error code is GROUP_ID_NOT_FOUND.
> > >
> > > AK4.1: I didn't include the members intentionally. This RPC is
> agnostic to group type. I think the common subset of group attributes is
> group ID, group type, group state and coordinator. If you dig deeper, you
> find that the group types differ. For example, assignments in consumer
> groups and share groups are partitions, while in streams groups they are
> tasks. I expect we'll add connect groups one day, and until we have the
> KIP, I wouldn't want to assume the details of membership.
> > >
> > > AK4.2: This is following existing convention I'm afraid and I want to
> be consistent. If the broker does not send back a list of ACL operations,
> then this will be null. This is different than an empty set which means
> "you are not authorized", as opposed to "you don't know anything about
> authorized operations".
> > >
> > > Thanks,
> > > Andrew
> > >
> > >> On 2026/07/27 05:45:27 Aditya Kousik wrote:
> > >> A small correction.
> > >>
> > >> I meant to add a statement before the numbered comments.
> > >>
> > >> “Since the KIP motivation revolves around usability of the script,
> would you be willing to consider the following:”
> > >>
> > >> Also, typo on AK2. “…As long as there aren’t any active members in
> the group subscription”.
> > >>
> > >> -Aditya
> > >>
> > >>>> On Jul 25, 2026, at 23:53, Aditya Kousik <ad...@icloud.com>
> wrote:
> > >>>
> > >>> Hi Andrew,
> > >>>
> > >>> Thanks for this KIP, I’ve personally faced more than one scenario
> where I’ve had to use the admin client to update the offsets.
> > >>>
> > >>> The main uses cases teams face ( and TIL about the various options
> the script offers):
> > >>> Teams migrating to a new consumer group from an existing one but
> don’t want to commit to either (or duration for newer clients) auto offset
> reset policy on the new group.
> > >>> Teams want to seek past certain offsets in partitions because they
> have poison pill messages/corrupted deserialization story and they have no
> control over the data. In this case, they actually want latest offset
> strategy but most don’t even realise that it applies to streams out of
> range and are surprised/frustrated that the property doesn’t move their
> position in the queues. (I see we can do this already in the script today)
> > >>> When I migrated our clients from 2.x directly to 4.x, out of
> abundance of caution, I wrote an hourly cron job that dumps the snapshot of
> group_id,topic,partition,offset into csv files so that if I had to revert a
> few affected partitions I can trim the csv down to those partitions and
> alter them with the admin client.
> > >>>
> > >>> AK1. A single command line with args —source-group-id
> —target-group-id —export so that the intermediate CSV file can be skipped.
> > >>> AK2. Minor: The KIP does not mention the alter/list consumer group
> offsets Admin call. alter only works as long as there are active members in
> the group subscription. I was bitten by this in the past; the documentation
> was sparse around that expectation but it does make sense.
> > >>> AK3. KIP references AdminClient but I believe this is deprecated in
> favour of just Admin which is the interface that has the methods?
> > >>> AK4. How do we leverage the describe groups RPC when using the
> script/admin client? I’m trying to imagine flows other than the export
> group offsets.
> > >>> AK4.1 Can we list active members/offset state in a given group in
> addition to the GroupState? This has helped identify zombie processes to
> hunt down. The broker’s view would be useful here. (related to AK2)
> > >>> AK4.2 Could we return empty set of acl operations instead of null?
> > >>>
> > >>> Best,
> > >>> Aditya
> > >>>
> > >>>
> > >>>> On 2026/06/17 06:34:11 Andrew Schofield wrote:
> > >>>> Hi,
> > >>>> I’d like to start discussion on a small KIP to improve the
> usability of the command-line tools for resetting group offsets. When
> implementing KIP-1323 and reviewing the PRs, it became clear that the
> usability of this area of Kafka isn’t great. This KIP improves the
> usability of these tools.
> > >>>>
> > >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1359%3A+Improve+usability+of+resetting+group+offsets
> > >>>>
> > >>>> Thanks,
> > >>>> Andrew
> > >>>>
> > >>
> >
>

Re: [DISCUSS] KIP-1359: Improve usability of resetting group offsets

Posted by Andrew Schofield <sc...@apache.org>.
Hi Aditya,
Thanks for your reply. I'm not dismissing your concern, but I'm not entirely happy with a way forward yet.

AK1: I wonder whether we could use a pipe here to avoid the intermediate file. After all, export uses stdout as the destination, so if we could use stdin as the source. For example, I've used '-' with --from-file below to indicate stdin. It could be --from-stdin or similar too.

$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --export-offsets --group CG1 --topic T1 --to-current | bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --reset-offsets --group CG2 --from-file - --topic T1 --execute

This would still allow resetting between different group types, which is actually the use case that got me into this, taking offsets from a consumer group to initialise a share group.

Thanks,
Andrew

On 2026/08/12 18:12:08 Aditya Kousik wrote:
> Hi Andrew. 
> 
> Thanks for addressing AK2-4: lgtm
> 
> On AK1: I’m thinking from a user experience standpoint. The teams managing the kafka clusters and the ones using the clients diverge - but the Admin role may be exposed via self-service typically via API requests and a K8S CRD. I’m thinking of the latter case where Kafka SRE allow a thin wrapper for client teams to manage their own groups and topics and also isolating the environment where the scripts can be run. In such a case, it becomes cumbersome to fetch/write the csv file from a Pod that runs the migration ; it may not have any volume to persist it or running kubectl to scp the file is usually denied. So in such a case, the migrate offsets script could just be a single command that fetches offsets from one group and applies it to another (or several) without the middleman CSV.
> 
> Thanks,
> Aditya
> 
> > On Aug 12, 2026, at 06:39, Andrew Schofield <sc...@apache.org> wrote:
> > 
> > Hi Aditya,
> > Thanks for your reply. Sorry for the delay.
> > 
> > AK1: One of the advantages of the file is that it's possible to export from one group type and import into another. You can also export offsets for multiple groups, and this changes the file format (4 columns instead of 3). Sadly, this is all existing behaviour that I only learnt by trial and error. Personally, I think that export-import is a relatively infrequent operation, and the KIP makes enough of a step forward. I wonder if you could elaborate a bit more on your idea. Feel free to push back if you disagree.
> > 
> > AK2: The KIP does not mention this and I wasn't quite sure where to add it. You are correct. Should we improve the javadoc for the admin client?
> > 
> > AK3: Yes, Admin is better. KIP updated.
> > 
> > AK4: The DescribeGroupsGeneric RPC is intended as a way of getting a minimal set of group information for groups of any group types. It is used in Admin.describeGroups and this would be the initial Admin call from bin/kafka-XXXX-groups.sh when exporting or resetting offsets. It gives a way to check the group type and existence unambiguously. By contrast, if you use Admin.describeXXXGroups() for an existing group ID of the wrong type, the error code is GROUP_ID_NOT_FOUND.
> > 
> > AK4.1: I didn't include the members intentionally. This RPC is agnostic to group type. I think the common subset of group attributes is group ID, group type, group state and coordinator. If you dig deeper, you find that the group types differ. For example, assignments in consumer groups and share groups are partitions, while in streams groups they are tasks. I expect we'll add connect groups one day, and until we have the KIP, I wouldn't want to assume the details of membership.
> > 
> > AK4.2: This is following existing convention I'm afraid and I want to be consistent. If the broker does not send back a list of ACL operations, then this will be null. This is different than an empty set which means "you are not authorized", as opposed to "you don't know anything about authorized operations".
> > 
> > Thanks,
> > Andrew
> > 
> >> On 2026/07/27 05:45:27 Aditya Kousik wrote:
> >> A small correction.
> >> 
> >> I meant to add a statement before the numbered comments.
> >> 
> >> “Since the KIP motivation revolves around usability of the script, would you be willing to consider the following:”
> >> 
> >> Also, typo on AK2. “…As long as there aren’t any active members in the group subscription”.
> >> 
> >> -Aditya
> >> 
> >>>> On Jul 25, 2026, at 23:53, Aditya Kousik <ad...@icloud.com> wrote:
> >>> 
> >>> Hi Andrew,
> >>> 
> >>> Thanks for this KIP, I’ve personally faced more than one scenario where I’ve had to use the admin client to update the offsets.
> >>> 
> >>> The main uses cases teams face ( and TIL about the various options the script offers):
> >>> Teams migrating to a new consumer group from an existing one but don’t want to commit to either (or duration for newer clients) auto offset reset policy on the new group.
> >>> Teams want to seek past certain offsets in partitions because they have poison pill messages/corrupted deserialization story and they have no control over the data. In this case, they actually want latest offset strategy but most don’t even realise that it applies to streams out of range and are surprised/frustrated that the property doesn’t move their position in the queues. (I see we can do this already in the script today)
> >>> When I migrated our clients from 2.x directly to 4.x, out of abundance of caution, I wrote an hourly cron job that dumps the snapshot of group_id,topic,partition,offset into csv files so that if I had to revert a few affected partitions I can trim the csv down to those partitions and alter them with the admin client.
> >>> 
> >>> AK1. A single command line with args —source-group-id —target-group-id —export so that the intermediate CSV file can be skipped.
> >>> AK2. Minor: The KIP does not mention the alter/list consumer group offsets Admin call. alter only works as long as there are active members in the group subscription. I was bitten by this in the past; the documentation was sparse around that expectation but it does make sense.
> >>> AK3. KIP references AdminClient but I believe this is deprecated in favour of just Admin which is the interface that has the methods?
> >>> AK4. How do we leverage the describe groups RPC when using the script/admin client? I’m trying to imagine flows other than the export group offsets.
> >>> AK4.1 Can we list active members/offset state in a given group in addition to the GroupState? This has helped identify zombie processes to hunt down. The broker’s view would be useful here. (related to AK2)
> >>> AK4.2 Could we return empty set of acl operations instead of null?
> >>> 
> >>> Best,
> >>> Aditya
> >>> 
> >>> 
> >>>> On 2026/06/17 06:34:11 Andrew Schofield wrote:
> >>>> Hi,
> >>>> I’d like to start discussion on a small KIP to improve the usability of the command-line tools for resetting group offsets. When implementing KIP-1323 and reviewing the PRs, it became clear that the usability of this area of Kafka isn’t great. This KIP improves the usability of these tools.
> >>>> 
> >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1359%3A+Improve+usability+of+resetting+group+offsets
> >>>> 
> >>>> Thanks,
> >>>> Andrew
> >>>> 
> >> 
> 

Re: [DISCUSS] KIP-1359: Improve usability of resetting group offsets

Posted by Aditya Kousik <ad...@icloud.com>.
Hi Andrew. 

Thanks for addressing AK2-4: lgtm

On AK1: I’m thinking from a user experience standpoint. The teams managing the kafka clusters and the ones using the clients diverge - but the Admin role may be exposed via self-service typically via API requests and a K8S CRD. I’m thinking of the latter case where Kafka SRE allow a thin wrapper for client teams to manage their own groups and topics and also isolating the environment where the scripts can be run. In such a case, it becomes cumbersome to fetch/write the csv file from a Pod that runs the migration ; it may not have any volume to persist it or running kubectl to scp the file is usually denied. So in such a case, the migrate offsets script could just be a single command that fetches offsets from one group and applies it to another (or several) without the middleman CSV.

Thanks,
Aditya

> On Aug 12, 2026, at 06:39, Andrew Schofield <sc...@apache.org> wrote:
> 
> Hi Aditya,
> Thanks for your reply. Sorry for the delay.
> 
> AK1: One of the advantages of the file is that it's possible to export from one group type and import into another. You can also export offsets for multiple groups, and this changes the file format (4 columns instead of 3). Sadly, this is all existing behaviour that I only learnt by trial and error. Personally, I think that export-import is a relatively infrequent operation, and the KIP makes enough of a step forward. I wonder if you could elaborate a bit more on your idea. Feel free to push back if you disagree.
> 
> AK2: The KIP does not mention this and I wasn't quite sure where to add it. You are correct. Should we improve the javadoc for the admin client?
> 
> AK3: Yes, Admin is better. KIP updated.
> 
> AK4: The DescribeGroupsGeneric RPC is intended as a way of getting a minimal set of group information for groups of any group types. It is used in Admin.describeGroups and this would be the initial Admin call from bin/kafka-XXXX-groups.sh when exporting or resetting offsets. It gives a way to check the group type and existence unambiguously. By contrast, if you use Admin.describeXXXGroups() for an existing group ID of the wrong type, the error code is GROUP_ID_NOT_FOUND.
> 
> AK4.1: I didn't include the members intentionally. This RPC is agnostic to group type. I think the common subset of group attributes is group ID, group type, group state and coordinator. If you dig deeper, you find that the group types differ. For example, assignments in consumer groups and share groups are partitions, while in streams groups they are tasks. I expect we'll add connect groups one day, and until we have the KIP, I wouldn't want to assume the details of membership.
> 
> AK4.2: This is following existing convention I'm afraid and I want to be consistent. If the broker does not send back a list of ACL operations, then this will be null. This is different than an empty set which means "you are not authorized", as opposed to "you don't know anything about authorized operations".
> 
> Thanks,
> Andrew
> 
>> On 2026/07/27 05:45:27 Aditya Kousik wrote:
>> A small correction.
>> 
>> I meant to add a statement before the numbered comments.
>> 
>> “Since the KIP motivation revolves around usability of the script, would you be willing to consider the following:”
>> 
>> Also, typo on AK2. “…As long as there aren’t any active members in the group subscription”.
>> 
>> -Aditya
>> 
>>>> On Jul 25, 2026, at 23:53, Aditya Kousik <ad...@icloud.com> wrote:
>>> 
>>> Hi Andrew,
>>> 
>>> Thanks for this KIP, I’ve personally faced more than one scenario where I’ve had to use the admin client to update the offsets.
>>> 
>>> The main uses cases teams face ( and TIL about the various options the script offers):
>>> Teams migrating to a new consumer group from an existing one but don’t want to commit to either (or duration for newer clients) auto offset reset policy on the new group.
>>> Teams want to seek past certain offsets in partitions because they have poison pill messages/corrupted deserialization story and they have no control over the data. In this case, they actually want latest offset strategy but most don’t even realise that it applies to streams out of range and are surprised/frustrated that the property doesn’t move their position in the queues. (I see we can do this already in the script today)
>>> When I migrated our clients from 2.x directly to 4.x, out of abundance of caution, I wrote an hourly cron job that dumps the snapshot of group_id,topic,partition,offset into csv files so that if I had to revert a few affected partitions I can trim the csv down to those partitions and alter them with the admin client.
>>> 
>>> AK1. A single command line with args —source-group-id —target-group-id —export so that the intermediate CSV file can be skipped.
>>> AK2. Minor: The KIP does not mention the alter/list consumer group offsets Admin call. alter only works as long as there are active members in the group subscription. I was bitten by this in the past; the documentation was sparse around that expectation but it does make sense.
>>> AK3. KIP references AdminClient but I believe this is deprecated in favour of just Admin which is the interface that has the methods?
>>> AK4. How do we leverage the describe groups RPC when using the script/admin client? I’m trying to imagine flows other than the export group offsets.
>>> AK4.1 Can we list active members/offset state in a given group in addition to the GroupState? This has helped identify zombie processes to hunt down. The broker’s view would be useful here. (related to AK2)
>>> AK4.2 Could we return empty set of acl operations instead of null?
>>> 
>>> Best,
>>> Aditya
>>> 
>>> 
>>>> On 2026/06/17 06:34:11 Andrew Schofield wrote:
>>>> Hi,
>>>> I’d like to start discussion on a small KIP to improve the usability of the command-line tools for resetting group offsets. When implementing KIP-1323 and reviewing the PRs, it became clear that the usability of this area of Kafka isn’t great. This KIP improves the usability of these tools.
>>>> 
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1359%3A+Improve+usability+of+resetting+group+offsets
>>>> 
>>>> Thanks,
>>>> Andrew
>>>> 
>>