You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Mickael Maison <mi...@gmail.com> on 2025/11/25 14:40:48 UTC
[DISCUSS] KIP-1239: Batch offset translation in RemoteClusterUtils
Hi,
I'd like to start a discussion about KIP-1239: Batch offset
translation in RemoteClusterUtils.
This proposes introducing new methods in
RemoteClusterUtils/MirrorClient to translate the offsets multiple
consumer groups at the same time.
https://cwiki.apache.org/confluence/x/h4HMFw
Let me know if you have any feedback.
Thanks,
Mickael
Re: [DISCUSS] KIP-1239: Batch offset translation in RemoteClusterUtils
Posted by Mickael Maison <mi...@gmail.com>.
Hi,
If I don't get any other comments, I'll start a vote in the next few days.
Thanks,
Mickael
On Wed, Feb 4, 2026 at 7:35 PM Chris Egerton <fe...@gmail.com> wrote:
>
> Hi,
>
> Valid point about consistency in null handling. My thinking was to make it
> easy to acquire the information about all translated offsets, and have it
> be opt-in to narrow things down. But we can just add a note about using
> /.*/ to capture every group's offset if users want.
>
> Again, not a blocker, the KIP is fine as-is.
>
> Cheers,
>
> Chris
>
> On Wed, Feb 4, 2026, 13:20 Mickael Maison <mi...@gmail.com> wrote:
>
> > Hi,
> >
> > I'm not very keen on handling null as a sentinel value. Do we do that
> > anywhere else in the Kafka APIs?
> > Did you have a specific use case in mind?
> >
> > Considering translation is an expensive operation, especially when
> > returning all groups, I'd rather have users to provide a valid Pattern
> > and throw immediately if null is provided.
> > For clarity I added @throw to the javadoc examples.
> >
> > Thanks,
> > Mickael
> >
> > On Wed, Feb 4, 2026 at 6:28 PM Chris Egerton <fe...@gmail.com>
> > wrote:
> > >
> > > Hi Mickael,
> > >
> > > Thanks, I like this approach and would be willing to +1 it on a vote
> > > thread. I think it may be useful to make the Pattern nullable and, if
> > null
> > > is passed, return translated offsets for all consumer groups, but I don't
> > > consider that a blocker for the KIP.
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Wed, Feb 4, 2026, 11:57 Mickael Maison <mi...@gmail.com>
> > wrote:
> > >
> > > > Hi Chris,
> > > >
> > > > Thanks for the feedback.
> > > >
> > > > 1. Yes I agree that supporting the "translate all" use case is worth
> > > > considering.
> > > > Instead of adding an additional method, I updated the proposal to
> > > > specify the consumer groups to translate via a regex instead of a
> > > > collection.
> > > > I think the memory footprint would only become an issue with extremely
> > > > large number of committed offsets.
> > > >
> > > > 2. Not applicable anymore with the new API
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Tue, Nov 25, 2025 at 4:03 PM Chris Egerton <fearthecellos@gmail.com
> > >
> > > > wrote:
> > > > >
> > > > > Hi Mickael,
> > > > >
> > > > > Thanks for the KIP. It looks fine, there's a few details I think we
> > > > should
> > > > > iron out.
> > > > >
> > > > > 1. Does it make sense to allow callers to retrieve translated
> > offsets for
> > > > > all consumer groups, without having to specify a subset of group IDs
> > at
> > > > the
> > > > > call site? The only reason to reject this alternative that comes to
> > mind
> > > > > right now is that the memory footprint of the resulting map may be
> > too
> > > > > large in some cases, but it'd be nice to document this rationale if
> > > > that's
> > > > > the case.
> > > > >
> > > > > 2. Based on the existing MirrorClient::remoteConsumerOffsets
> > > > > implementation, it looks like when there are no checkpoints for the
> > group
> > > > > ID, the resulting map will just be empty. I'm guessing the intention
> > with
> > > > > the new batched method variant is similar--if there are no
> > checkpoints
> > > > for
> > > > > one or more of the specified group IDs, then each of those group IDs
> > will
> > > > > have an entry in the batched map, but its value will be an empty
> > map. Is
> > > > > that correct, instead of alternatives like there being no entry in
> > the
> > > > > batched map for those group IDs, or there being an entry with a null
> > > > value,
> > > > > or an exception being thrown, etc.?
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Tue, Nov 25, 2025 at 9:41 AM Mickael Maison <
> > mickael.maison@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I'd like to start a discussion about KIP-1239: Batch offset
> > > > > > translation in RemoteClusterUtils.
> > > > > > This proposes introducing new methods in
> > > > > > RemoteClusterUtils/MirrorClient to translate the offsets multiple
> > > > > > consumer groups at the same time.
> > > > > >
> > > > > > https://cwiki.apache.org/confluence/x/h4HMFw
> > > > > >
> > > > > > Let me know if you have any feedback.
> > > > > >
> > > > > > Thanks,
> > > > > > Mickael
> > > > > >
> > > >
> >
Re: [DISCUSS] KIP-1239: Batch offset translation in RemoteClusterUtils
Posted by Chris Egerton <fe...@gmail.com>.
Hi,
Valid point about consistency in null handling. My thinking was to make it
easy to acquire the information about all translated offsets, and have it
be opt-in to narrow things down. But we can just add a note about using
/.*/ to capture every group's offset if users want.
Again, not a blocker, the KIP is fine as-is.
Cheers,
Chris
On Wed, Feb 4, 2026, 13:20 Mickael Maison <mi...@gmail.com> wrote:
> Hi,
>
> I'm not very keen on handling null as a sentinel value. Do we do that
> anywhere else in the Kafka APIs?
> Did you have a specific use case in mind?
>
> Considering translation is an expensive operation, especially when
> returning all groups, I'd rather have users to provide a valid Pattern
> and throw immediately if null is provided.
> For clarity I added @throw to the javadoc examples.
>
> Thanks,
> Mickael
>
> On Wed, Feb 4, 2026 at 6:28 PM Chris Egerton <fe...@gmail.com>
> wrote:
> >
> > Hi Mickael,
> >
> > Thanks, I like this approach and would be willing to +1 it on a vote
> > thread. I think it may be useful to make the Pattern nullable and, if
> null
> > is passed, return translated offsets for all consumer groups, but I don't
> > consider that a blocker for the KIP.
> >
> > Cheers,
> >
> > Chris
> >
> > On Wed, Feb 4, 2026, 11:57 Mickael Maison <mi...@gmail.com>
> wrote:
> >
> > > Hi Chris,
> > >
> > > Thanks for the feedback.
> > >
> > > 1. Yes I agree that supporting the "translate all" use case is worth
> > > considering.
> > > Instead of adding an additional method, I updated the proposal to
> > > specify the consumer groups to translate via a regex instead of a
> > > collection.
> > > I think the memory footprint would only become an issue with extremely
> > > large number of committed offsets.
> > >
> > > 2. Not applicable anymore with the new API
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Tue, Nov 25, 2025 at 4:03 PM Chris Egerton <fearthecellos@gmail.com
> >
> > > wrote:
> > > >
> > > > Hi Mickael,
> > > >
> > > > Thanks for the KIP. It looks fine, there's a few details I think we
> > > should
> > > > iron out.
> > > >
> > > > 1. Does it make sense to allow callers to retrieve translated
> offsets for
> > > > all consumer groups, without having to specify a subset of group IDs
> at
> > > the
> > > > call site? The only reason to reject this alternative that comes to
> mind
> > > > right now is that the memory footprint of the resulting map may be
> too
> > > > large in some cases, but it'd be nice to document this rationale if
> > > that's
> > > > the case.
> > > >
> > > > 2. Based on the existing MirrorClient::remoteConsumerOffsets
> > > > implementation, it looks like when there are no checkpoints for the
> group
> > > > ID, the resulting map will just be empty. I'm guessing the intention
> with
> > > > the new batched method variant is similar--if there are no
> checkpoints
> > > for
> > > > one or more of the specified group IDs, then each of those group IDs
> will
> > > > have an entry in the batched map, but its value will be an empty
> map. Is
> > > > that correct, instead of alternatives like there being no entry in
> the
> > > > batched map for those group IDs, or there being an entry with a null
> > > value,
> > > > or an exception being thrown, etc.?
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Tue, Nov 25, 2025 at 9:41 AM Mickael Maison <
> mickael.maison@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I'd like to start a discussion about KIP-1239: Batch offset
> > > > > translation in RemoteClusterUtils.
> > > > > This proposes introducing new methods in
> > > > > RemoteClusterUtils/MirrorClient to translate the offsets multiple
> > > > > consumer groups at the same time.
> > > > >
> > > > > https://cwiki.apache.org/confluence/x/h4HMFw
> > > > >
> > > > > Let me know if you have any feedback.
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > >
>
Re: [DISCUSS] KIP-1239: Batch offset translation in RemoteClusterUtils
Posted by Mickael Maison <mi...@gmail.com>.
Hi,
I'm not very keen on handling null as a sentinel value. Do we do that
anywhere else in the Kafka APIs?
Did you have a specific use case in mind?
Considering translation is an expensive operation, especially when
returning all groups, I'd rather have users to provide a valid Pattern
and throw immediately if null is provided.
For clarity I added @throw to the javadoc examples.
Thanks,
Mickael
On Wed, Feb 4, 2026 at 6:28 PM Chris Egerton <fe...@gmail.com> wrote:
>
> Hi Mickael,
>
> Thanks, I like this approach and would be willing to +1 it on a vote
> thread. I think it may be useful to make the Pattern nullable and, if null
> is passed, return translated offsets for all consumer groups, but I don't
> consider that a blocker for the KIP.
>
> Cheers,
>
> Chris
>
> On Wed, Feb 4, 2026, 11:57 Mickael Maison <mi...@gmail.com> wrote:
>
> > Hi Chris,
> >
> > Thanks for the feedback.
> >
> > 1. Yes I agree that supporting the "translate all" use case is worth
> > considering.
> > Instead of adding an additional method, I updated the proposal to
> > specify the consumer groups to translate via a regex instead of a
> > collection.
> > I think the memory footprint would only become an issue with extremely
> > large number of committed offsets.
> >
> > 2. Not applicable anymore with the new API
> >
> > Thanks,
> > Mickael
> >
> > On Tue, Nov 25, 2025 at 4:03 PM Chris Egerton <fe...@gmail.com>
> > wrote:
> > >
> > > Hi Mickael,
> > >
> > > Thanks for the KIP. It looks fine, there's a few details I think we
> > should
> > > iron out.
> > >
> > > 1. Does it make sense to allow callers to retrieve translated offsets for
> > > all consumer groups, without having to specify a subset of group IDs at
> > the
> > > call site? The only reason to reject this alternative that comes to mind
> > > right now is that the memory footprint of the resulting map may be too
> > > large in some cases, but it'd be nice to document this rationale if
> > that's
> > > the case.
> > >
> > > 2. Based on the existing MirrorClient::remoteConsumerOffsets
> > > implementation, it looks like when there are no checkpoints for the group
> > > ID, the resulting map will just be empty. I'm guessing the intention with
> > > the new batched method variant is similar--if there are no checkpoints
> > for
> > > one or more of the specified group IDs, then each of those group IDs will
> > > have an entry in the batched map, but its value will be an empty map. Is
> > > that correct, instead of alternatives like there being no entry in the
> > > batched map for those group IDs, or there being an entry with a null
> > value,
> > > or an exception being thrown, etc.?
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Tue, Nov 25, 2025 at 9:41 AM Mickael Maison <mickael.maison@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I'd like to start a discussion about KIP-1239: Batch offset
> > > > translation in RemoteClusterUtils.
> > > > This proposes introducing new methods in
> > > > RemoteClusterUtils/MirrorClient to translate the offsets multiple
> > > > consumer groups at the same time.
> > > >
> > > > https://cwiki.apache.org/confluence/x/h4HMFw
> > > >
> > > > Let me know if you have any feedback.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> >
Re: [DISCUSS] KIP-1239: Batch offset translation in RemoteClusterUtils
Posted by Chris Egerton <fe...@gmail.com>.
Hi Mickael,
Thanks, I like this approach and would be willing to +1 it on a vote
thread. I think it may be useful to make the Pattern nullable and, if null
is passed, return translated offsets for all consumer groups, but I don't
consider that a blocker for the KIP.
Cheers,
Chris
On Wed, Feb 4, 2026, 11:57 Mickael Maison <mi...@gmail.com> wrote:
> Hi Chris,
>
> Thanks for the feedback.
>
> 1. Yes I agree that supporting the "translate all" use case is worth
> considering.
> Instead of adding an additional method, I updated the proposal to
> specify the consumer groups to translate via a regex instead of a
> collection.
> I think the memory footprint would only become an issue with extremely
> large number of committed offsets.
>
> 2. Not applicable anymore with the new API
>
> Thanks,
> Mickael
>
> On Tue, Nov 25, 2025 at 4:03 PM Chris Egerton <fe...@gmail.com>
> wrote:
> >
> > Hi Mickael,
> >
> > Thanks for the KIP. It looks fine, there's a few details I think we
> should
> > iron out.
> >
> > 1. Does it make sense to allow callers to retrieve translated offsets for
> > all consumer groups, without having to specify a subset of group IDs at
> the
> > call site? The only reason to reject this alternative that comes to mind
> > right now is that the memory footprint of the resulting map may be too
> > large in some cases, but it'd be nice to document this rationale if
> that's
> > the case.
> >
> > 2. Based on the existing MirrorClient::remoteConsumerOffsets
> > implementation, it looks like when there are no checkpoints for the group
> > ID, the resulting map will just be empty. I'm guessing the intention with
> > the new batched method variant is similar--if there are no checkpoints
> for
> > one or more of the specified group IDs, then each of those group IDs will
> > have an entry in the batched map, but its value will be an empty map. Is
> > that correct, instead of alternatives like there being no entry in the
> > batched map for those group IDs, or there being an entry with a null
> value,
> > or an exception being thrown, etc.?
> >
> > Cheers,
> >
> > Chris
> >
> > On Tue, Nov 25, 2025 at 9:41 AM Mickael Maison <mickael.maison@gmail.com
> >
> > wrote:
> >
> > > Hi,
> > >
> > > I'd like to start a discussion about KIP-1239: Batch offset
> > > translation in RemoteClusterUtils.
> > > This proposes introducing new methods in
> > > RemoteClusterUtils/MirrorClient to translate the offsets multiple
> > > consumer groups at the same time.
> > >
> > > https://cwiki.apache.org/confluence/x/h4HMFw
> > >
> > > Let me know if you have any feedback.
> > >
> > > Thanks,
> > > Mickael
> > >
>
Re: [DISCUSS] KIP-1239: Batch offset translation in RemoteClusterUtils
Posted by Mickael Maison <mi...@gmail.com>.
Hi Chris,
Thanks for the feedback.
1. Yes I agree that supporting the "translate all" use case is worth
considering.
Instead of adding an additional method, I updated the proposal to
specify the consumer groups to translate via a regex instead of a
collection.
I think the memory footprint would only become an issue with extremely
large number of committed offsets.
2. Not applicable anymore with the new API
Thanks,
Mickael
On Tue, Nov 25, 2025 at 4:03 PM Chris Egerton <fe...@gmail.com> wrote:
>
> Hi Mickael,
>
> Thanks for the KIP. It looks fine, there's a few details I think we should
> iron out.
>
> 1. Does it make sense to allow callers to retrieve translated offsets for
> all consumer groups, without having to specify a subset of group IDs at the
> call site? The only reason to reject this alternative that comes to mind
> right now is that the memory footprint of the resulting map may be too
> large in some cases, but it'd be nice to document this rationale if that's
> the case.
>
> 2. Based on the existing MirrorClient::remoteConsumerOffsets
> implementation, it looks like when there are no checkpoints for the group
> ID, the resulting map will just be empty. I'm guessing the intention with
> the new batched method variant is similar--if there are no checkpoints for
> one or more of the specified group IDs, then each of those group IDs will
> have an entry in the batched map, but its value will be an empty map. Is
> that correct, instead of alternatives like there being no entry in the
> batched map for those group IDs, or there being an entry with a null value,
> or an exception being thrown, etc.?
>
> Cheers,
>
> Chris
>
> On Tue, Nov 25, 2025 at 9:41 AM Mickael Maison <mi...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I'd like to start a discussion about KIP-1239: Batch offset
> > translation in RemoteClusterUtils.
> > This proposes introducing new methods in
> > RemoteClusterUtils/MirrorClient to translate the offsets multiple
> > consumer groups at the same time.
> >
> > https://cwiki.apache.org/confluence/x/h4HMFw
> >
> > Let me know if you have any feedback.
> >
> > Thanks,
> > Mickael
> >
Re: [DISCUSS] KIP-1239: Batch offset translation in RemoteClusterUtils
Posted by Chris Egerton <fe...@gmail.com>.
Hi Mickael,
Thanks for the KIP. It looks fine, there's a few details I think we should
iron out.
1. Does it make sense to allow callers to retrieve translated offsets for
all consumer groups, without having to specify a subset of group IDs at the
call site? The only reason to reject this alternative that comes to mind
right now is that the memory footprint of the resulting map may be too
large in some cases, but it'd be nice to document this rationale if that's
the case.
2. Based on the existing MirrorClient::remoteConsumerOffsets
implementation, it looks like when there are no checkpoints for the group
ID, the resulting map will just be empty. I'm guessing the intention with
the new batched method variant is similar--if there are no checkpoints for
one or more of the specified group IDs, then each of those group IDs will
have an entry in the batched map, but its value will be an empty map. Is
that correct, instead of alternatives like there being no entry in the
batched map for those group IDs, or there being an entry with a null value,
or an exception being thrown, etc.?
Cheers,
Chris
On Tue, Nov 25, 2025 at 9:41 AM Mickael Maison <mi...@gmail.com>
wrote:
> Hi,
>
> I'd like to start a discussion about KIP-1239: Batch offset
> translation in RemoteClusterUtils.
> This proposes introducing new methods in
> RemoteClusterUtils/MirrorClient to translate the offsets multiple
> consumer groups at the same time.
>
> https://cwiki.apache.org/confluence/x/h4HMFw
>
> Let me know if you have any feedback.
>
> Thanks,
> Mickael
>