You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Alexander Shapiro (ashapiro)" <Al...@amdocs.com.INVALID> on 2023/10/22 12:36:08 UTC
Mirror Maker 2 - offset sync from source to target
Hi
Can someone advise please
if sync.group.offsets.enabled : true to sync offset from source to target for particular consumer group
That group must be created on target, even if no actual consumption will be done ?
This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service <https://www.amdocs.com/about/email-terms-of-service>
Re: Mirror Maker 2 - offset sync from source to target
Posted by Greg Harris <gr...@aiven.io.INVALID>.
Alexander,
Looking at the MM2 documentation
https://kafka.apache.org/documentation/#georeplication it appears that
there's no "summary" of the auto-sync functionality, only the
documentation strings for the individual configurations.
Point (5) above is not covered in the public documentation, and you
would need to read the KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0
to learn about it.
Point (6) above is also not covered in the public documentation, but
has always been a hidden limitation of MirrorMaker2 for a long time
that has recently become more apparent. We still have some work to do
in order to make MirrorMaker2 more resilient to restarts.
Thanks!
Greg
On Mon, Oct 23, 2023 at 9:00 PM Alexander Shapiro (ashapiro)
<Al...@amdocs.com.invalid> wrote:
>
> Not a problem Greg.
>
> Is there some documentation explaining the same ?
> I tried to find a lot in the past
>
> ________________________________
> From: Greg Harris <gr...@aiven.io.INVALID>
> Sent: Monday, October 23, 2023 11:23:26 PM
> To: users@kafka.apache.org <us...@kafka.apache.org>
> Subject: Re: Mirror Maker 2 - offset sync from source to target
>
> [You don't often get email from greg.harris@aiven.io.invalid. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
>
> CAUTION: This email is from an external source. Please don’t open any unknown links or attachments.
>
> Alexander,
>
> My apologies for calling you Andrew.
>
> Greg
>
> On Mon, Oct 23, 2023 at 1:22 PM Greg Harris <gr...@aiven.io> wrote:
> >
> > Andrew,
> >
> > Yes, there isn't an explicit "create consumer group" operation, it
> > should be created when MM2 emits a sync for it.
> >
> > Best,
> > Greg
> >
> > On Mon, Oct 23, 2023 at 1:15 PM Alexander Shapiro (ashapiro)
> > <Al...@amdocs.com.invalid> wrote:
> > >
> > > Thanks, one clarification plz
> > >
> > > In bullet for You mention "4. The target group does not exist, or has no active consumers"
> > > If group on target does not exist, will it be created without active consumers ?
> > >
> > > -----Original Message-----
> > > From: Greg Harris <gr...@aiven.io.INVALID>
> > > Sent: Monday, October 23, 2023 8:56 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: Mirror Maker 2 - offset sync from source to target
> > >
> > > [You don't often get email from greg.harris@aiven.io.invalid. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
> > >
> > > CAUTION: This email is from an external source. Please don’t open any unknown links or attachments.
> > >
> > > Hi Alexander,
> > >
> > > Sorry I noticed an inconsistency in my last email. For point 6:
> > > If the MirrorCheckpointTask restarts after replication but before offset is translated, then it may not be able to perform a translation.
> > > If the MirrorCheckpointTask does not restart, it should be able to perform translation.
> > >
> > > So if your MirrorMaker2 is restarting frequently, that may prevent consistent translation.
> > >
> > > Thanks,
> > > Greg
> > >
> > > On Mon, Oct 23, 2023 at 10:46 AM Alexander Shapiro (ashapiro) <Al...@amdocs.com.invalid> wrote:
> > > >
> > > > Hi Greg,
> > > > Thank you very much,
> > > > it is the most detailed answer I would expect.
> > > >
> > > > -----Original Message-----
> > > > From: Greg Harris <gr...@aiven.io.INVALID>
> > > > Sent: Monday, October 23, 2023 8:42 PM
> > > > To: users@kafka.apache.org
> > > > Subject: Re: Mirror Maker 2 - offset sync from source to target
> > > >
> > > > CAUTION: This email is from an external source. Please don’t open any unknown links or attachments.
> > > >
> > > > Hi Alexander,
> > > >
> > > > Thanks for using MirrorMaker2!
> > > >
> > > > If you turn on `sync.group.offsets.enabled`, then the
> > > > MirrorCheckpointTask will sync the offsets if all of the following is
> > > > true:
> > > > 1. The source group exists
> > > > 2. The source group name matches the configured group filter (group.filter.class, groups, groups.exclude) 3. The source group has an offset for a topic which matches the configured topic filter (topic.filter.class, topics, topics.exclude) 4. The target group does not exist, or has no active consumers 5. The target group has no offset for a specified partition, or the offset is earlier than the translated offset 6. MirrorCheckpointTask restarted after replication happened but before the offset could be translated.
> > > >
> > > > If one of these isn't true, you won't see translation happening.
> > > >
> > > > Are you having a problem with too many consumer groups being created?
> > > > You can restrict the group or topic filters, as they're very permissive by default.
> > > > Or is the problem that you expect the consumer groups to be created, but they aren't? One of the above conditions may not be satisfied, or your MirrorCheckpointConnector/Task may be malfunctioning and you'll need to inspect the logs to debug it.
> > > >
> > > > Best,
> > > > Greg Harris
> > > >
> > > > On Sun, Oct 22, 2023 at 5:36 AM Alexander Shapiro (ashapiro) <Al...@amdocs.com.invalid> wrote:
> > > > >
> > > > > Hi
> > > > > Can someone advise please
> > > > > if sync.group.offsets.enabled : true to sync offset from source to
> > > > > target for particular consumer group That group must be created on target, even if no actual consumption will be done ?
> > > > >
> > > > >
> > > > >
> > > > > This email and the information contained herein is proprietary and
> > > > > confidential and subject to the Amdocs Email Terms of Service, which
> > > > > you may review at
> > > > > https://www.amdocs.com/about/email-terms-of-service
> > > > > <https://www.amdocs.com/about/email-terms-of-service>
> > > > This email and the information contained herein is proprietary and
> > > > confidential and subject to the Amdocs Email Terms of Service, which
> > > > you may review at https://www.amdocs.com/about/email-terms-of-service
> > > > <https://www.amdocs.com/about/email-terms-of-service>
> > > This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service <https://www.amdocs.com/about/email-terms-of-service>
> This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service <https://www.amdocs.com/about/email-terms-of-service>
Re: Mirror Maker 2 - offset sync from source to target
Posted by "Alexander Shapiro (ashapiro)" <Al...@amdocs.com.INVALID>.
Not a problem Greg.
Is there some documentation explaining the same ?
I tried to find a lot in the past
________________________________
From: Greg Harris <gr...@aiven.io.INVALID>
Sent: Monday, October 23, 2023 11:23:26 PM
To: users@kafka.apache.org <us...@kafka.apache.org>
Subject: Re: Mirror Maker 2 - offset sync from source to target
[You don't often get email from greg.harris@aiven.io.invalid. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
CAUTION: This email is from an external source. Please don’t open any unknown links or attachments.
Alexander,
My apologies for calling you Andrew.
Greg
On Mon, Oct 23, 2023 at 1:22 PM Greg Harris <gr...@aiven.io> wrote:
>
> Andrew,
>
> Yes, there isn't an explicit "create consumer group" operation, it
> should be created when MM2 emits a sync for it.
>
> Best,
> Greg
>
> On Mon, Oct 23, 2023 at 1:15 PM Alexander Shapiro (ashapiro)
> <Al...@amdocs.com.invalid> wrote:
> >
> > Thanks, one clarification plz
> >
> > In bullet for You mention "4. The target group does not exist, or has no active consumers"
> > If group on target does not exist, will it be created without active consumers ?
> >
> > -----Original Message-----
> > From: Greg Harris <gr...@aiven.io.INVALID>
> > Sent: Monday, October 23, 2023 8:56 PM
> > To: users@kafka.apache.org
> > Subject: Re: Mirror Maker 2 - offset sync from source to target
> >
> > [You don't often get email from greg.harris@aiven.io.invalid. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
> >
> > CAUTION: This email is from an external source. Please don’t open any unknown links or attachments.
> >
> > Hi Alexander,
> >
> > Sorry I noticed an inconsistency in my last email. For point 6:
> > If the MirrorCheckpointTask restarts after replication but before offset is translated, then it may not be able to perform a translation.
> > If the MirrorCheckpointTask does not restart, it should be able to perform translation.
> >
> > So if your MirrorMaker2 is restarting frequently, that may prevent consistent translation.
> >
> > Thanks,
> > Greg
> >
> > On Mon, Oct 23, 2023 at 10:46 AM Alexander Shapiro (ashapiro) <Al...@amdocs.com.invalid> wrote:
> > >
> > > Hi Greg,
> > > Thank you very much,
> > > it is the most detailed answer I would expect.
> > >
> > > -----Original Message-----
> > > From: Greg Harris <gr...@aiven.io.INVALID>
> > > Sent: Monday, October 23, 2023 8:42 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: Mirror Maker 2 - offset sync from source to target
> > >
> > > CAUTION: This email is from an external source. Please don’t open any unknown links or attachments.
> > >
> > > Hi Alexander,
> > >
> > > Thanks for using MirrorMaker2!
> > >
> > > If you turn on `sync.group.offsets.enabled`, then the
> > > MirrorCheckpointTask will sync the offsets if all of the following is
> > > true:
> > > 1. The source group exists
> > > 2. The source group name matches the configured group filter (group.filter.class, groups, groups.exclude) 3. The source group has an offset for a topic which matches the configured topic filter (topic.filter.class, topics, topics.exclude) 4. The target group does not exist, or has no active consumers 5. The target group has no offset for a specified partition, or the offset is earlier than the translated offset 6. MirrorCheckpointTask restarted after replication happened but before the offset could be translated.
> > >
> > > If one of these isn't true, you won't see translation happening.
> > >
> > > Are you having a problem with too many consumer groups being created?
> > > You can restrict the group or topic filters, as they're very permissive by default.
> > > Or is the problem that you expect the consumer groups to be created, but they aren't? One of the above conditions may not be satisfied, or your MirrorCheckpointConnector/Task may be malfunctioning and you'll need to inspect the logs to debug it.
> > >
> > > Best,
> > > Greg Harris
> > >
> > > On Sun, Oct 22, 2023 at 5:36 AM Alexander Shapiro (ashapiro) <Al...@amdocs.com.invalid> wrote:
> > > >
> > > > Hi
> > > > Can someone advise please
> > > > if sync.group.offsets.enabled : true to sync offset from source to
> > > > target for particular consumer group That group must be created on target, even if no actual consumption will be done ?
> > > >
> > > >
> > > >
> > > > This email and the information contained herein is proprietary and
> > > > confidential and subject to the Amdocs Email Terms of Service, which
> > > > you may review at
> > > > https://www.amdocs.com/about/email-terms-of-service
> > > > <https://www.amdocs.com/about/email-terms-of-service>
> > > This email and the information contained herein is proprietary and
> > > confidential and subject to the Amdocs Email Terms of Service, which
> > > you may review at https://www.amdocs.com/about/email-terms-of-service
> > > <https://www.amdocs.com/about/email-terms-of-service>
> > This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service <https://www.amdocs.com/about/email-terms-of-service>
This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service <https://www.amdocs.com/about/email-terms-of-service>
Re: Mirror Maker 2 - offset sync from source to target
Posted by sherry <sc...@gmail.com>.
Offset is an excellent artist
On Mon, Oct 23, 2566 BE at 22:24 Greg Harris <gr...@aiven.io.invalid>
wrote:
> Alexander,
>
> My apologies for calling you Andrew.
>
> Greg
>
> On Mon, Oct 23, 2023 at 1:22 PM Greg Harris <gr...@aiven.io> wrote:
> >
> > Andrew,
> >
> > Yes, there isn't an explicit "create consumer group" operation, it
> > should be created when MM2 emits a sync for it.
> >
> > Best,
> > Greg
> >
> > On Mon, Oct 23, 2023 at 1:15 PM Alexander Shapiro (ashapiro)
> > <Al...@amdocs.com.invalid> wrote:
> > >
> > > Thanks, one clarification plz
> > >
> > > In bullet for You mention "4. The target group does not exist, or has
> no active consumers"
> > > If group on target does not exist, will it be created without active
> consumers ?
> > >
> > > -----Original Message-----
> > > From: Greg Harris <gr...@aiven.io.INVALID>
> > > Sent: Monday, October 23, 2023 8:56 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: Mirror Maker 2 - offset sync from source to target
> > >
> > > [You don't often get email from greg.harris@aiven.io.invalid. Learn
> why this is important at https://aka.ms/LearnAboutSenderIdentification ]
> > >
> > > CAUTION: This email is from an external source. Please don’t open any
> unknown links or attachments.
> > >
> > > Hi Alexander,
> > >
> > > Sorry I noticed an inconsistency in my last email. For point 6:
> > > If the MirrorCheckpointTask restarts after replication but before
> offset is translated, then it may not be able to perform a translation.
> > > If the MirrorCheckpointTask does not restart, it should be able to
> perform translation.
> > >
> > > So if your MirrorMaker2 is restarting frequently, that may prevent
> consistent translation.
> > >
> > > Thanks,
> > > Greg
> > >
> > > On Mon, Oct 23, 2023 at 10:46 AM Alexander Shapiro (ashapiro) <
> Alexander.Shapiro@amdocs.com.invalid> wrote:
> > > >
> > > > Hi Greg,
> > > > Thank you very much,
> > > > it is the most detailed answer I would expect.
> > > >
> > > > -----Original Message-----
> > > > From: Greg Harris <gr...@aiven.io.INVALID>
> > > > Sent: Monday, October 23, 2023 8:42 PM
> > > > To: users@kafka.apache.org
> > > > Subject: Re: Mirror Maker 2 - offset sync from source to target
> > > >
> > > > CAUTION: This email is from an external source. Please don’t open
> any unknown links or attachments.
> > > >
> > > > Hi Alexander,
> > > >
> > > > Thanks for using MirrorMaker2!
> > > >
> > > > If you turn on `sync.group.offsets.enabled`, then the
> > > > MirrorCheckpointTask will sync the offsets if all of the following is
> > > > true:
> > > > 1. The source group exists
> > > > 2. The source group name matches the configured group filter
> (group.filter.class, groups, groups.exclude) 3. The source group has an
> offset for a topic which matches the configured topic filter
> (topic.filter.class, topics, topics.exclude) 4. The target group does not
> exist, or has no active consumers 5. The target group has no offset for a
> specified partition, or the offset is earlier than the translated offset 6.
> MirrorCheckpointTask restarted after replication happened but before the
> offset could be translated.
> > > >
> > > > If one of these isn't true, you won't see translation happening.
> > > >
> > > > Are you having a problem with too many consumer groups being created?
> > > > You can restrict the group or topic filters, as they're very
> permissive by default.
> > > > Or is the problem that you expect the consumer groups to be created,
> but they aren't? One of the above conditions may not be satisfied, or your
> MirrorCheckpointConnector/Task may be malfunctioning and you'll need to
> inspect the logs to debug it.
> > > >
> > > > Best,
> > > > Greg Harris
> > > >
> > > > On Sun, Oct 22, 2023 at 5:36 AM Alexander Shapiro (ashapiro) <
> Alexander.Shapiro@amdocs.com.invalid> wrote:
> > > > >
> > > > > Hi
> > > > > Can someone advise please
> > > > > if sync.group.offsets.enabled : true to sync offset from source to
> > > > > target for particular consumer group That group must be created on
> target, even if no actual consumption will be done ?
> > > > >
> > > > >
> > > > >
> > > > > This email and the information contained herein is proprietary and
> > > > > confidential and subject to the Amdocs Email Terms of Service,
> which
> > > > > you may review at
> > > > > https://www.amdocs.com/about/email-terms-of-service
> > > > > <https://www.amdocs.com/about/email-terms-of-service>
> > > > This email and the information contained herein is proprietary and
> > > > confidential and subject to the Amdocs Email Terms of Service, which
> > > > you may review at
> https://www.amdocs.com/about/email-terms-of-service
> > > > <https://www.amdocs.com/about/email-terms-of-service>
> > > This email and the information contained herein is proprietary and
> confidential and subject to the Amdocs Email Terms of Service, which you
> may review at https://www.amdocs.com/about/email-terms-of-service <
> https://www.amdocs.com/about/email-terms-of-service>
>
Re: Mirror Maker 2 - offset sync from source to target
Posted by Greg Harris <gr...@aiven.io.INVALID>.
Alexander,
My apologies for calling you Andrew.
Greg
On Mon, Oct 23, 2023 at 1:22 PM Greg Harris <gr...@aiven.io> wrote:
>
> Andrew,
>
> Yes, there isn't an explicit "create consumer group" operation, it
> should be created when MM2 emits a sync for it.
>
> Best,
> Greg
>
> On Mon, Oct 23, 2023 at 1:15 PM Alexander Shapiro (ashapiro)
> <Al...@amdocs.com.invalid> wrote:
> >
> > Thanks, one clarification plz
> >
> > In bullet for You mention "4. The target group does not exist, or has no active consumers"
> > If group on target does not exist, will it be created without active consumers ?
> >
> > -----Original Message-----
> > From: Greg Harris <gr...@aiven.io.INVALID>
> > Sent: Monday, October 23, 2023 8:56 PM
> > To: users@kafka.apache.org
> > Subject: Re: Mirror Maker 2 - offset sync from source to target
> >
> > [You don't often get email from greg.harris@aiven.io.invalid. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
> >
> > CAUTION: This email is from an external source. Please don’t open any unknown links or attachments.
> >
> > Hi Alexander,
> >
> > Sorry I noticed an inconsistency in my last email. For point 6:
> > If the MirrorCheckpointTask restarts after replication but before offset is translated, then it may not be able to perform a translation.
> > If the MirrorCheckpointTask does not restart, it should be able to perform translation.
> >
> > So if your MirrorMaker2 is restarting frequently, that may prevent consistent translation.
> >
> > Thanks,
> > Greg
> >
> > On Mon, Oct 23, 2023 at 10:46 AM Alexander Shapiro (ashapiro) <Al...@amdocs.com.invalid> wrote:
> > >
> > > Hi Greg,
> > > Thank you very much,
> > > it is the most detailed answer I would expect.
> > >
> > > -----Original Message-----
> > > From: Greg Harris <gr...@aiven.io.INVALID>
> > > Sent: Monday, October 23, 2023 8:42 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: Mirror Maker 2 - offset sync from source to target
> > >
> > > CAUTION: This email is from an external source. Please don’t open any unknown links or attachments.
> > >
> > > Hi Alexander,
> > >
> > > Thanks for using MirrorMaker2!
> > >
> > > If you turn on `sync.group.offsets.enabled`, then the
> > > MirrorCheckpointTask will sync the offsets if all of the following is
> > > true:
> > > 1. The source group exists
> > > 2. The source group name matches the configured group filter (group.filter.class, groups, groups.exclude) 3. The source group has an offset for a topic which matches the configured topic filter (topic.filter.class, topics, topics.exclude) 4. The target group does not exist, or has no active consumers 5. The target group has no offset for a specified partition, or the offset is earlier than the translated offset 6. MirrorCheckpointTask restarted after replication happened but before the offset could be translated.
> > >
> > > If one of these isn't true, you won't see translation happening.
> > >
> > > Are you having a problem with too many consumer groups being created?
> > > You can restrict the group or topic filters, as they're very permissive by default.
> > > Or is the problem that you expect the consumer groups to be created, but they aren't? One of the above conditions may not be satisfied, or your MirrorCheckpointConnector/Task may be malfunctioning and you'll need to inspect the logs to debug it.
> > >
> > > Best,
> > > Greg Harris
> > >
> > > On Sun, Oct 22, 2023 at 5:36 AM Alexander Shapiro (ashapiro) <Al...@amdocs.com.invalid> wrote:
> > > >
> > > > Hi
> > > > Can someone advise please
> > > > if sync.group.offsets.enabled : true to sync offset from source to
> > > > target for particular consumer group That group must be created on target, even if no actual consumption will be done ?
> > > >
> > > >
> > > >
> > > > This email and the information contained herein is proprietary and
> > > > confidential and subject to the Amdocs Email Terms of Service, which
> > > > you may review at
> > > > https://www.amdocs.com/about/email-terms-of-service
> > > > <https://www.amdocs.com/about/email-terms-of-service>
> > > This email and the information contained herein is proprietary and
> > > confidential and subject to the Amdocs Email Terms of Service, which
> > > you may review at https://www.amdocs.com/about/email-terms-of-service
> > > <https://www.amdocs.com/about/email-terms-of-service>
> > This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service <https://www.amdocs.com/about/email-terms-of-service>
Re: Mirror Maker 2 - offset sync from source to target
Posted by Greg Harris <gr...@aiven.io.INVALID>.
Andrew,
Yes, there isn't an explicit "create consumer group" operation, it
should be created when MM2 emits a sync for it.
Best,
Greg
On Mon, Oct 23, 2023 at 1:15 PM Alexander Shapiro (ashapiro)
<Al...@amdocs.com.invalid> wrote:
>
> Thanks, one clarification plz
>
> In bullet for You mention "4. The target group does not exist, or has no active consumers"
> If group on target does not exist, will it be created without active consumers ?
>
> -----Original Message-----
> From: Greg Harris <gr...@aiven.io.INVALID>
> Sent: Monday, October 23, 2023 8:56 PM
> To: users@kafka.apache.org
> Subject: Re: Mirror Maker 2 - offset sync from source to target
>
> [You don't often get email from greg.harris@aiven.io.invalid. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
>
> CAUTION: This email is from an external source. Please don’t open any unknown links or attachments.
>
> Hi Alexander,
>
> Sorry I noticed an inconsistency in my last email. For point 6:
> If the MirrorCheckpointTask restarts after replication but before offset is translated, then it may not be able to perform a translation.
> If the MirrorCheckpointTask does not restart, it should be able to perform translation.
>
> So if your MirrorMaker2 is restarting frequently, that may prevent consistent translation.
>
> Thanks,
> Greg
>
> On Mon, Oct 23, 2023 at 10:46 AM Alexander Shapiro (ashapiro) <Al...@amdocs.com.invalid> wrote:
> >
> > Hi Greg,
> > Thank you very much,
> > it is the most detailed answer I would expect.
> >
> > -----Original Message-----
> > From: Greg Harris <gr...@aiven.io.INVALID>
> > Sent: Monday, October 23, 2023 8:42 PM
> > To: users@kafka.apache.org
> > Subject: Re: Mirror Maker 2 - offset sync from source to target
> >
> > CAUTION: This email is from an external source. Please don’t open any unknown links or attachments.
> >
> > Hi Alexander,
> >
> > Thanks for using MirrorMaker2!
> >
> > If you turn on `sync.group.offsets.enabled`, then the
> > MirrorCheckpointTask will sync the offsets if all of the following is
> > true:
> > 1. The source group exists
> > 2. The source group name matches the configured group filter (group.filter.class, groups, groups.exclude) 3. The source group has an offset for a topic which matches the configured topic filter (topic.filter.class, topics, topics.exclude) 4. The target group does not exist, or has no active consumers 5. The target group has no offset for a specified partition, or the offset is earlier than the translated offset 6. MirrorCheckpointTask restarted after replication happened but before the offset could be translated.
> >
> > If one of these isn't true, you won't see translation happening.
> >
> > Are you having a problem with too many consumer groups being created?
> > You can restrict the group or topic filters, as they're very permissive by default.
> > Or is the problem that you expect the consumer groups to be created, but they aren't? One of the above conditions may not be satisfied, or your MirrorCheckpointConnector/Task may be malfunctioning and you'll need to inspect the logs to debug it.
> >
> > Best,
> > Greg Harris
> >
> > On Sun, Oct 22, 2023 at 5:36 AM Alexander Shapiro (ashapiro) <Al...@amdocs.com.invalid> wrote:
> > >
> > > Hi
> > > Can someone advise please
> > > if sync.group.offsets.enabled : true to sync offset from source to
> > > target for particular consumer group That group must be created on target, even if no actual consumption will be done ?
> > >
> > >
> > >
> > > This email and the information contained herein is proprietary and
> > > confidential and subject to the Amdocs Email Terms of Service, which
> > > you may review at
> > > https://www.amdocs.com/about/email-terms-of-service
> > > <https://www.amdocs.com/about/email-terms-of-service>
> > This email and the information contained herein is proprietary and
> > confidential and subject to the Amdocs Email Terms of Service, which
> > you may review at https://www.amdocs.com/about/email-terms-of-service
> > <https://www.amdocs.com/about/email-terms-of-service>
> This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service <https://www.amdocs.com/about/email-terms-of-service>
RE: Mirror Maker 2 - offset sync from source to target
Posted by "Alexander Shapiro (ashapiro)" <Al...@amdocs.com.INVALID>.
Thanks, one clarification plz
In bullet for You mention "4. The target group does not exist, or has no active consumers"
If group on target does not exist, will it be created without active consumers ?
-----Original Message-----
From: Greg Harris <gr...@aiven.io.INVALID>
Sent: Monday, October 23, 2023 8:56 PM
To: users@kafka.apache.org
Subject: Re: Mirror Maker 2 - offset sync from source to target
[You don't often get email from greg.harris@aiven.io.invalid. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
CAUTION: This email is from an external source. Please don’t open any unknown links or attachments.
Hi Alexander,
Sorry I noticed an inconsistency in my last email. For point 6:
If the MirrorCheckpointTask restarts after replication but before offset is translated, then it may not be able to perform a translation.
If the MirrorCheckpointTask does not restart, it should be able to perform translation.
So if your MirrorMaker2 is restarting frequently, that may prevent consistent translation.
Thanks,
Greg
On Mon, Oct 23, 2023 at 10:46 AM Alexander Shapiro (ashapiro) <Al...@amdocs.com.invalid> wrote:
>
> Hi Greg,
> Thank you very much,
> it is the most detailed answer I would expect.
>
> -----Original Message-----
> From: Greg Harris <gr...@aiven.io.INVALID>
> Sent: Monday, October 23, 2023 8:42 PM
> To: users@kafka.apache.org
> Subject: Re: Mirror Maker 2 - offset sync from source to target
>
> CAUTION: This email is from an external source. Please don’t open any unknown links or attachments.
>
> Hi Alexander,
>
> Thanks for using MirrorMaker2!
>
> If you turn on `sync.group.offsets.enabled`, then the
> MirrorCheckpointTask will sync the offsets if all of the following is
> true:
> 1. The source group exists
> 2. The source group name matches the configured group filter (group.filter.class, groups, groups.exclude) 3. The source group has an offset for a topic which matches the configured topic filter (topic.filter.class, topics, topics.exclude) 4. The target group does not exist, or has no active consumers 5. The target group has no offset for a specified partition, or the offset is earlier than the translated offset 6. MirrorCheckpointTask restarted after replication happened but before the offset could be translated.
>
> If one of these isn't true, you won't see translation happening.
>
> Are you having a problem with too many consumer groups being created?
> You can restrict the group or topic filters, as they're very permissive by default.
> Or is the problem that you expect the consumer groups to be created, but they aren't? One of the above conditions may not be satisfied, or your MirrorCheckpointConnector/Task may be malfunctioning and you'll need to inspect the logs to debug it.
>
> Best,
> Greg Harris
>
> On Sun, Oct 22, 2023 at 5:36 AM Alexander Shapiro (ashapiro) <Al...@amdocs.com.invalid> wrote:
> >
> > Hi
> > Can someone advise please
> > if sync.group.offsets.enabled : true to sync offset from source to
> > target for particular consumer group That group must be created on target, even if no actual consumption will be done ?
> >
> >
> >
> > This email and the information contained herein is proprietary and
> > confidential and subject to the Amdocs Email Terms of Service, which
> > you may review at
> > https://www.amdocs.com/about/email-terms-of-service
> > <https://www.amdocs.com/about/email-terms-of-service>
> This email and the information contained herein is proprietary and
> confidential and subject to the Amdocs Email Terms of Service, which
> you may review at https://www.amdocs.com/about/email-terms-of-service
> <https://www.amdocs.com/about/email-terms-of-service>
This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service <https://www.amdocs.com/about/email-terms-of-service>
Re: Mirror Maker 2 - offset sync from source to target
Posted by Greg Harris <gr...@aiven.io.INVALID>.
Hi Alexander,
Sorry I noticed an inconsistency in my last email. For point 6:
If the MirrorCheckpointTask restarts after replication but before
offset is translated, then it may not be able to perform a
translation.
If the MirrorCheckpointTask does not restart, it should be able to
perform translation.
So if your MirrorMaker2 is restarting frequently, that may prevent
consistent translation.
Thanks,
Greg
On Mon, Oct 23, 2023 at 10:46 AM Alexander Shapiro (ashapiro)
<Al...@amdocs.com.invalid> wrote:
>
> Hi Greg,
> Thank you very much,
> it is the most detailed answer I would expect.
>
> -----Original Message-----
> From: Greg Harris <gr...@aiven.io.INVALID>
> Sent: Monday, October 23, 2023 8:42 PM
> To: users@kafka.apache.org
> Subject: Re: Mirror Maker 2 - offset sync from source to target
>
> CAUTION: This email is from an external source. Please don’t open any unknown links or attachments.
>
> Hi Alexander,
>
> Thanks for using MirrorMaker2!
>
> If you turn on `sync.group.offsets.enabled`, then the MirrorCheckpointTask will sync the offsets if all of the following is
> true:
> 1. The source group exists
> 2. The source group name matches the configured group filter (group.filter.class, groups, groups.exclude) 3. The source group has an offset for a topic which matches the configured topic filter (topic.filter.class, topics, topics.exclude) 4. The target group does not exist, or has no active consumers 5. The target group has no offset for a specified partition, or the offset is earlier than the translated offset 6. MirrorCheckpointTask restarted after replication happened but before the offset could be translated.
>
> If one of these isn't true, you won't see translation happening.
>
> Are you having a problem with too many consumer groups being created?
> You can restrict the group or topic filters, as they're very permissive by default.
> Or is the problem that you expect the consumer groups to be created, but they aren't? One of the above conditions may not be satisfied, or your MirrorCheckpointConnector/Task may be malfunctioning and you'll need to inspect the logs to debug it.
>
> Best,
> Greg Harris
>
> On Sun, Oct 22, 2023 at 5:36 AM Alexander Shapiro (ashapiro) <Al...@amdocs.com.invalid> wrote:
> >
> > Hi
> > Can someone advise please
> > if sync.group.offsets.enabled : true to sync offset from source to
> > target for particular consumer group That group must be created on target, even if no actual consumption will be done ?
> >
> >
> >
> > This email and the information contained herein is proprietary and
> > confidential and subject to the Amdocs Email Terms of Service, which
> > you may review at https://www.amdocs.com/about/email-terms-of-service
> > <https://www.amdocs.com/about/email-terms-of-service>
> This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service <https://www.amdocs.com/about/email-terms-of-service>
RE: Mirror Maker 2 - offset sync from source to target
Posted by "Alexander Shapiro (ashapiro)" <Al...@amdocs.com.INVALID>.
Hi Greg,
Thank you very much,
it is the most detailed answer I would expect.
-----Original Message-----
From: Greg Harris <gr...@aiven.io.INVALID>
Sent: Monday, October 23, 2023 8:42 PM
To: users@kafka.apache.org
Subject: Re: Mirror Maker 2 - offset sync from source to target
CAUTION: This email is from an external source. Please don’t open any unknown links or attachments.
Hi Alexander,
Thanks for using MirrorMaker2!
If you turn on `sync.group.offsets.enabled`, then the MirrorCheckpointTask will sync the offsets if all of the following is
true:
1. The source group exists
2. The source group name matches the configured group filter (group.filter.class, groups, groups.exclude) 3. The source group has an offset for a topic which matches the configured topic filter (topic.filter.class, topics, topics.exclude) 4. The target group does not exist, or has no active consumers 5. The target group has no offset for a specified partition, or the offset is earlier than the translated offset 6. MirrorCheckpointTask restarted after replication happened but before the offset could be translated.
If one of these isn't true, you won't see translation happening.
Are you having a problem with too many consumer groups being created?
You can restrict the group or topic filters, as they're very permissive by default.
Or is the problem that you expect the consumer groups to be created, but they aren't? One of the above conditions may not be satisfied, or your MirrorCheckpointConnector/Task may be malfunctioning and you'll need to inspect the logs to debug it.
Best,
Greg Harris
On Sun, Oct 22, 2023 at 5:36 AM Alexander Shapiro (ashapiro) <Al...@amdocs.com.invalid> wrote:
>
> Hi
> Can someone advise please
> if sync.group.offsets.enabled : true to sync offset from source to
> target for particular consumer group That group must be created on target, even if no actual consumption will be done ?
>
>
>
> This email and the information contained herein is proprietary and
> confidential and subject to the Amdocs Email Terms of Service, which
> you may review at https://www.amdocs.com/about/email-terms-of-service
> <https://www.amdocs.com/about/email-terms-of-service>
This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service <https://www.amdocs.com/about/email-terms-of-service>
Re: Mirror Maker 2 - offset sync from source to target
Posted by Greg Harris <gr...@aiven.io.INVALID>.
Hi Alexander,
Thanks for using MirrorMaker2!
If you turn on `sync.group.offsets.enabled`, then the
MirrorCheckpointTask will sync the offsets if all of the following is
true:
1. The source group exists
2. The source group name matches the configured group filter
(group.filter.class, groups, groups.exclude)
3. The source group has an offset for a topic which matches the
configured topic filter (topic.filter.class, topics, topics.exclude)
4. The target group does not exist, or has no active consumers
5. The target group has no offset for a specified partition, or the
offset is earlier than the translated offset
6. MirrorCheckpointTask restarted after replication happened but
before the offset could be translated.
If one of these isn't true, you won't see translation happening.
Are you having a problem with too many consumer groups being created?
You can restrict the group or topic filters, as they're very
permissive by default.
Or is the problem that you expect the consumer groups to be created,
but they aren't? One of the above conditions may not be satisfied, or
your MirrorCheckpointConnector/Task may be malfunctioning and you'll
need to inspect the logs to debug it.
Best,
Greg Harris
On Sun, Oct 22, 2023 at 5:36 AM Alexander Shapiro (ashapiro)
<Al...@amdocs.com.invalid> wrote:
>
> Hi
> Can someone advise please
> if sync.group.offsets.enabled : true to sync offset from source to target for particular consumer group
> That group must be created on target, even if no actual consumption will be done ?
>
>
>
> This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service <https://www.amdocs.com/about/email-terms-of-service>