You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Samruddhi Naik <sa...@gmail.com> on 2021/04/05 09:24:28 UTC
RE: Maintaining same offset while migrating from Confluent
Replicator to Apache Mirror Maker 2.0
Hello Ning,
I was following this email thread and read through your article. I tried executing your script which I found here- https://github.com/ning2008wisc/mm2-migration but to my dismay that did not work. Me and my colleague would really like some help on this one. We still see the messages which were replicated back on the same topic when we transition from mm1 to mm2. Any help will be appreciated. Looking forward to your response.
Thanks,
Samruddhi
On 2020/12/08 06:28:50, <A....@cognizant.com> wrote:
> Hi Ning,>
>
>
>
>
>
>
>
> It did not worked. Here are the logs from the replicator and mirror maker 2 respectively:>
>
>
>
> Replicator:>
>
>
>
> [2020-12-08 05:11:06,611] INFO [Consumer clientId=onprem-aws-replicator-0, groupId=onprem-aws-replicator] Seeking to offset 83 for partition ONPREM.AWS.REPLICA.TOPIC.P3R3-0 (org.apache.kafka.clients.consumer.KafkaConsumer:1545)>
>
>
>
> [2020-12-08 05:11:07,113] INFO [Consumer clientId=onprem-aws-replicator-0, groupId=onprem-aws-replicator] Seeking to offset 49 for partition ONPREM.AWS.REPLICA.TOPIC.P3R3-1 (org.apache.kafka.clients.consumer.KafkaConsumer:1545)>
>
>
>
> [2020-12-08 05:11:07,615] INFO [Consumer clientId=onprem-aws-replicator-0, groupId=onprem-aws-replicator] Seeking to offset 53 for partition ONPREM.AWS.REPLICA.TOPIC.P3R3-2 (org.apache.kafka.clients.consumer.KafkaConsumer:1545)>
>
>
>
>
>
>
>
> Mirror Maker 2.0:>
>
>
>
> [2020-12-08 06:10:51,385] INFO [Consumer clientId=consumer-4, groupId=null] Seeking to offset 80 for partition ONPREM.AWS.REPLICA.TOPIC.P3R3-0 (org.apache.kafka.clients.consumer.KafkaConsumer:1545)>
>
>
>
> [2020-12-08 06:10:51.385] INFO [Consumer clientId= consumer-4, groupId=null] Seeking to offset 49 for partition ONPREM.AWS.REPLICA.TOPIC.P3R3-0 (org.apache.kafka.clients.consumer.KafkaConsumer:1545)>
>
>
>
> [2020-12-08 06:10:51.386] INFO [Consumer clientId= consumer-4, groupId=null] Seeking to offset 52 for partition ONPREM.AWS.REPLICA.TOPIC.P3R3-0 (org.apache.kafka.clients.consumer.KafkaConsumer:1545)>
>
>
>
>
>
>
>
> You can see that groupId is still null in MM2 and the offsets are previous offset meaning it will replicate those messages as well which has been already replicated by Replicator.>
>
>
>
>
>
>
>
> Thanks and regards,>
>
>
>
> Amit>
>
>
>
>
>
>
>
> -----Original Message----->
>
> From: Ning Zhang <ni...@gmail.com>>
>
> Sent: Monday, December 7, 2020 10:29 PM>
>
> To: users@kafka.apache.org>
>
> Subject: Re: RE: Maintaining same offset while migrating from Confluent Replicator to Apache Mirror Maker 2.0>
>
>
>
>
>
>
>
> [External]>
>
>
>
>
>
>
>
>
>
>
>
> Hi Amit,>
>
>
>
>
>
>
>
> After looking into a little bit, do you mind to override `connector.consumer.group`? The default consumer group may be called 'connect-MirrorSourceConnector' or similar>
>
>
>
>
>
>
>
> On 2020/12/07 03:32:30, <Am...@cognizant.com>> wrote:>
>
>
>
> > Hi Ning,>
>
>
>
> >>
>
>
>
> > Thank you for the response.>
>
>
>
> >>
>
>
>
> > I probably tried to change every possible consumer group id in all ways MM2 can run. But I went through the code and it seems consumer config is using assign method which does not requires group id to be assigned and also could not find anywhere that property being read and set.>
>
>
>
> >>
>
>
>
> > One crude way we found and is in testing is to manually change the offset of the topic in the internal topics which MM2 reads to get the latest offset of the message.>
>
>
>
> >>
>
>
>
> > Thanks,>
>
>
>
> > Amit>
>
>
>
> >>
>
>
>
> > -----Original Message----->
>
>
>
> > From: Ning Zhang <ni...@gmail.com>>>
>
>
>
> > Sent: Monday, December 7, 2020 3:46 AM>
>
>
>
> > To: users@kafka.apache.org<ma...@kafka.apache.org>>
>
>
>
> > Subject: Re: Maintaining same offset while migrating from Confluent Replicator to Apache Mirror Maker 2.0>
>
>
>
> >>
>
>
>
> > [External]>
>
>
>
> >>
>
>
>
> >>
>
>
>
> > Hi Amit, I guess you may need to override the actual consumer group config (probably not consumer.group.id) that is used in Kafka Connect>
>
>
>
> >>
>
>
>
> > On 2020/11/26 06:47:11, <Am...@cognizant.com>> wrote:>
>
>
>
> > > Hi All,>
>
>
>
> > >>
>
>
>
> > > We are currently trying to migrate Confluent replicator to Apache Open Source Mirror Maker v2.0. We are facing an issue where the messages which are already replicated by replicator is getting replicated again when the mirror maker is started on the same topic. This should not happen as messages are getting duplicated at the target cluster. Here are more details:>
>
>
>
> > >>
>
>
>
> > >>
>
>
>
> > > 1. RCA: replicator assign a consumer group for replicating messages. This consumer group maintains the offset of the source topic. But we are not able to assign same consumer group to the Consumer config in mirror maker 2.>
>
>
>
> > >>
>
>
>
> > > 2. Mirror Maker 1.0 : working as same consumer group can be assigned in consumer.properties file and the messages are picked right after where replicator was stopped.>
>
>
>
> > >>
>
>
>
> > > 3. Tried running and configuring source.cluster.consumer.group.id in mirror maker 2.0 in all available options (in cluster mode, in connect-standalone and connect-distributed mode) but mirror maker 2.0 is assigning consumer group id as null while replicating messages.>
>
>
>
> > >>
>
>
>
> > >>
>
>
>
> > > Any pointers if anyone has done same and tried to maintain the same offset with mirror maker 2.0.>
>
>
>
> > >>
>
>
>
> > > Thanks and regards,>
>
>
>
> > > Amit>
>
>
>
> > > This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.>
>
>
>
> > >>
>
>
>
> > This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.>
>
>
>
> >>
>
>
>
> This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.>
>
>
Re: RE: Maintaining same offset while migrating from Confluent Replicator to Apache Mirror Maker 2.0
Posted by Ning Zhang <ni...@gmail.com>.
Hello Samruddhi,
if you want to migrate from mm1 to mm2, that tool will not work, since it was made for confluent replicator to mm2 for experimental purposes.
But the high-level idea should work, since some people solved the mirrormaker migration problem by using the method mentioned in the article.
https://lists.apache.org/thread.html/r928922036031df0db11a873ac076dae071a57a7f638bcb5911d34580%40%3Cusers.kafka.apache.org%3E
On 2021/04/05 09:24:28, Samruddhi Naik <sa...@gmail.com> wrote:
> Hello Ning,
> I was following this email thread and read through your article. I tried executing your script which I found here- https://github.com/ning2008wisc/mm2-migration but to my dismay that did not work. Me and my colleague would really like some help on this one. We still see the messages which were replicated back on the same topic when we transition from mm1 to mm2. Any help will be appreciated. Looking forward to your response.
>
> Thanks,
> Samruddhi
>
>
> On 2020/12/08 06:28:50, <A....@cognizant.com> wrote:
> > Hi Ning,>
> >
> >
> >
> >
> >
> >
> >
> > It did not worked. Here are the logs from the replicator and mirror maker 2 respectively:>
> >
> >
> >
> > Replicator:>
> >
> >
> >
> > [2020-12-08 05:11:06,611] INFO [Consumer clientId=onprem-aws-replicator-0, groupId=onprem-aws-replicator] Seeking to offset 83 for partition ONPREM.AWS.REPLICA.TOPIC.P3R3-0 (org.apache.kafka.clients.consumer.KafkaConsumer:1545)>
> >
> >
> >
> > [2020-12-08 05:11:07,113] INFO [Consumer clientId=onprem-aws-replicator-0, groupId=onprem-aws-replicator] Seeking to offset 49 for partition ONPREM.AWS.REPLICA.TOPIC.P3R3-1 (org.apache.kafka.clients.consumer.KafkaConsumer:1545)>
> >
> >
> >
> > [2020-12-08 05:11:07,615] INFO [Consumer clientId=onprem-aws-replicator-0, groupId=onprem-aws-replicator] Seeking to offset 53 for partition ONPREM.AWS.REPLICA.TOPIC.P3R3-2 (org.apache.kafka.clients.consumer.KafkaConsumer:1545)>
> >
> >
> >
> >
> >
> >
> >
> > Mirror Maker 2.0:>
> >
> >
> >
> > [2020-12-08 06:10:51,385] INFO [Consumer clientId=consumer-4, groupId=null] Seeking to offset 80 for partition ONPREM.AWS.REPLICA.TOPIC.P3R3-0 (org.apache.kafka.clients.consumer.KafkaConsumer:1545)>
> >
> >
> >
> > [2020-12-08 06:10:51.385] INFO [Consumer clientId= consumer-4, groupId=null] Seeking to offset 49 for partition ONPREM.AWS.REPLICA.TOPIC.P3R3-0 (org.apache.kafka.clients.consumer.KafkaConsumer:1545)>
> >
> >
> >
> > [2020-12-08 06:10:51.386] INFO [Consumer clientId= consumer-4, groupId=null] Seeking to offset 52 for partition ONPREM.AWS.REPLICA.TOPIC.P3R3-0 (org.apache.kafka.clients.consumer.KafkaConsumer:1545)>
> >
> >
> >
> >
> >
> >
> >
> > You can see that groupId is still null in MM2 and the offsets are previous offset meaning it will replicate those messages as well which has been already replicated by Replicator.>
> >
> >
> >
> >
> >
> >
> >
> > Thanks and regards,>
> >
> >
> >
> > Amit>
> >
> >
> >
> >
> >
> >
> >
> > -----Original Message----->
> >
> > From: Ning Zhang <ni...@gmail.com>>
> >
> > Sent: Monday, December 7, 2020 10:29 PM>
> >
> > To: users@kafka.apache.org>
> >
> > Subject: Re: RE: Maintaining same offset while migrating from Confluent Replicator to Apache Mirror Maker 2.0>
> >
> >
> >
> >
> >
> >
> >
> > [External]>
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Hi Amit,>
> >
> >
> >
> >
> >
> >
> >
> > After looking into a little bit, do you mind to override `connector.consumer.group`? The default consumer group may be called 'connect-MirrorSourceConnector' or similar>
> >
> >
> >
> >
> >
> >
> >
> > On 2020/12/07 03:32:30, <Am...@cognizant.com>> wrote:>
> >
> >
> >
> > > Hi Ning,>
> >
> >
> >
> > >>
> >
> >
> >
> > > Thank you for the response.>
> >
> >
> >
> > >>
> >
> >
> >
> > > I probably tried to change every possible consumer group id in all ways MM2 can run. But I went through the code and it seems consumer config is using assign method which does not requires group id to be assigned and also could not find anywhere that property being read and set.>
> >
> >
> >
> > >>
> >
> >
> >
> > > One crude way we found and is in testing is to manually change the offset of the topic in the internal topics which MM2 reads to get the latest offset of the message.>
> >
> >
> >
> > >>
> >
> >
> >
> > > Thanks,>
> >
> >
> >
> > > Amit>
> >
> >
> >
> > >>
> >
> >
> >
> > > -----Original Message----->
> >
> >
> >
> > > From: Ning Zhang <ni...@gmail.com>>>
> >
> >
> >
> > > Sent: Monday, December 7, 2020 3:46 AM>
> >
> >
> >
> > > To: users@kafka.apache.org<ma...@kafka.apache.org>>
> >
> >
> >
> > > Subject: Re: Maintaining same offset while migrating from Confluent Replicator to Apache Mirror Maker 2.0>
> >
> >
> >
> > >>
> >
> >
> >
> > > [External]>
> >
> >
> >
> > >>
> >
> >
> >
> > >>
> >
> >
> >
> > > Hi Amit, I guess you may need to override the actual consumer group config (probably not consumer.group.id) that is used in Kafka Connect>
> >
> >
> >
> > >>
> >
> >
> >
> > > On 2020/11/26 06:47:11, <Am...@cognizant.com>> wrote:>
> >
> >
> >
> > > > Hi All,>
> >
> >
> >
> > > >>
> >
> >
> >
> > > > We are currently trying to migrate Confluent replicator to Apache Open Source Mirror Maker v2.0. We are facing an issue where the messages which are already replicated by replicator is getting replicated again when the mirror maker is started on the same topic. This should not happen as messages are getting duplicated at the target cluster. Here are more details:>
> >
> >
> >
> > > >>
> >
> >
> >
> > > >>
> >
> >
> >
> > > > 1. RCA: replicator assign a consumer group for replicating messages. This consumer group maintains the offset of the source topic. But we are not able to assign same consumer group to the Consumer config in mirror maker 2.>
> >
> >
> >
> > > >>
> >
> >
> >
> > > > 2. Mirror Maker 1.0 : working as same consumer group can be assigned in consumer.properties file and the messages are picked right after where replicator was stopped.>
> >
> >
> >
> > > >>
> >
> >
> >
> > > > 3. Tried running and configuring source.cluster.consumer.group.id in mirror maker 2.0 in all available options (in cluster mode, in connect-standalone and connect-distributed mode) but mirror maker 2.0 is assigning consumer group id as null while replicating messages.>
> >
> >
> >
> > > >>
> >
> >
> >
> > > >>
> >
> >
> >
> > > > Any pointers if anyone has done same and tried to maintain the same offset with mirror maker 2.0.>
> >
> >
> >
> > > >>
> >
> >
> >
> > > > Thanks and regards,>
> >
> >
> >
> > > > Amit>
> >
> >
> >
> > > > This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.>
> >
> >
> >
> > > >>
> >
> >
> >
> > > This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.>
> >
> >
> >
> > >>
> >
> >
> >
> > This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.>
> >
> >