You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Dmitri Pavlov <dp...@perforce.com> on 2022/03/03 14:59:35 UTC
Is MirrorMaker 2 horizontally scalable?
Hi,
A quick question, maybe you can help?
Trying to follow this article https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0 -> "Walkthrough: Running MirrorMaker 2.0", and the last lines in the paragraph are
======
Second, launch one or more MirrorMaker cluster nodes:
$ ./bin/connect-mirror-maker.sh mm2.properties
======
But apparently it does not work this way, one of 2 simultaneously started instance will remain idle, confirmed with Jconsole -> Mbeans.
Setup: Broker A (in cluster A) -> MM2 2 instances -> Broker B (cluster B), for simplicity there is only one broker per cluster.
A simple experiment is, when one instance is configured and started to replicate topic A only and another topic B only, only one topic will be replicated, when 2 instances are running in parallel. While, if only one of the instances is running at a time, each topic will be replicated correctly.
The main question -> is Mirrormaker 2 horizontally scalable? And if yes, would be possible to share a link to a document that describes the setup process?
Thanks in advance,
Dmitri.
This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
RE: Is MirrorMaker 2 horizontally scalable?
Posted by Dmitri Pavlov <dp...@perforce.com>.
Thank you very much Chris!
-----Original Message-----
From: Chris Egerton <fe...@gmail.com>
Sent: Thursday, March 3, 2022 5:30 PM
To: users@kafka.apache.org
Subject: Re: Is MirrorMaker 2 horizontally scalable?
Hi Dmitri,
There's at least one issue with MirrorMaker 2 that impacts horizontal scalability and has not yet been addressed:
https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FKAFKA-9981&data=04%7C01%7Cdpavlov%40perforce.com%7C53b6338153194b386f5408d9fd2abb2b%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C637819183008636777%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=EKpH5nmW6OljbRz8HSUlrSp8f6p%2FZBeRPV6GpEJlLcU%3D&reserved=0. There is some work in progress to fix it ( https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-710%253A%2BFull%2Bsupport%2Bfor%2Bdistributed%2Bmode%2Bin%2Bdedicated%2BMirrorMaker%2B2.0%2Bclusters&data=04%7C01%7Cdpavlov%40perforce.com%7C53b6338153194b386f5408d9fd2abb2b%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C637819183008636777%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=6jTfhFhh3shJaS%2BPTTs5NJhFZ8uEXQ%2B1vlQfePecE08%3D&reserved=0),
but the effort hasn't received much attention to date.
There may be other issues as well, but until KAFKA-9981 is resolved, running MirrorMaker 2 in a multi-node cluster will be at best difficult and at worst, impossible.
Cheers,
Chris
On Thu, Mar 3, 2022 at 10:00 AM Dmitri Pavlov <dp...@perforce.com> wrote:
> Hi,
>
> A quick question, maybe you can help?
>
> Trying to follow this article
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwik
> i.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-382%253A%2BMirrorMak
> er%2B2.0&data=04%7C01%7Cdpavlov%40perforce.com%7C53b6338153194b386
> f5408d9fd2abb2b%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C637819183
> 008636777%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIi
> LCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2FrA3yOCYijD6gP7lUDiD
> 6SutEg3cN%2Bo3Dywlr%2BIZsdk%3D&reserved=0
> -> "Walkthrough: Running MirrorMaker 2.0", and the last lines in the
> paragraph are
>
> ======
> Second, launch one or more MirrorMaker cluster nodes:
> $ ./bin/connect-mirror-maker.sh mm2.properties ======
>
> But apparently it does not work this way, one of 2 simultaneously
> started instance will remain idle, confirmed with Jconsole -> Mbeans.
> Setup: Broker A (in cluster A) -> MM2 2 instances -> Broker B (cluster
> B), for simplicity there is only one broker per cluster.
> A simple experiment is, when one instance is configured and started to
> replicate topic A only and another topic B only, only one topic will
> be replicated, when 2 instances are running in parallel. While, if
> only one of the instances is running at a time, each topic will be replicated correctly.
>
> The main question -> is Mirrormaker 2 horizontally scalable? And if
> yes, would be possible to share a link to a document that describes
> the setup process?
>
> Thanks in advance,
> Dmitri.
>
>
> This e-mail may contain information that is privileged or
> confidential. If you are not the intended recipient, please delete the
> e-mail and any attachments and notify us immediately.
>
>
CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
Re: Is MirrorMaker 2 horizontally scalable?
Posted by Chris Egerton <fe...@gmail.com>.
Hi Dmitri,
There's at least one issue with MirrorMaker 2 that impacts horizontal
scalability and has not yet been addressed:
https://issues.apache.org/jira/browse/KAFKA-9981. There is some work in
progress to fix it (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-710%3A+Full+support+for+distributed+mode+in+dedicated+MirrorMaker+2.0+clusters),
but the effort hasn't received much attention to date.
There may be other issues as well, but until KAFKA-9981 is resolved,
running MirrorMaker 2 in a multi-node cluster will be at best difficult and
at worst, impossible.
Cheers,
Chris
On Thu, Mar 3, 2022 at 10:00 AM Dmitri Pavlov <dp...@perforce.com> wrote:
> Hi,
>
> A quick question, maybe you can help?
>
> Trying to follow this article
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0
> -> "Walkthrough: Running MirrorMaker 2.0", and the last lines in the
> paragraph are
>
> ======
> Second, launch one or more MirrorMaker cluster nodes:
> $ ./bin/connect-mirror-maker.sh mm2.properties
> ======
>
> But apparently it does not work this way, one of 2 simultaneously started
> instance will remain idle, confirmed with Jconsole -> Mbeans.
> Setup: Broker A (in cluster A) -> MM2 2 instances -> Broker B (cluster B),
> for simplicity there is only one broker per cluster.
> A simple experiment is, when one instance is configured and started to
> replicate topic A only and another topic B only, only one topic will be
> replicated, when 2 instances are running in parallel. While, if only one of
> the instances is running at a time, each topic will be replicated correctly.
>
> The main question -> is Mirrormaker 2 horizontally scalable? And if yes,
> would be possible to share a link to a document that describes the setup
> process?
>
> Thanks in advance,
> Dmitri.
>
>
> This e-mail may contain information that is privileged or confidential. If
> you are not the intended recipient, please delete the e-mail and any
> attachments and notify us immediately.
>
>