You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Kalimova, Julia" <Ju...@blackrock.com.INVALID> on 2022/03/23 15:15:15 UTC
Running multiple MM2 instances
Hello!
I’m wondering if it’s possible to scale up MM2 by running multiple instances per data center?
For scalability purposes, I would like to run 2 instances of connect-mirror-maker.sh in one data center. However, I cannot get 2 instances of mirror maker to work at the same time: once I start up the second mirror maker, it takes over for the first one, and the first one completely stops replicating: i.e. rather than scaling up and rebalancing, the whole workload is still handled by a single instance.
What am I missing here? Would greatly appreciate any guidance with this! Thank you!
This message may contain information that is confidential or privileged. If you are not the intended recipient, please advise the sender immediately and delete this message. See http://www.blackrock.com/corporate/compliance/email-disclaimers for further information. Please refer to http://www.blackrock.com/corporate/compliance/privacy-policy for more information about BlackRock’s Privacy Policy.
For a list of BlackRock's office addresses worldwide, see http://www.blackrock.com/corporate/about-us/contacts-locations.
© 2022 BlackRock, Inc. All rights reserved.
Re: Running multiple MM2 instances
Posted by Chris Egerton <fe...@gmail.com>.
Hi Julia,
Sounds like KAFKA-9981 [1]. This is a known issue with MirrorMaker 2 that
impacts horizontal scalability and has not yet been addressed. There is
some work in progress to fix this issue [2], but the effort hasn't received
much attention to date. There may be other issues as well, but until
KAFKA-9981 is addressed, running MirrorMaker 2 in a multi-node cluster will
be at best difficult and worst, impossible.
[1] - https://issues.apache.org/jira/browse/KAFKA-9981
[2] -
https://cwiki.apache.org/confluence/display/KAFKA/KIP-710%3A+Full+support+for+distributed+mode+in+dedicated+MirrorMaker+2.0+clusters
Cheers,
Chris
On Wed, Mar 23, 2022 at 11:15 AM Kalimova, Julia
<Ju...@blackrock.com.invalid> wrote:
>
>
> Hello!
>
>
>
> I’m wondering if it’s possible to scale up MM2 by running multiple
> instances per data center?
>
> For scalability purposes, I would like to run 2 instances of
> connect-mirror-maker.sh in one data center. However, I cannot get 2
> instances of mirror maker to work at the same time: once I start up the
> second mirror maker, it takes over for the first one, and the first one
> completely stops replicating: i.e. rather than scaling up and rebalancing,
> the whole workload is still handled by a single instance.
>
> What am I missing here? Would greatly appreciate any guidance with this!
> Thank you!
>
>
> This message may contain information that is confidential or privileged.
> If you are not the intended recipient, please advise the sender immediately
> and delete this message. See
> http://www.blackrock.com/corporate/compliance/email-disclaimers for
> further information. Please refer to
> http://www.blackrock.com/corporate/compliance/privacy-policy for more
> information about BlackRock’s Privacy Policy.
>
>
> For a list of BlackRock's office addresses worldwide, see
> http://www.blackrock.com/corporate/about-us/contacts-locations.
>
> © 2022 BlackRock, Inc. All rights reserved.
>