Kafka Mirror Maker 2

[Himanshu Tyagi <ty...@gmail.com>]

Hey Team,
I've a few doubts regarding how the producers work after failover in Mirror
Maker 2

Let's say that we have two clusters K1 and K2 and configured MM2
replication for TOPIC1 (originally created in just K1).

We configured the active-active replication:

K1->K2.enabled = true
K2->K1.enabled = true
K1->K2.topics = .*
K2->K1.topics = .*

On starting mirror maker 2, I see that topics are replicated from cluster K1
to K2 in the naming format K1.topic_name_here and vice versa for topics
from cluster K2 to K1.

I see that there was no topic TOPIC1 created in K2, only K1.TOPIC1 was
created. I see this scenario working for consumers, as in the beginning the
consumers consume TOPIC1 from cluster K1. When the cluster K1 out of
service, fail over happens. Consumers start to consume K1.TOPIC1 from K2.

My questions are as follows:

   1. For producers, they won't producer to the topic K1.TOPIC1 in cluster
   K2, my question is how do the producers go about producing data. Do I need
   to manually create a topic TOPIC1 in cluster K2 and this topic will be
   used for producing messages to when failover happens.
   2. If the above scenario is true, how do I move back to my primary
   cluster K1. As, now the topic TOPIC1 in cluster K2 has digressed from
   the topic TOPIC in K1. How do we sync the messages in this scenario ?


-- 
Best,
Himanshu Tyagi
Contact No.- +1 480 465 0625

Re: Kafka Mirror Maker 2

[Ryanne Dolan <ry...@gmail.com>]

> 1. ... Do I need to manually create a topic TOPIC1 in cluster K2 and this
topic will be used for producing messages to when failover happens.

Correct. When you fail-over producers from K1 to K2, producers will send to
TOPIC1 in K2. You may need to manually create that topic and set up write
ACLs, etc. MM2 will automatically create and configure "remote topics",
including ACLs -- but it won't touch "normal" topics. I recommend you set
this up ahead of time.

> 2. .. how do I move back to my primary

The operation is the same, just in reverse. Any data produced to TOPIC1 in
K2 will get replicated to K2.TOPIC1 in K1. A consumer in K1 can then
consume from TOPIC1 _and_ K2.TOPIC1 in order to see records that were
produced to K2 during failover.

Ryanne

On Wed, Apr 29, 2020 at 3:05 PM Himanshu Tyagi <ty...@gmail.com>
wrote:

> Hey Team,
> I've a few doubts regarding how the producers work after failover in Mirror
> Maker 2
>
> Let's say that we have two clusters K1 and K2 and configured MM2
> replication for TOPIC1 (originally created in just K1).
>
> We configured the active-active replication:
>
> K1->K2.enabled = true
> K2->K1.enabled = true
> K1->K2.topics = .*
> K2->K1.topics = .*
>
> On starting mirror maker 2, I see that topics are replicated from cluster
> K1
> to K2 in the naming format K1.topic_name_here and vice versa for topics
> from cluster K2 to K1.
>
> I see that there was no topic TOPIC1 created in K2, only K1.TOPIC1 was
> created. I see this scenario working for consumers, as in the beginning the
> consumers consume TOPIC1 from cluster K1. When the cluster K1 out of
> service, fail over happens. Consumers start to consume K1.TOPIC1 from K2.
>
> My questions are as follows:
>
>    1. For producers, they won't producer to the topic K1.TOPIC1 in cluster
>    K2, my question is how do the producers go about producing data. Do I
> need
>    to manually create a topic TOPIC1 in cluster K2 and this topic will be
>    used for producing messages to when failover happens.
>    2. If the above scenario is true, how do I move back to my primary
>    cluster K1. As, now the topic TOPIC1 in cluster K2 has digressed from
>    the topic TOPIC in K1. How do we sync the messages in this scenario ?
>
>
> --
> Best,
> Himanshu Tyagi
> Contact No.- +1 480 465 0625
>

Re: Kafka Mirror Maker 2

[vishnu murali <vi...@gmail.com>]

Hi Himanshu

Can u pls tell how to use MM2..

I am using Apache Kafka,in this normal mirror maker is only available..

Most of the people saying like to use MM2 but I didn't able to know where
to get that MM2.

Is it related to Apache or from some other Distributors?

Can u pls explain how to install that version

On Thu, Apr 30, 2020, 01:35 Himanshu Tyagi <ty...@gmail.com>
wrote:

> Hey Team,
> I've a few doubts regarding how the producers work after failover in Mirror
> Maker 2
>
> Let's say that we have two clusters K1 and K2 and configured MM2
> replication for TOPIC1 (originally created in just K1).
>
> We configured the active-active replication:
>
> K1->K2.enabled = true
> K2->K1.enabled = true
> K1->K2.topics = .*
> K2->K1.topics = .*
>
> On starting mirror maker 2, I see that topics are replicated from cluster
> K1
> to K2 in the naming format K1.topic_name_here and vice versa for topics
> from cluster K2 to K1.
>
> I see that there was no topic TOPIC1 created in K2, only K1.TOPIC1 was
> created. I see this scenario working for consumers, as in the beginning the
> consumers consume TOPIC1 from cluster K1. When the cluster K1 out of
> service, fail over happens. Consumers start to consume K1.TOPIC1 from K2.
>
> My questions are as follows:
>
>    1. For producers, they won't producer to the topic K1.TOPIC1 in cluster
>    K2, my question is how do the producers go about producing data. Do I
> need
>    to manually create a topic TOPIC1 in cluster K2 and this topic will be
>    used for producing messages to when failover happens.
>    2. If the above scenario is true, how do I move back to my primary
>    cluster K1. As, now the topic TOPIC1 in cluster K2 has digressed from
>    the topic TOPIC in K1. How do we sync the messages in this scenario ?
>
>
> --
> Best,
> Himanshu Tyagi
> Contact No.- +1 480 465 0625
>