You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by dixingxing <di...@163.com> on 2020/01/16 08:18:33 UTC
How to achieve "Effectively one big consumer group" in
active/active clusters
hello everyone:
I've read this slide recently(https://www.confluent.io/kafka-summit-lon19/disaster-recovery-with-mirrormaker-2-0/), i'm not sure how to deploy active-active clusters like this.
There is no detail about how to make an "Effectively one big consumer group", since there is MM2 between the two clusters, so the topic should like this:
primary cluster:
topic1
secondary.topic1
secondary cluster:
topic1
primary.topic1
I think the simplest way to make an "Effectively one big consumer group" is :
Each client should make 2 cosumer instances, one subscribe primary.topic1, and the other subscribe secondary.topic1, so if one cluster crash down, LB just need to failover to another cluster, the consumer should do nothing, every thing will works fine.
But it seems there is no need for MM2, or MM2 is just for disaster recovery?
Am i misunderstanding about this slide?
Re:Re: How to achieve "Effectively one big consumer group" in
active/active clusters
Posted by dixingxing <di...@163.com>.
Ryanne, thanks for the reply，that just helps me a lot.
At 2020-01-16 22:27:22, "Ryanne Dolan" <ry...@gmail.com> wrote:
In this case the consumers just subscribe to "topic1" like normal, and the remote topics (primary.topic1, secondary.topic1) are just for DR. MM2 is not required for things to work under normal circumstances, but if one cluster goes down you can recover its data from the other.
Ryanne
On Thu, Jan 16, 2020, 2:35 AM dixingxing <di...@163.com> wrote:
hello everyone:
I've read this slide recently(https://www.confluent.io/kafka-summit-lon19/disaster-recovery-with-mirrormaker-2-0/), i'm not sure how to deploy active-active clusters like this.
There is no detail about how to make an "Effectively one big consumer group", since there is MM2 between the two clusters, so the topic should like this:
primary cluster:
topic1
secondary.topic1
secondary cluster:
topic1
primary.topic1
I think the simplest way to make an "Effectively one big consumer group" is :
Each client should make 2 cosumer instances, one subscribe primary.topic1, and the other subscribe secondary.topic1, so if one cluster crash down, LB just need to failover to another cluster, the consumer should do nothing, every thing will works fine.
But it seems there is no need for MM2, or MM2 is just for disaster recovery?
Am i misunderstanding about this slide?
Re: How to achieve "Effectively one big consumer group" in
active/active clusters
Posted by Ryanne Dolan <ry...@gmail.com>.
In this case the consumers just subscribe to "topic1" like normal, and the
remote topics (primary.topic1, secondary.topic1) are just for DR. MM2 is
not required for things to work under normal circumstances, but if one
cluster goes down you can recover its data from the other.
Ryanne
On Thu, Jan 16, 2020, 2:35 AM dixingxing <di...@163.com> wrote:
> hello everyone:
>
> I've read this slide recently(
> https://www.confluent.io/kafka-summit-lon19/disaster-recovery-with-mirrormaker-2-0/),
> i'm not sure how to deploy active-active clusters like this.
>
>
> There is no detail about how to make an "Effectively one big consumer
> group", since there is MM2 between the two clusters, so the topic should
> like this:
> *primary cluster:*
> topic1
> secondary.topic1
>
> *secondary cluster:*
> topic1
> primary.topic1
>
> I think the simplest way to make an "Effectively one big consumer group"
> is :
> Each client should make 2 cosumer instances, one subscribe
> primary.topic1, and the other subscribe secondary.topic1, so if one cluster
> crash down, LB just need to failover to another cluster, the consumer
> should do nothing, every thing will works fine.
> But it seems there is no need for MM2, or MM2 is just for disaster
> recovery?
> Am i misunderstanding about this slide?
>
>
>
>
>
>
>
>
>
>
>
>