You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Mark Flores <ma...@expeditors.com> on 2015/03/09 22:04:09 UTC
Kafka Questions
One of our development teams is considering implementing a Kafka solution. If the development team were to assume implementing 6 separate regional Kafka clusters:
* How could we implement global Pub/Sub between clusters?
* Can we do bi-directional replication with MirrorMaker between each cluster?
* How do we prevent a message published at one cluster being replicated to another cluster and into an infinite loop (assuming we use a MirrorMaker whitelist like Global.* at each cluster)?
Regards,
Mark Flores
Project Manager, Enterprise Technology
Direct 206-576-2675
Email mark.flores@expeditors.com
[expd]
Global Headquarters, Seattle
1015 Third Avenue, 12th Floor
Seattle, WA 98104
Re: Kafka Questions
Posted by Jiangjie Qin <jq...@linkedin.com.INVALID>.
Hi Mark,
For global Pub/Sub between clusters, I think you might need another layer of service to direct users to the right Kafka cluster.
Yes, mirror maker could be used for cross colo replication. Currently mirror maker cannot be used for bi-direction mirror yet, after KAFKA-1997 get checked in, you can use the message handler to do filtering so you will not have the infinite loop.
Jiangjie (Becket) Qin
From: Mark Flores <ma...@expeditors.com>>
Reply-To: "users@kafka.apache.org<ma...@kafka.apache.org>" <us...@kafka.apache.org>>
Date: Monday, March 9, 2015 at 1:04 PM
To: "users@kafka.apache.org<ma...@kafka.apache.org>" <us...@kafka.apache.org>>
Subject: Kafka Questions
One of our development teams is considering implementing a Kafka solution. If the development team were to assume implementing 6 separate regional Kafka clusters:
· How could we implement global Pub/Sub between clusters?
· Can we do bi-directional replication with MirrorMaker between each cluster?
· How do we prevent a message published at one cluster being replicated to another cluster and into an infinite loop (assuming we use a MirrorMaker whitelist like Global.* at each cluster)?
Regards,
Mark Flores
Project Manager, Enterprise Technology
Direct 206-576-2675
Email mark.flores@expeditors.com<ma...@expeditors.com>
[expd]
Global Headquarters, Seattle
1015 Third Avenue, 12th Floor
Seattle, WA 98104