Kafka MirrorMaker - target or source datacenter deployment

[Vu Nguyen <vu...@gmail.com>]

Many of the descriptions and diagrams online describe deploying Kafka
MirrorMaker into the target data center (near the target Kafka cluster).
Since MirrorMaker is supposed to not lose messages, does it matter which
data center MirrorMaker is deployed in--source or target data center (with
any Kafka MirrorMaker version 0.10.1+)?

It might be easier to collect and observe metrics in the source data center
if MirrorMaker is also in the source data center (near the source Kafka
cluster), especially if I can't aggregate the metrics from the 2 data
centers.  Is there anything else that would influence me to deploy in
MirrorMaker in either data center?

Thanks

Vu

Re: Kafka MirrorMaker - target or source datacenter deployment

[Todd Palino <tp...@gmail.com>]

Always in the target datacenter. While you can set up mirror maker for no
data loss operation, it’s still a good idea to put the connection more
likely to fail (remote) on the consumer side. Additionally, there are
significant performance problems with setting it up for remote produce as
you must run with one in flight batch in order to maintain message ordering.

-Todd


On Thu, Sep 14, 2017 at 9:46 PM, Vu Nguyen <vu...@gmail.com> wrote:

> Many of the descriptions and diagrams online describe deploying Kafka
> MirrorMaker into the target data center (near the target Kafka cluster).
> Since MirrorMaker is supposed to not lose messages, does it matter which
> data center MirrorMaker is deployed in--source or target data center (with
> any Kafka MirrorMaker version 0.10.1+)?
>
> It might be easier to collect and observe metrics in the source data center
> if MirrorMaker is also in the source data center (near the source Kafka
> cluster), especially if I can't aggregate the metrics from the 2 data
> centers.  Is there anything else that would influence me to deploy in
> MirrorMaker in either data center?
>
> Thanks
>
> Vu
>



-- 
*Todd Palino*
Senior Staff Engineer, Site Reliability
Data Infrastructure Streaming



linkedin.com/in/toddpalino