You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Zachary Mixon <Za...@freemanco.com> on 2020/03/16 18:39:00 UTC
MirrorMaker 2 Configuration?
Hello,
I'm trying to work with MirrorMaker 2 and have been having some troubles with the configuration and my end result. As I understand, it seems there's updated documentation coming in the next Kafka release?
I've been tasked with examining MM2 as a solution to an Active->Stand-by two cluster configuration. We would like to have our active cluster constantly replicating to a disaster recovery cluster so that we can quickly fail-over if something horrible happens to the active cluster. I'm aware there's other, potentially easier, ways to solve this problem but this is what I'm tasked with trying.
The first trouble I ran into was the difference in configuration between using the Connect JSON API and trying to run in dedicated stand-alone mode. Perhaps this is better documented elsewhere but it was quite confusing for me, I ultimately ended with something resembling this:
{
"name": "efw-mirrormaker-source-nonprod-target-dr-1",
"connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
"tasks.max": 1,
"topics": "mm2\\.efw.*",
"errors.log.enable": true,
"errors.log.include.messages": true,
"key.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
"value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
"clusters": "mm2.source.nonprod, dr",
"source.cluster.alias": "mm2.source.nonprod",
"target.cluster.alias": "dr",
"source.cluster.bootstrap.servers": "kafka-bootstrap:9092",
"target.cluster.bootstrap.servers": "kafka-boostrap-dr:9092",
"source.admin.client.id": "efwmm2sourceadmin",
"source.cluster.security.protocol": "SSL",
"target.admin.client.id": "efwmm2targetadmin"
}
Which successfully ran but has some key differences to the following properties file I used to configure standalone mode:
name = efw-mirrormaker-source-nonprod-target-dr-2
tasks.max = 1
emit.heartbeats.enabled = false
sync.topics.configs.enabled = true
sync.topic.acls.enabled = true
errors.log.enable = true
errors.log.include.messages = true
key.converter = org.apache.kafka.connect.converters.ByteArrayConverter
value.converter = org.apache.kafka.connect.converters.ByteArrayConverter
clusters = mm2.source.nonprod, dr
mm2.source.nonprod.bootstrap.servers = kafka-bootstrap:9092
dr.bootstrap.servers = kafka-dr-bootstrap:9092
mm2.source.nonprod.client.id = efw-mm2-source-nonprod
dr.client.id = efw-mm2-dr
mm2.source.nonprod.security.protocol = SSL
mm2.source.nonprod.cluster.security.protocol = SSL
mm2.source.nonprod->dr.enabled = true
mm2.source.nonprod->dr.emit.heartbeats.enabled = false
mm2.source.nonprod->dr.emit.checkpoints.enabled = true
mm2.source.nonprod->dr.sync.topics.configs.enabled = true
mm2.source.nonprod->dr.sync.topics.acls.enabled = true
mm2.source.nonprod->dr.topics = mm2\\.efw.*
mm2.source.nonprod.group.id = efw-mm2-group1
Which leads me to the bulk of my questions:
First, does anybody have the differences in configuration well documented anywhere? For instance, the "A->B" properties don't seem to be functional at all on the JSON API. Additionally, configuring the admin client in JSON requires "source.cluster.{config property}" whereas in the properties file it's "{your source alias}.{config property}".
Secondly, I seem unable to turn off Heartbeats when using the dedicated stand-alone method. I've tried every combination of configuration settings I can imagine and I was never able to get it to actually stop. I mainly wanted to turn it off because it was flooding my logs due to ACL issues on the "heartbeat" topic. I managed to solve that part, but I still can't disable heartbeats, am I just misconfiguring? Has anybody else had this issue?
Thirdly, and lastly, I've managed to get MirrorMaker 2 running and mirroring data using the dedicated standalone method, but I'm receiving 3 copies of each message on my mirrored topics (the same issue mentioned in this SO post: https://stackoverflow.com/questions/60005462/kafka-mirrormaker-2-0-duplicate-each-messages). For instance, if I have the message "Foo" in my active cluster, in the DR cluster I receive 3 separate messages "Foo" "Foo" "Foo" in the same topic. This seems independent of partition, replication factors, how many brokers are running, if I'm running locally (in cluster) or remotely, and I'm quite confused as to why. Does anybody have any guidance here?
Thanks for taking a moment to read this, I'm hoping I'm not alone in these issues and I can help shine some light for others.