You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2020/08/27 13:48:12 UTC
[GitHub] [kafka] mimaison commented on a change in pull request #9215: KAFKA-10133: MM2 readme update on config
mimaison commented on a change in pull request #9215:
URL: https://github.com/apache/kafka/pull/9215#discussion_r478425022
##########
File path: connect/mirror/README.md
##########
@@ -141,7 +141,38 @@ nearby clusters.
N.B. that the `--clusters` parameter is not technically required here. MM2 will work fine without it; however, throughput may suffer from "producer lag" between
data centers, and you may incur unnecessary data transfer costs.
-## Shared configuration
+## Configuration
+### General Kafka Connect Config
+All Kafka Connect, Source Connector, Sink Connector configs, as defined in [Kafka official doc] (https://kafka.apache.org/documentation/#connectconfigs), can be
+directly used in MM2 configuration without prefix in the configuration name. As the starting point, most of these default configs may work well with the exception of `tasks.max`.
+
+In order to evenly distribute the workload across more than one MM2 instance, it is advised to set `tasks.max` at least to 2 or even larger depending on the hardware resources
+and the total number partitions to be replicated.
+
+### Kafka Connect Config for a Specific Connector
+If needed, Kafka Connect worker-level configs could be even specified "per connector", which needs to follow the format of `cluster_alias.config_name` in MM2 configuration. For example,
+
+ backup.ssl.truststore.location = /usr/lib/jvm/zulu-8-amd64/jre/lib/security/cacerts // SSL cert location
+ backup.security.protocol = SSL // if target cluster needs SSL to send message
+
+### MM2 configs for a Specific Connector
+MM2 itself has many configs to control how MM2 behaves. To override those default values, add the config name by the format of `source_cluster_alias->target_cluster_alias.config_name` in MM2 configuration. For example,
+
+ backup->primary.enabled = false // set to false if one-way replication is desired
+ primary->backup.topics.blacklist = topics_to_blacklist
+ primary->backup.emit.heartbeats.enabled = false
+ primary->backup.sync.group.offsets = true
+
+### Producer / Consumer Config used by MM2
Review comment:
Can we also cover Admin here?
##########
File path: connect/mirror/README.md
##########
@@ -141,7 +141,38 @@ nearby clusters.
N.B. that the `--clusters` parameter is not technically required here. MM2 will work fine without it; however, throughput may suffer from "producer lag" between
data centers, and you may incur unnecessary data transfer costs.
-## Shared configuration
+## Configuration
Review comment:
We should mention this describes configuring MM2 in driver mode?
##########
File path: connect/mirror/README.md
##########
@@ -141,7 +141,38 @@ nearby clusters.
N.B. that the `--clusters` parameter is not technically required here. MM2 will work fine without it; however, throughput may suffer from "producer lag" between
data centers, and you may incur unnecessary data transfer costs.
-## Shared configuration
+## Configuration
+### General Kafka Connect Config
+All Kafka Connect, Source Connector, Sink Connector configs, as defined in [Kafka official doc] (https://kafka.apache.org/documentation/#connectconfigs), can be
Review comment:
The link does not render correctly, remove the space between `] (`
##########
File path: connect/mirror/README.md
##########
@@ -141,7 +141,38 @@ nearby clusters.
N.B. that the `--clusters` parameter is not technically required here. MM2 will work fine without it; however, throughput may suffer from "producer lag" between
data centers, and you may incur unnecessary data transfer costs.
-## Shared configuration
+## Configuration
+### General Kafka Connect Config
+All Kafka Connect, Source Connector, Sink Connector configs, as defined in [Kafka official doc] (https://kafka.apache.org/documentation/#connectconfigs), can be
+directly used in MM2 configuration without prefix in the configuration name. As the starting point, most of these default configs may work well with the exception of `tasks.max`.
+
+In order to evenly distribute the workload across more than one MM2 instance, it is advised to set `tasks.max` at least to 2 or even larger depending on the hardware resources
+and the total number partitions to be replicated.
+
+### Kafka Connect Config for a Specific Connector
+If needed, Kafka Connect worker-level configs could be even specified "per connector", which needs to follow the format of `cluster_alias.config_name` in MM2 configuration. For example,
+
+ backup.ssl.truststore.location = /usr/lib/jvm/zulu-8-amd64/jre/lib/security/cacerts // SSL cert location
+ backup.security.protocol = SSL // if target cluster needs SSL to send message
+
+### MM2 configs for a Specific Connector
+MM2 itself has many configs to control how MM2 behaves. To override those default values, add the config name by the format of `source_cluster_alias->target_cluster_alias.config_name` in MM2 configuration. For example,
Review comment:
We can replace the 2nd `MM2` by `it`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org