You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Sarita (Jira)" <ji...@apache.org> on 2020/11/02 13:08:01 UTC
[jira] [Comment Edited] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17224651#comment-17224651 ]
Sarita edited comment on KAFKA-7500 at 11/2/20, 1:07 PM:
---------------------------------------------------------
Hi [~ryannedolan]
We are trying to set up MM2 with connection distributed setup.
Worker configs are as follows
```
bootstrap.servers=abc-broker:9093
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="superuser" password="superuser_password";
group.id=connect-tails
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
offset.storage.topic=connect-offsets-test
config.storage.topic=connect-configs-test
status.storage.topic=connect-status-test
offset.flush.interval.ms=300000
producer.buffer.memory=1234
producer.ssl.truststore.password=truststore_password
producer.ssl.truststore.location=/opt/projects/confluent/wildcard.kafka.iggroup.local.jks
producer.ssl.keystore.password=keystore_password
producer.ssl.keystore.location=/opt/projects/confluent/wildcard.kafka.iggroup.local.jks
```
When i run below command, I can see worker connector created:
```nohup sh connect-distributed ../etc/kafka/connect-distributed-1.properties &```
When I try to create a connector using POST call, I start seeing message as
```WARN [Producer clientId=connector-producer-MM9-0] Bootstrap broker abc-broker:9093 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient:1037)```
json content for POST call is as below
```
{{ "name": "MM9",
"config":
{
"connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
"tasks.max": 3,
"topics": "messaging_ops_mm8",
"errors.log.enable": true,
"errors.log.include.messages": true,
"key.converter": "org.apache.kafka.connect.converters.ByteArrayConverter", "value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
"clusters": "xyz-broker, abc-broker",
"source.cluster.alias": "xyz-broker",
"target.cluster.alias": "abc-broker",
"source.cluster.bootstrap.servers": "xyz-broker:9093",
"source.cluster.security.protocol": "SASL_SSL",
"source.cluster.sasl.mechanism": "PLAIN",
"source.cluster.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username='superuser' password='superuser_password';",
"source.cluster.ssl.truststore.password" : "truststore_password", "source.cluster.ssl.truststore.location" : "/opt/projects/confluent/wildcard.kafka.iggroup.local.jks", "source.cluster.ssl.keystore.password" : "keystore_password", "source.cluster.ssl.keystore.location" : "/opt/projects/confluent/ssl/wildcard.kafka.iggroup.local.jks", "target.cluster.bootstrap.servers": "abc-broker:9093",
"target.cluster.security.protocol": "SASL_SSL",
"target.cluster.sasl.mechanism": "PLAIN",
"target.cluster.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username='superuser' password='superuser_password';",
"target.cluster.ssl.truststore.password" : "truststore_password", "target.cluster.ssl.truststore.location" : "/opt/projects/confluent/ssl/wildcard.kafka.iggroup.local.jks", "target.cluster.ssl.keystore.password" : "keystore_password", "target.cluster.ssl.keystore.location" : "/opt/projects/confluent/ssl/wildcard.kafka.iggroup.local.jks" }
}
```
Note the bootstrap-server added in connect-distributed.properties file and the target cluster bootstrap-server are same(abc-broker).
I have added all the SASL credentials, did a swap of source and destination cluster in the json but I continue to get this broker disconnected WARNING.
We are on kafka version 0.11.0.3.
What is it that we are missing?
was (Author: saritago):
Hi [~ryannedolan]
We are trying to set up MM2 with connection distributed setup.
Worker configs are as follows
```
bootstrap.servers=abc-broker:9093
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="superuser" password="superuser_password";
group.id=connect-tails
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
offset.storage.topic=connect-offsets-test
config.storage.topic=connect-configs-test
status.storage.topic=connect-status-test
offset.flush.interval.ms=300000
producer.buffer.memory=1234
producer.ssl.truststore.password=truststore_password
producer.ssl.truststore.location=/opt/projects/confluent/wildcard.kafka.iggroup.local.jks
producer.ssl.keystore.password=keystore_password
producer.ssl.keystore.location=/opt/projects/confluent/wildcard.kafka.iggroup.local.jks
```
When i run below command, I can see worker connector created:
```nohup sh connect-distributed ../etc/kafka/connect-distributed-1.properties &```
When I try to create a connector using POST call, I start seeing message as
```WARN [Producer clientId=connector-producer-MM9-0] Bootstrap broker abc-broker:9093 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient:1037)```
json content for POST call is as below
```
{{ "name": "MM9",
"config":
{ "connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
"tasks.max": 3,
"topics": "messaging_ops_mm8",
"errors.log.enable": true,
"errors.log.include.messages": true,
"key.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
"value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
"clusters": "xyz-broker, abc-broker",
"source.cluster.alias": "xyz-broker",
"target.cluster.alias": "abc-broker",
"source.cluster.bootstrap.servers": "xyz-broker:9093",
"source.cluster.security.protocol": "SASL_SSL",
"source.cluster.sasl.mechanism": "PLAIN",
"source.cluster.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username='superuser' password='superuser_password';",
"source.cluster.ssl.truststore.password" : "truststore_password",
"source.cluster.ssl.truststore.location" : "/opt/projects/confluent/wildcard.kafka.iggroup.local.jks", "source.cluster.ssl.keystore.password" : "keystore_password",
"source.cluster.ssl.keystore.location" : "/opt/projects/confluent/ssl/wildcard.kafka.iggroup.local.jks", "target.cluster.bootstrap.servers": "abc-broker:9093",
"target.cluster.security.protocol": "SASL_SSL",
"target.cluster.sasl.mechanism": "PLAIN",
"target.cluster.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username='superuser' password='superuser_password';",
"target.cluster.ssl.truststore.password" : "truststore_password",
"target.cluster.ssl.truststore.location" : "/opt/projects/confluent/ssl/wildcard.kafka.iggroup.local.jks", "target.cluster.ssl.keystore.password" : "keystore_password",
"target.cluster.ssl.keystore.location" : "/opt/projects/confluent/ssl/wildcard.kafka.iggroup.local.jks"
}
}
```
Note the bootstrap-server added in connect-distributed.properties file and the target cluster bootstrap-server are same(abc-broker).
I have added all the SASL credentials, did a swap of source and destination cluster in the json but I continue to get this broker disconnected WARNING.
We are on kafka version 0.11.0.3.
What is it that we are missing?
> MirrorMaker 2.0 (KIP-382)
> -------------------------
>
> Key: KAFKA-7500
> URL: https://issues.apache.org/jira/browse/KAFKA-7500
> Project: Kafka
> Issue Type: New Feature
> Components: KafkaConnect, mirrormaker
> Affects Versions: 2.4.0
> Reporter: Ryanne Dolan
> Assignee: Ryanne Dolan
> Priority: Major
> Labels: pull-request-available, ready-to-commit
> Fix For: 2.4.0
>
> Attachments: Active-Active XDCR setup.png
>
>
> Implement a drop-in replacement for MirrorMaker leveraging the Connect framework.
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0]
> [https://github.com/apache/kafka/pull/6295]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)