You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Anamika Nadkarni (Jira)" <ji...@apache.org> on 2021/08/31 06:47:00 UTC
[jira] [Commented] (KAFKA-13255) Mirrormaker config property
config.properties.exclude is not working as expected
[ https://issues.apache.org/jira/browse/KAFKA-13255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407116#comment-17407116 ]
Anamika Nadkarni commented on KAFKA-13255:
------------------------------------------
I keep getting HTTP error 403 when trying to attach file with curl commands to this jira. As a work around I am pasting the file contents below.
{code:java}
curl -X POST http://10.xx.xx.xx:8083/connectors -H "Content-Type: application/json" -d '{
"name":"mm2-msc",
"config":{
"connector.class":"org.apache.kafka.connect.mirror.MirrorSourceConnector",
"clusters":"mysource,mydestination",
"source.cluster.alias":"mysource",
"target.cluster.alias":"mydestination",
"replication.policy.class":"com.amazonaws.kafka.samples.CustomMM2ReplicationPolicy",
"target.cluster.bootstrap.servers":"b-2.mydestination.amazonaws.com:9096,b-3.mydestination.amazonaws.com:9096,b-1.mydestination.amazonaws.com:9096",
"source.cluster.bootstrap.servers":"10.xx.xx.xx:9093,10.xx.xx.xx:9093,10.xx.xx.xx:9093,10.xx.xx.xx:9093,10.xx.xx.xx:9093",
"topics":"dev.portlandDc.anamika.helloMsk",
"config.properties.exclude":"^confluent\\..*$",
"tasks.max":"1",
"key.converter":" org.apache.kafka.connect.converters.ByteArrayConverter",
"value.converter":"org.apache.kafka.connect.converters.ByteArrayConverter",
"replication.factor":"3",
"offset-syncs.topic.replication.factor":"3",
"sync.topic.acls.interval.seconds":"20",
"sync.topic.configs.interval.seconds":"20",
"refresh.topics.interval.seconds":"20",
"refresh.groups.interval.seconds":"20",
"consumer.group.id":"mm2-msc-cons",
"producer.enable.idempotence":"true",
"source.cluster.security.protocol":"SASL_SSL",
"source.cluster.sasl.mechanism":"PLAIN",
"source.cluster.ssl.truststore.location":"/home/ec2-user/dc/kafka.client.truststore.jks",
"source.cluster.ssl.truststore.password":"xxxxx",
"source.cluster.ssl.endpoint.identification.algorithm":"",
"source.cluster.sasl.jaas.config":"org.apache.kafka.common.security.plain.PlainLoginModule required username=\"xxxxx\" password=\"xxxxx\";",
"target.cluster.security.protocol":"SASL_SSL",
"target.cluster.sasl.mechanism":"SCRAM-SHA-512",
"target.cluster.ssl.truststore.location":"/home/ec2-user/kafka_2.13-2.8.0/config/msk/kafka.client.truststore.jks",
"target.cluster.sasl.jaas.config":"org.apache.kafka.common.security.scram.ScramLoginModule required username=\"xxxxx\" password=\"xxxxx\";"
}
}' | jq .
curl -X POST http://10.xx.xx.xx:8083/connectors -H "Content-Type: application/json" -d '{
"name":"mm2-cpc",
"config":{
"connector.class":"org.apache.kafka.connect.mirror.MirrorCheckpointConnector",
"clusters":"mysource,mydestination",
"source.cluster.alias":"mysource",
"target.cluster.alias":"mydestination",
"config.properties.exclude":"confluent.value.schema.validation",
"config.properties.blacklist":"confluent.value.schema.validation",
"target.cluster.bootstrap.servers":"b-2.mydestination.amazonaws.com:9096,b-3.mydestination.amazonaws.com:9096,b-1.mydestination.amazonaws.com:9096",
"source.cluster.bootstrap.servers":"10.xx.xx.xx:9093,10.xx.xx.xx:9093,10.xx.xx.xx:9093,10.xx.xx.xx:9093,10.xx.xx.xx:9093",
"tasks.max":"1",
"key.converter":" org.apache.kafka.connect.converters.ByteArrayConverter",
"value.converter":"org.apache.kafka.connect.converters.ByteArrayConverter",
"replication.policy.class":"com.amazonaws.kafka.samples.CustomMM2ReplicationPolicy",
"replication.factor":"3",
"sync.topic.configs.enabled":"false",
"checkpoints.topic.replication.factor":"3",
"emit.checkpoints.interval.seconds":"20",
"source.cluster.security.protocol":"SASL_SSL",
"source.cluster.sasl.mechanism":"PLAIN",
"source.cluster.ssl.truststore.location":"/home/ec2-user/dc/kafka.client.truststore.jks",
"source.cluster.ssl.truststore.password":"xxxxx",
"source.cluster.ssl.endpoint.identification.algorithm":"",
"source.cluster.sasl.jaas.config":"org.apache.kafka.common.security.plain.PlainLoginModule required username=\"xxxxx\" password=\"xxxxx\";",
"target.cluster.security.protocol":"SASL_SSL",
"target.cluster.sasl.mechanism":"SCRAM-SHA-512",
"target.cluster.ssl.truststore.location":"/home/ec2-user/kafka_2.13-2.8.0/config/msk/kafka.client.truststore.jks",
"target.cluster.sasl.jaas.config":"org.apache.kafka.common.security.scram.ScramLoginModule required username=\"xxxxx\" password=\"xxxxx\";"
}
}' | jq .
curl -X POST http://10.xx.xx.xx:8083/connectors -H "Content-Type: application/json" -d '{
"name":"mm2-hbc",
"config":{
"connector.class":"org.apache.kafka.connect.mirror.MirrorHeartbeatConnector",
"clusters":"mysource,mydestination",
"source.cluster.alias":"mysource",
"target.cluster.alias":"mydestination",
"target.cluster.bootstrap.servers":"b-2.mydestination.amazonaws.com:9096,b-3.mydestination.amazonaws.com:9096,b-1.mydestination.amazonaws.com:9096",
"source.cluster.bootstrap.servers":"10.xx.xx.xx:9093,10.xx.xx.xx:9093,10.xx.xx.xx:9093,10.xx.xx.xx:9093,10.xx.xx.xx:9093",
"tasks.max":"1",
"key.converter":" org.apache.kafka.connect.converters.ByteArrayConverter",
"value.converter":"org.apache.kafka.connect.converters.ByteArrayConverter",
"replication.factor":"3",
"heartbeats.topic.replication.factor":"3",
"emit.heartbeats.interval.seconds":"20",
"source.cluster.security.protocol":"SASL_SSL",
"source.cluster.ssl.truststore.location":"/home/ec2-user/dc/kafka.client.truststore.jks",
"source.cluster.ssl.truststore.password":"xxxxx",
"source.cluster.sasl.jaas.config":"org.apache.kafka.common.security.plain.PlainLoginModule required username=\"xxxxx\" password=\"xxxxx\";",
"source.cluster.sasl.mechanism":"PLAIN",
"target.cluster.security.protocol":"SASL_SSL",
"target.cluster.ssl.truststore.location":"/home/ec2-user/kafka_2.13-2.8.0/config/msk/kafka.client.truststore.jks",
"target.cluster.sasl.mechanism":"SCRAM-SHA-512",
"target.cluster.sasl.jaas.config":"org.apache.kafka.common.security.scram.ScramLoginModule required username=\"xxxxx\" password=\"xxxxx\";"
}
}
' | jq .
{code}
> Mirrormaker config property config.properties.exclude is not working as expected
> ---------------------------------------------------------------------------------
>
> Key: KAFKA-13255
> URL: https://issues.apache.org/jira/browse/KAFKA-13255
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
> Affects Versions: 2.8.0
> Reporter: Anamika Nadkarni
> Priority: Major
>
> Objective - Use MM2 (kafka connect in distributed cluster) for data migration between cluster hosted in private data center and aws msk cluster.
> Steps performed -
> # Started kafka-connect service.
> # Created 3 MM2 connectors (i.e. source connector, checkpoint connector and heartbeat connector). Curl commands used to create connectors are in the attached file. To exclude certain config properties while topic replication, we are using the 'config.properties.exclude' property in the MM2 source connector.
> Expected -
> Source topic 'dev.portlandDc.anamika.helloMsk' should be successfully created in destination cluster.
> Actual -
> Creation of the source topic 'dev.portlandDc.anamika.helloMsk' in destination cluster fails with an error. Error is
> {code:java}
> [2021-08-06 06:13:40,944] WARN [mm2-msc|worker] Could not create topic dev.portlandDc.anamika.helloMsk. (org.apache.kafka.connect.mirror.MirrorSourceConnector:371)
> org.apache.kafka.common.errors.InvalidConfigurationException: Unknown topic config name: confluent.value.schema.validation{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)