You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Ed Berezitsky (Jira)" <ji...@apache.org> on 2021/10/14 22:01:00 UTC
[jira] [Comment Edited] (KAFKA-13255) Mirrormaker config property
config.properties.exclude is not working as expected
[ https://issues.apache.org/jira/browse/KAFKA-13255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17429026#comment-17429026 ]
Ed Berezitsky edited comment on KAFKA-13255 at 10/14/21, 10:00 PM:
-------------------------------------------------------------------
[~ryannedolan], I recommend to fix this bug report:
Subject: Mirrormaker doesn't honor exclude filters for new topics
The bug was indeed introduced in 2.8, when MM started creating topics WITH configs. Prior to this, MM would create new topic on target without any config from source, and then `syncTopicConfigs` would replicate config while honoring the exclude filter defined in `DefaultConfigPropertyFilter` or explicitly specified in MM property `config.properties.exclude`.
After the update in 2.8, MM started creating new topics along with their configs from source cluster, but the configs aren't being filtered against exclude param.
I've submitted PR with a little update to use the same method as used for `syncTopicConfigs` to apply the filter.
https://github.com/apache/kafka/pull/11401
was (Author: bdesert):
[~ryannedolan], I recommend to fix this bug report:
Subject: Mirrormaker doesn't honor exclude filters for new topics
The bug was indeed introduced in 2.8, when MM started creating topics WITH configs. Prior to this, MM would create new topic on target without any config from source, and then `syncTopicConfigs` would replicate config while honoring the exclude filter defined in `DefaultConfigPropertyFilter` or explicitly specified in MM property `config.properties.exclude`.
After the update in 2.8, MM started creating new topics along with their configs from source cluster, but the configs aren't being filtered against exclude param.
I've submitted PR with a little update to use the same method as used for `syncTopicConfigs` to apply the filter.
> Mirrormaker config property config.properties.exclude is not working as expected
> ---------------------------------------------------------------------------------
>
> Key: KAFKA-13255
> URL: https://issues.apache.org/jira/browse/KAFKA-13255
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
> Affects Versions: 2.8.0
> Reporter: Anamika Nadkarni
> Priority: Major
>
> Objective - Use MM2 (kafka connect in distributed cluster) for data migration between cluster hosted in private data center and aws msk cluster.
> Steps performed -
> # Started kafka-connect service.
> # Created 3 MM2 connectors (i.e. source connector, checkpoint connector and heartbeat connector). Curl commands used to create connectors are in the attached file. To exclude certain config properties while topic replication, we are using the 'config.properties.exclude' property in the MM2 source connector.
> Expected -
> Source topic 'dev.portlandDc.anamika.helloMsk' should be successfully created in destination cluster.
> Actual -
> Creation of the source topic 'dev.portlandDc.anamika.helloMsk' in destination cluster fails with an error. Error is
> {code:java}
> [2021-08-06 06:13:40,944] WARN [mm2-msc|worker] Could not create topic dev.portlandDc.anamika.helloMsk. (org.apache.kafka.connect.mirror.MirrorSourceConnector:371)
> org.apache.kafka.common.errors.InvalidConfigurationException: Unknown topic config name: confluent.value.schema.validation{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)