[jira] [Resolved] (KAFKA-10440) MM2 using Kconnect Framework Creates topics recursively on source cluster

["Nandini Anagondi (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-10440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nandini Anagondi resolved KAFKA-10440.
--------------------------------------
    Resolution: Invalid

> MM2 using Kconnect Framework Creates topics recursively on source cluster
> -------------------------------------------------------------------------
>
>                 Key: KAFKA-10440
>                 URL: https://issues.apache.org/jira/browse/KAFKA-10440
>             Project: Kafka
>          Issue Type: Bug
>          Components: KafkaConnect
>    Affects Versions: 2.5.0
>            Reporter: Nandini Anagondi
>            Priority: Major
>             Fix For: 2.5.0
>
>         Attachments: MirrorSourceConnector-config.json, MirrorSourceConnector-override-config.json, list_of_topics.txt
>
>
> I found an issue while running MM2 using Kconnect Framework. Based on the configuration provided in this [https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorConnectorConfig.java]
> *source : localhost:9092, Alias : A*
> *target : localhost:9093, Alias : B*
> *Issue Faced:*
> Topics from the source cluster are created at target cluster but instead of producing data of source topics to target it is producing to source cluster topics and those topics are getting created at source because the *allowAutoTopicCreation=true.*
> *How the issue arrises:* 
> Kconnect is running on source cluster and connectors are configured from A->B. Connect framework uses connect-distributed.properties to initialise producerConfig, consumerConfig. So, producer and consumer are initialised with source bootstrap servers i.e., with _localhost:9092("source.cluster.producer.bootstrap.servers" : "localhost:9092”, "source.cluster.consumer.bootstrap.servers" : "localhost:9092”)_.
> This configuration works perfectly fine with all traditional connectors because they produce and consume from that cluster itself. But MM2 connectors are contrary to this. Let's see this example
> Ideally MM2 should work like this
>  A                                               B
> test-mm2                             A.test-mm2 
> Indeed it is working like this and creating topics recursively. Please refer to the attachments.
>  A                                               B
> A.test-mm2                         A.A.test-mm2
> test-mm2                             A.test-mm2 
> This applies to all the topics where data need to be copied. Suppose if it doesn't have any data it won't cause this issue. 
> *Fixing the issue:*
> MirrorSourceConnector should produce to destination cluster topics while copying data. This issue can be fixed by overriding bootstrap.servers. To do this connector.client.config.override.policy should be set to "All" in connect-distributed.properties &  _(producer.override.bootstrap.servers, consumer.override.bootstrap.servers)_ should match with _target.cluster.bootstrap.servers_ and this need to be provided in connectors payload.So, connectors now produce the data to target cluster.
> A                                            B
> test-mm2                           A.test-mm2  
> List of the topics in both cases: [^list_of_topics.txt]
> You can find the config in both cases here:
> [^MirrorSourceConnector-config.json] - Config that matches with MirrorConnectorConfig
> [^MirrorSourceConnector-override-config.json]Config that works.
> To imitate the scenario: Start the kconnect at source cluster,  Configure connector with MirrorSourceConnector-config.json, create a topic and produce data to it. Check the list of topics in 5-10seconds.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)