[jira] [Commented] (KAFKA-9746) MM2 produces messages in the wrong cluster

["Ryanne Dolan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-9746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17071927#comment-17071927 ] 

Ryanne Dolan commented on KAFKA-9746:
-------------------------------------

This is because MM2 hands off records to Connect to produce to the target cluster. If you need to produce to a cluster that is not the same cluster Connect is producing to, you'd need to have a MirrorSinkConnector (not yet released) in the middle, _or_ you can override Connect's producer config to point to your target cluster.

> MM2 produces messages in the wrong cluster
> ------------------------------------------
>
>                 Key: KAFKA-9746
>                 URL: https://issues.apache.org/jira/browse/KAFKA-9746
>             Project: Kafka
>          Issue Type: Bug
>          Components: mirrormaker
>    Affects Versions: 2.4.0, 2.4.1
>            Reporter: Marcel Renders
>            Priority: Major
>
> When the MirrorSourceConnector (MM2) is used to get messages from a remote Kafka cluster to the Kafka cluster that is connected to Kafka Connect, MM2 is working fine. However, when the connected cluster is being copied to the remote Kafka cluster, the messages are sent to the connected cluster, causing an infinite loop.
> Note that the topics are in fact created as expected in the remote Kafka cluster and no errors are thrown except flush failures due to the infinite loop.
> In short:
>  * Connected target -> unconnected source: OK
>  * Connected source -> unconnected target: NOK
> *Desired situation*
> The connected Kafka instance should only be used by Kafka Connect to store configs, offsets and status. It should be possible to mirror one Kafka cluster to another while neither are connected to the Kafka Connect instance.
> Typically the Kafka connectors are either sink or source, but the MM2 is different because the connector settings require both target and source configurations.
> *Reproducing the issue*
> This issue occurs with the following configuration, where localhost:9092 is used for Kafka Connect:
> {code:java}
>  {
>   "name": "MM_TEST",
>   "connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
>   "errors.log.enable": "true",
>   "enable.auto.commit": "true",
>   "auto.commit.interval.ms": "20000",
>   "refresh.topics.interval.seconds": "60",
>   "replication.factor": "1",
>   "sync.topic.acls.enabled": "false",
>   "checkpoints.topic.replication.factor": "1",
>   "heartbeats.topic.replication.factor": "1",
>   "offset-syncs.topic.replication.factor": "1",
>   "replication.policy.separator": "",
>   "topics": "'test1'",
>   "tasks.max": "1",
>   "key.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
>   "value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
>   "source.cluster.bootstrap.servers": "localhost:9092",
>   "source.cluster.alias": "",
>   "source.cluster.security.protocol": "PLAINTEXT",
>   "source.cluster.group.id": "consumer_mm_test",
>   "target.cluster.bootstrap.servers": "localhost:9093",
>   "target.cluster.alias": "",
>   "target.cluster.security.protocol": "PLAINTEXT"
> }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)