[jira] [Commented] (KAFKA-12768) Mirrormaker2 consumer config not using newly assigned client id

["Dongjin Lee (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-12768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17344283#comment-17344283 ] 

Dongjin Lee commented on KAFKA-12768:
-------------------------------------

I reviewed this problem and found the following:

The MirrorMaker 1 as a standalone application had been included in the Kafka distribution for a long time ago and MirrorMaker 2 was added with [KIP-382|https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] by [~ryannedolan] in 2.4.0, with Kafka Connect support.

MirrorMaker 1 supported multi-cluster mirroring with 'clusters' config property. (Which is what you are using here.) However, since Kafka connector supports point to point mirroring only, this property is not supported - in short, *the configuration is a little bit different between standalone application way and connector plugin way, especially for 'clusters'.* In this case, you should use 'source.cluster.consumer.client.id', not 'source.consumer.client.id' nor 'source.client.id'.

I guess [~ryannedolan] had no choice here for the inherent difference between standalone application and connector. But I think this glitch may confuse, and document it would be worth, for those who without historical context.

[~ryannedolan] [~tombentley] How do you think? Do we need some documentation here?

> Mirrormaker2 consumer config not using newly assigned client id
> ---------------------------------------------------------------
>
>                 Key: KAFKA-12768
>                 URL: https://issues.apache.org/jira/browse/KAFKA-12768
>             Project: Kafka
>          Issue Type: Bug
>          Components: mirrormaker
>    Affects Versions: 2.6.0
>            Reporter: Vincent
>            Priority: Major
>
> Component: MirrorMaker2 from the 2.6.0 distribution.
> We tried to set quotas based client.id in mirrormaker2. We tried the setting source.consumer.client.id and source.client.id properties with no luck.
> I was able to update the consumer client id using the US->EUROPE.consumer.client.id config (from the customer) with a single instance of MM2. With a single instance, everything works fine without any issue. However, we are running 2 instances of MirrorMaker 2 with tasks.max set to 2 and it doesn't work with multiple MM2 processes. We also tried stopping all mirromaker2 instances and starting them again but it didn't help.
> Currently, the only workaround is to recreate (or rename) the Connect internal topics.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)