[GitHub] [kafka] michael-trelinski edited a comment on pull request #6295: KIP-382: MirrorMaker 2.0

[GitBox <gi...@apache.org>]

michael-trelinski edited a comment on pull request #6295:
URL: https://github.com/apache/kafka/pull/6295#issuecomment-620318837


   Hi @ryannedolan -
   
   I just got done setting up Mirror Maker V2 where I work.  It was successful, but the documentation doesn't match the code in various places.
   
   A couple of things that "got me" -
   
   - The JMX Metrics in the documentation:
   
   >  The mbean name for these metrics will be: kafka.mirror.connect:type=MirrorSourceConnect,target=([.\w]+),topic=([.\w]+),partition=([.\d]+) and kafka.mirror.connect:type=MirrorCheckpointConnector,target=([.\w]+),source=([.\w]+),group=([.\w]+)
   
   It should actually be something like (pardon me, I'm using the JMX-prometheus exporter): 
   `kafka.connect.mirror<type=MirrorSourceConnector, target=([\-\w]+), topic=([\-\w]+), partition=(\d+)`. Note how it's actually **kafka.connect.mirror** not **kafka.mirror.connect**, also it's MirrorSourceConnector, not MirrorSourceConnect.
   
   - In the documentation, there is a part about blacklisting groups: 
   
   > groups.blacklist | empty string | groups to exclude from replication
   
   In the code, it is `public static final String GROUPS_BLACKLIST_DEFAULT = "console-consumer-.*, connect-.*, __.*";`; here: https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/DefaultGroupFilter.java#L36 - this got me because we're actually using some Kafka Connect stuff for other projects, but we're running Mirror Maker as it's own cluster.  Maybe you could blacklist only Mirror Maker's Kafka Connect connect- consumer group?
   
   - This one is kind of a nit-pick, but in RemoteClusterUtils.translateOffsets, the timeout parameter isn't really a timeout in the conventional sense.  Stepping further into the call that's made using timeout,  `return client.remoteConsumerOffsets(consumerGroupId, remoteClusterAlias, timeout);` we can see that it's being used as a deadline, not a timeout.  In MirrorClient's remoteConsumerOffsets method, it's being used as a timeout, as a deadline.  Setting this too low caused me to miss some offsets because I didn't understand how it was being used.
   
   All in all a great product and vastly superior to Mirror Maker V1.  Thanks!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org