[jira] [Commented] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)

["Adam Bellemare (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16868544#comment-16868544 ] 

Adam Bellemare commented on KAFKA-7500:
---------------------------------------

[~ryannedolan] - Thanks for driving this forward, it looks extremely useful.

I'm currently evaluating this for my company's needs. I have a few questions.

1) Are the offset translation topics included in the patch you have above? I ask because I see that https://issues.apache.org/jira/browse/KAFKA-7815 is still open, and I am not sure if SourceTask is the provider of this info and to the overall status of the offset translations work.

2) How do you go about switching a consumer from one cluster to another, say in the case of a total cluster failure? I have noted that there exists the offsets topics that indicate the source offset and equivalent destination offset for a consumer group. Aside from the mechanisms of redirecting the consumer to the replica cluster, what do you do in terms of ensuring that the consumer is able to read from this destination cluster? My first instinct is to have a service that consumes from the offsets-sync topic and sets the offsets for each consumer group as the new sync events come in. If this is up to date with the latest events, then it should be possible to restart the consumer from the checkpoint in the destination cluster under the same consumer group id.

3) Do you use timestamps at all for failing over from one cluster to another? This is a bit of an extension to #2 above, and my current understanding is that it is not necessary to do so. I seem to recall that in previous versions of MM it was more common to use timestamps than offsets, since the guarantee for the offset wasn't present.

 

Anyways, thanks very much for your work here. I am eager to see this get into 2.4 because I believe it will make multi-cluster much more approachable for smaller businesses.

> MirrorMaker 2.0 (KIP-382)
> -------------------------
>
>                 Key: KAFKA-7500
>                 URL: https://issues.apache.org/jira/browse/KAFKA-7500
>             Project: Kafka
>          Issue Type: New Feature
>          Components: KafkaConnect, mirrormaker
>            Reporter: Ryanne Dolan
>            Priority: Minor
>             Fix For: 2.4.0
>
>
> Implement a drop-in replacement for MirrorMaker leveraging the Connect framework.
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0]
> [https://github.com/apache/kafka/pull/6295]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)