[jira] [Commented] (KAFKA-9076) MirrorMaker 2.0 automated consumer offset sync

["Ryanne Dolan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-9076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16957016#comment-16957016 ] 

Ryanne Dolan commented on KAFKA-9076:
-------------------------------------

[~yangguo1220] I agree this would be an improvement, but there is some complexity here:

- in order to write offsets for a consumer group, we need to know that the group is not already running on the target cluster. Otherwise we'd be stepping on that group's current offsets. The group coordinator won't allow this afaik.
- we could kick out a target group and force it to seek to the new offsets by revoking group membership and forcing a rebalance etc. But we wouldn't want to do this periodically.
- we could write offsets to a new group ID, eg. us-west group1, just like we do with topics, s.t. we avoid the above issues. Then migrating groups would involve changing the group ID. That works fine, but consumers would need a way to determine which group ID to use. Translating group ID like that is more cumbersome than translating offsets, since offsets can be altered using existing tools, but there is no way to tell a consumer to change its group ID.

I think there are scenarios where automatically writing offsets as you propose might make sense, e.g. in an active/standby scenario where consumers only connect to one cluster at a time. But if you are automating that behavior, you might as well automate the offset translation via RemoteClusterUtils, IMO.

My team has built external tooling using RemoteClusterUtils that works with existing consumers. It's possible to fully automate failover and failback this way. I'm skeptical that automatically writing offsets as you propose would make this process simpler.

An alternative to automatically writing offsets is to provide such tooling, e.g. as part of kafka-consumer-groups command. In addition to resetting consumers to a particular timestamp or offset, the tool could cause consumers to seek to the latest MM2 checkpoint.

> MirrorMaker 2.0 automated consumer offset sync
> ----------------------------------------------
>
>                 Key: KAFKA-9076
>                 URL: https://issues.apache.org/jira/browse/KAFKA-9076
>             Project: Kafka
>          Issue Type: Improvement
>          Components: mirrormaker
>    Affects Versions: 2.4.0
>            Reporter: Ning Zhang
>            Priority: Major
>              Labels: mirrormaker, pull-request-available
>             Fix For: 2.5.0
>
>
> To calculate the translated consumer offset in the target cluster, currently `Mirror-client` provides a function called "remoteConsumerOffsets()" that is used by "RemoteClusterUtils" for one-time purpose.
> In order to make the consumer migration from source to target cluster transparent and convenient, e.g. in event of source cluster failure, it is better to have a background job to continuously and periodically sync the consumer offsets from the source to target cluster, so that when the consumer switches to the target cluster, it will resume to consume from where it left off at source cluster.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)