[jira] [Resolved] (KAFKA-13452) MM2 creates invalid checkpoint when offset mapping is not available

["Viktor Somogyi-Vass (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-13452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Viktor Somogyi-Vass resolved KAFKA-13452.
-----------------------------------------
    Resolution: Duplicate

> MM2 creates invalid checkpoint when offset mapping is not available
> -------------------------------------------------------------------
>
>                 Key: KAFKA-13452
>                 URL: https://issues.apache.org/jira/browse/KAFKA-13452
>             Project: Kafka
>          Issue Type: Improvement
>          Components: mirrormaker
>            Reporter: Daniel Urban
>            Assignee: Viktor Somogyi-Vass
>            Priority: Major
>
> MM2 checkpointing reads the offset-syncs topic to create offset mappings for committed consumer group offsets. In some corner cases, it is possible that a mapping is not available in offset-syncs - in that case, MM2 simply copies the source offset, which might not be a valid offset in the replica topic at all.
> One possible situation is if there is an empty topic in the source cluster with a non-zero endoffset (e.g. retention already removed the records), and a consumer group which has a committed offset set to the end offset. If replication is configured to start replicating this topic, it will not have an offset mapping available in offset-syncs (as the topic is empty), causing MM2 to copy the source offset.
> This can cause issues when auto offset sync is enabled, as the consumer group offset can be potentially set to a high number. MM2 never rewinds these offsets, so even when there is a correct offset mapping available, the offset will not be updated correctly.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)