[jira] [Commented] (KAFKA-13452) MM2 creates invalid checkpoint when offset mapping is not available

["Praveen Sinha (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-13452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17475255#comment-17475255 ] 

Praveen Sinha commented on KAFKA-13452:
---------------------------------------

This looks related to this one - https://issues.apache.org/jira/browse/KAFKA-12635. [~viktorsomogyi] can you please comment if the behavior mentioned in KAFKA-12635 is what we experience here?

> MM2 creates invalid checkpoint when offset mapping is not available
> -------------------------------------------------------------------
>
>                 Key: KAFKA-13452
>                 URL: https://issues.apache.org/jira/browse/KAFKA-13452
>             Project: Kafka
>          Issue Type: Improvement
>          Components: mirrormaker
>            Reporter: Daniel Urban
>            Assignee: Viktor Somogyi-Vass
>            Priority: Major
>
> MM2 checkpointing reads the offset-syncs topic to create offset mappings for committed consumer group offsets. In some corner cases, it is possible that a mapping is not available in offset-syncs - in that case, MM2 simply copies the source offset, which might not be a valid offset in the replica topic at all.
> One possible situation is if there is an empty topic in the source cluster with a non-zero endoffset (e.g. retention already removed the records), and a consumer group which has a committed offset set to the end offset. If replication is configured to start replicating this topic, it will not have an offset mapping available in offset-syncs (as the topic is empty), causing MM2 to copy the source offset.
> This can cause issues when auto offset sync is enabled, as the consumer group offset can be potentially set to a high number. MM2 never rewinds these offsets, so even when there is a correct offset mapping available, the offset will not be updated correctly.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)