[jira] [Created] (KAFKA-13452) MM2 creates invalid checkpoint when offset mapping is not available

["Daniel Urban (Jira)" <ji...@apache.org>]

Daniel Urban created KAFKA-13452:
------------------------------------

             Summary: MM2 creates invalid checkpoint when offset mapping is not available
                 Key: KAFKA-13452
                 URL: https://issues.apache.org/jira/browse/KAFKA-13452
             Project: Kafka
          Issue Type: Improvement
          Components: mirrormaker
            Reporter: Daniel Urban


MM2 checkpointing reads the offset-syncs topic to create offset mappings for committed consumer group offsets. In some corner cases, it is possible that a mapping is not available in offset-syncs - in that case, MM2 simply copies the source offset, which might not be a valid offset in the replica topic at all.

One possible situation is if there is an empty topic in the source cluster with a non-zero endoffset (e.g. retention already removed the records), and a consumer group which has a committed offset set to the end offset. If replication is configured to start replicating this topic, it will not have an offset mapping available in offset-syncs (as the topic is empty), causing MM2 to copy the source offset.

This can cause issues when auto offset sync is enabled, as the consumer group offset can be potentially set to a high number. MM2 never rewinds these offsets, so even when there is a correct offset mapping available, the offset will not be updated correctly.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)