[jira] [Created] (KAFKA-13659) MM2 should read all offset syncs at start up and should not set consumer offset higher than the end offset

["Kanalas Vidor (Jira)" <ji...@apache.org>]

Kanalas Vidor created KAFKA-13659:
-------------------------------------

             Summary: MM2 should read all offset syncs at start up and should not set consumer offset higher than the end offset
                 Key: KAFKA-13659
                 URL: https://issues.apache.org/jira/browse/KAFKA-13659
             Project: Kafka
          Issue Type: Improvement
          Components: mirrormaker
            Reporter: Kanalas Vidor


- MirrorCheckpointTask uses OffsetSyncStore, and does not check whether OffsetSyncStore managed to read to the "end" of the offset-syncs topic. OffsetSyncStore should fetch the endoffset of the topic at startup, and set a flag when it finally reaches the endoffset in consumption. MirrorCheckpointTask.poll should wait for this flag to be true before doing any in-memory updates and group offset management.
 - MirrorCheckpointTask can create checkpoints which point into the "future" - meaning it sometimes translates consumer offsets in a way that the target offset is greater than the endoffset of the replica topic partition. MirrorCheckpointTask should fetch the endoffsets of the affected topics, and make sure that it does not try to set the consumer offset to anything higher than the endoffset.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)