You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Chris Egerton (Jira)" <ji...@apache.org> on 2023/02/08 21:08:00 UTC
[jira] [Updated] (KAFKA-13659) MM2 should read all offset syncs at start up and should not set consumer offset higher than the end offset
[ https://issues.apache.org/jira/browse/KAFKA-13659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Egerton updated KAFKA-13659:
----------------------------------
Description: MirrorCheckpointTask uses OffsetSyncStore, and does not check whether OffsetSyncStore managed to read to the "end" of the offset-syncs topic. OffsetSyncStore should fetch the endoffset of the topic at startup, and set a flag when it finally reaches the endoffset in consumption. MirrorCheckpointTask.poll should wait for this flag to be true before doing any in-memory updates and group offset management. (was: - MirrorCheckpointTask uses OffsetSyncStore, and does not check whether OffsetSyncStore managed to read to the "end" of the offset-syncs topic. OffsetSyncStore should fetch the endoffset of the topic at startup, and set a flag when it finally reaches the endoffset in consumption. MirrorCheckpointTask.poll should wait for this flag to be true before doing any in-memory updates and group offset management.
- MirrorCheckpointTask can create checkpoints which point into the "future" - meaning it sometimes translates consumer offsets in a way that the target offset is greater than the endoffset of the replica topic partition. MirrorCheckpointTask should fetch the endoffsets of the affected topics, and make sure that it does not try to set the consumer offset to anything higher than the endoffset.)
> MM2 should read all offset syncs at start up and should not set consumer offset higher than the end offset
> ----------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-13659
> URL: https://issues.apache.org/jira/browse/KAFKA-13659
> Project: Kafka
> Issue Type: Improvement
> Components: mirrormaker
> Reporter: Kanalas Vidor
> Assignee: Kanalas Vidor
> Priority: Major
>
> MirrorCheckpointTask uses OffsetSyncStore, and does not check whether OffsetSyncStore managed to read to the "end" of the offset-syncs topic. OffsetSyncStore should fetch the endoffset of the topic at startup, and set a flag when it finally reaches the endoffset in consumption. MirrorCheckpointTask.poll should wait for this flag to be true before doing any in-memory updates and group offset management.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)