You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Satish Duggana (Jira)" <ji...@apache.org> on 2023/09/04 05:08:00 UTC
[jira] [Updated] (KAFKA-15351) Update log-start-offset after leader election for topics enabled with remote storage
[ https://issues.apache.org/jira/browse/KAFKA-15351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Satish Duggana updated KAFKA-15351:
-----------------------------------
Description:
Case-1:
In the FETCH response, the leader-log-start-offset will be piggy-backed. But, there can be a scenario:
# Leader deleted the remote log segment and updates it's log-start-offset
# Before the replica-2 update it's log-start-offset via FETCH-request, the leadership changed to replica-2.
# There are no more eligible segments to delete from remote.
# The log-start-offset will be stale (referring to old log-start-offset but the data was already removed from remote)
# If the consumer starts to read from the beginning of the topic, it will fail to read.
See this comment [https://github.com/apache/kafka/pull/13561#discussion_r1293081560] for more details.
Case-2:
The old-leader (follower) can delete the remote log segment in the middle of leader election. We need to update the log-start-offset metadata for this case.
Case-3:
When tiered storage is enabled on the topic, and the last-standing-replica is restarted, then the log-start-offset should be updated upto log-start-offset-checkpoint offset.
was:
Case-1:
In the FETCH response, the leader-log-start-offset will be piggy-backed. But, there can be a scenario:
# Leader deleted the remote log segment and updates it's log-start-offset
# Before the replica-2 update it's log-start-offset via FETCH-request, the leadership changed to replica-2.
# There are no more eligible segments to delete from remote.
# The log-start-offset will be stale (referring to old log-start-offset but the data was already removed from remote)
# If the consumer starts to read from the beginning of the topic, it will fail to read.
Case-2:
The old-leader (follower) can delete the remote log segment in middle of leader election. We need to update the log-start-offset metadata for this case.
See this comment [https://github.com/apache/kafka/pull/13561#discussion_r1293081560] for more details.
Case-3:
When tiered storage is enabled on the topic, and the last-standing-replica is restarted, then the log-start-offset should be updated upto log-start-offset-checkpoint offset.
> Update log-start-offset after leader election for topics enabled with remote storage
> ------------------------------------------------------------------------------------
>
> Key: KAFKA-15351
> URL: https://issues.apache.org/jira/browse/KAFKA-15351
> Project: Kafka
> Issue Type: Sub-task
> Reporter: Kamal Chandraprakash
> Assignee: Kamal Chandraprakash
> Priority: Blocker
> Fix For: 3.6.0
>
>
> Case-1:
> In the FETCH response, the leader-log-start-offset will be piggy-backed. But, there can be a scenario:
> # Leader deleted the remote log segment and updates it's log-start-offset
> # Before the replica-2 update it's log-start-offset via FETCH-request, the leadership changed to replica-2.
> # There are no more eligible segments to delete from remote.
> # The log-start-offset will be stale (referring to old log-start-offset but the data was already removed from remote)
> # If the consumer starts to read from the beginning of the topic, it will fail to read.
> See this comment [https://github.com/apache/kafka/pull/13561#discussion_r1293081560] for more details.
> Case-2:
> The old-leader (follower) can delete the remote log segment in the middle of leader election. We need to update the log-start-offset metadata for this case.
> Case-3:
> When tiered storage is enabled on the topic, and the last-standing-replica is restarted, then the log-start-offset should be updated upto log-start-offset-checkpoint offset.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)