You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Satish Duggana (Jira)" <ji...@apache.org> on 2023/09/04 05:08:00 UTC

[jira] [Updated] (KAFKA-15351) Update log-start-offset after leader election for topics enabled with remote storage

     [ https://issues.apache.org/jira/browse/KAFKA-15351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Satish Duggana updated KAFKA-15351:
-----------------------------------
    Description: 
Case-1:

In the FETCH response, the leader-log-start-offset will be piggy-backed. But, there can be a scenario:
 # Leader deleted the remote log segment and updates it's log-start-offset
 # Before the replica-2 update it's log-start-offset via FETCH-request, the leadership changed to replica-2.
 # There are no more eligible segments to delete from remote.
 # The log-start-offset will be stale (referring to old log-start-offset but the data was already removed from remote)
 # If the consumer starts to read from the beginning of the topic, it will fail to read.
See this comment [https://github.com/apache/kafka/pull/13561#discussion_r1293081560] for more details.

Case-2:
The old-leader (follower) can delete the remote log segment in the middle of leader election. We need to update the log-start-offset metadata for this case.

Case-3: 

When tiered storage is enabled on the topic, and the last-standing-replica is restarted, then the log-start-offset should be updated upto log-start-offset-checkpoint offset.

  was:
Case-1:

In the FETCH response, the leader-log-start-offset will be piggy-backed. But, there can be a scenario:
 # Leader deleted the remote log segment and updates it's log-start-offset
 # Before the replica-2 update it's log-start-offset via FETCH-request, the leadership changed to replica-2.
 # There are no more eligible segments to delete from remote.
 # The log-start-offset will be stale (referring to old log-start-offset but the data was already removed from remote)
 # If the consumer starts to read from the beginning of the topic, it will fail to read.

 

Case-2:

The old-leader (follower) can delete the remote log segment in middle of leader election. We need to update the log-start-offset metadata for this case.

See this comment [https://github.com/apache/kafka/pull/13561#discussion_r1293081560] for more details.

Case-3: 

When tiered storage is enabled on the topic, and the last-standing-replica is restarted, then the log-start-offset should be updated upto log-start-offset-checkpoint offset.


> Update log-start-offset after leader election for topics enabled with remote storage
> ------------------------------------------------------------------------------------
>
>                 Key: KAFKA-15351
>                 URL: https://issues.apache.org/jira/browse/KAFKA-15351
>             Project: Kafka
>          Issue Type: Sub-task
>            Reporter: Kamal Chandraprakash
>            Assignee: Kamal Chandraprakash
>            Priority: Blocker
>             Fix For: 3.6.0
>
>
> Case-1:
> In the FETCH response, the leader-log-start-offset will be piggy-backed. But, there can be a scenario:
>  # Leader deleted the remote log segment and updates it's log-start-offset
>  # Before the replica-2 update it's log-start-offset via FETCH-request, the leadership changed to replica-2.
>  # There are no more eligible segments to delete from remote.
>  # The log-start-offset will be stale (referring to old log-start-offset but the data was already removed from remote)
>  # If the consumer starts to read from the beginning of the topic, it will fail to read.
> See this comment [https://github.com/apache/kafka/pull/13561#discussion_r1293081560] for more details.
> Case-2:
> The old-leader (follower) can delete the remote log segment in the middle of leader election. We need to update the log-start-offset metadata for this case.
> Case-3: 
> When tiered storage is enabled on the topic, and the last-standing-replica is restarted, then the log-start-offset should be updated upto log-start-offset-checkpoint offset.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)