[jira] [Created] (KAFKA-21192) Skip deleting remote segments when the leader epoch cache does not cover the remote log range

["Doyeon Kim (Jira)" <ji...@apache.org>]

Doyeon Kim created KAFKA-21192:
----------------------------------

             Summary: Skip deleting remote segments when the leader epoch cache does not cover the remote log range
                 Key: KAFKA-21192
                 URL: https://issues.apache.org/jira/browse/KAFKA-21192
             Project: Kafka
          Issue Type: Bug
            Reporter: Doyeon Kim
            Assignee: Doyeon Kim


When the leader epoch cache of a leader does not cover its remote log range [logStartOffset, localLogStartOffset), which can happen to replicas hit by KAFKA-17249, the cleanup of unreferenced epochs in RemoteLogManager treats the remote segments of the missing epochs as unreferenced and deletes them. The remote tier is shared, so the data is lost for every replica.

RemoteLogManager should skip this deletion while the cache is incomplete. PR #23569 adds RemoteLogManager#isLeaderEpochCacheMissingRemoteEntries, which can be reused for the check.

Open questions:
- Such a leader also uploads its incomplete checkpoint with every new segment, so followers that later build their aux state from those segments inherit the gap. Should it skip copying segments too? The cost is that local segments cannot be deleted until they are copied.
- Skipping the deletion does not repair the leader, which still cannot serve the remote range. The missing entries could be filled from the checkpoint of the latest remote segment, if a healthy leader uploaded it, or rebuilt from the segment leader epochs in the remote log metadata. Both are larger changes.
- Filtering such replicas out of leader election would need the follower to report the state of its cache, which is a protocol change. With KAFKA-21191 , a broken follower cannot rejoin the ISR before it is rebuilt anyway, and when the broken replica is the only one in the ISR, filtering it out would make the partition unavailable.

Discussion: https://github.com/apache/kafka/pull/23569#discussion_r4100224664



--
This message was sent by Atlassian Jira
(v8.20.10#820010)