You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Luke Chen (Jira)" <ji...@apache.org> on 2024/05/13 09:35:00 UTC

[jira] [Created] (KAFKA-16711) highestOffsetInRemoteStorage is not updated after logDir altering within broker

Luke Chen created KAFKA-16711:
---------------------------------

             Summary: highestOffsetInRemoteStorage is not updated after logDir altering within broker
                 Key: KAFKA-16711
                 URL: https://issues.apache.org/jira/browse/KAFKA-16711
             Project: Kafka
          Issue Type: Sub-task
    Affects Versions: 3.7.0
            Reporter: Luke Chen
            Assignee: Luke Chen


We use topicIdPartition as the key for each RLM task. It will cause highestOffsetInRemoteStorage in log not updated after logDir alter completion. The reproducing flow is like this:

 
 # tp-0 locating in dirA is the leader of the partition
 # tp-0 is altering logDir to dirB
 # tp-0 is copying segments to remote storage (note: the log in dirA)
 # The logDir altering for tp-0 is completed
 # remoteLogManager#onLeadershipChange is invoked, copiedOffsetOption is reset to Optional.empty()
 # The copying segments to remote storage in step 3 for tp-0 is completed, updating copiedOffsetOption to new offset, as well as the log#highestOffsetInRemoteStorage. (note: the log in dirA)
 # In the next run of RLMTask, the log will be the one in target dir (dirB), and the log#highestOffsetInRemoteStorage (dirB) will be the default value (-1), which will block the log cleanup operation.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)