You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Ivan Andika (Jira)" <ji...@apache.org> on 2026/08/24 01:42:00 UTC
[jira] [Updated] (HDDS-16259) Ratis transfer leadership fails with DIFFERENT_LEADER
[ https://issues.apache.org/jira/browse/HDDS-16259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Andika updated HDDS-16259:
-------------------------------
Description:
We encountered an issue where transfer leadership failed with DIFFERENT_LEADER.
After looking at the audit logs, it was identified that it was due to the METADATA log created in the old leader. The old leader should block non-METADATA Raft log append through the use of RaftServerImpl#isSteppingDown.
The sequence looks like, say the Raft group consists of peer [A,B,C]
# User sends a transfer leadership request from Raft peer A to Raft peer B
# Raft peer A checks that Raft peer B log is up to date through TransferLeadership#isFollowUpToDate, say the Raft peer B log index matched with the Raft peer A for index 5
# Raft peer A sends a StartLeaderElectionRequest to Raft peer B
# {*}(Important{*}) Raft peer A creates a METADATA log entry at index 6 due to onFollowerSuccessAppendEntries (note that unlike normal log, METADATA log is not pretected by RaftServerImpl#isSteppingDown
# {*}(Important{*}) Raft peer A replicates the a METADATA log entry at index 6 to Raft peer C (another follower)
# Raft peer B starts an election, but Raft peer A and C rejected the election because Raft peer B entry is still at 5 (i.e. not up-to-date)
# Raft peer A or Raft peer C election timeout is hit and then starts an election, say Raft peer C becomes the new leader instead
# Raft peer A TransferLeadership#transfer finished, but it recorded DIFFERENT_LEADER (expect Raft peer B, but found Raft peer C to be the new leader)
We can disable METADATA log to prevent this issue, but we should also try to think how to prevent METADATA log from impacting the transfer leadership process. One easy way is that we ignore the METADATA log creation when transfer leadership is going on, but we need to double check whether this is safe (METADATA log is not in the original Raft paper)
was:
We encountered an issue where transfer leadership failed with DIFFERENT_LEADER.
After looking at the audit logs, it was identified that it was due to the METADATA log created in the old leader. The old leader should block non-METADATA Raft log append through the use of RaftServerImpl#isSteppingDown.
The sequence looks like, say the Raft group consists of peer [A,B,C]
# User sends a transfer leadership request from Raft peer A to Raft peer B
# Raft peer A checks that Raft peer B log is up to date through TransferLeadership#isFollowUpToDate, say the Raft peer B log index matched with the Raft peer A for index 5
# Raft peer A sends a StartLeaderElectionRequest to Raft peer B
# {*}(Important{*}) Raft peer A creates a METADATA log entry at index 6 due to onFollowerSuccessAppendEntries (note that unlike normal log, METADATA log is not pretected by RaftServerImpl#isSteppingDown
# {*}(Important{*}) Raft peer A replicates the a METADATA log entry at index 6 to Raft peer C (another follower)
# Raft peer B starts an election, but Raft peer A and C rejected the election because Raft peer B entry is still at 5 (i.e. not up-to-date)
# Raft peer A or Raft peer C election timeout is hit and then starts an election, say Raft peer C becomes the new leader instead
# Raft peer A TransferLeadership#transfer finished, but it recorded DIFFERENT_LEADER (expect Raft peer B, but found Raft peer C to be the new leader)
We can disable METADATA log to prevent this issue, but we should also try to think how to prevent METADATA log from impacting the transfer leadership process.
> Ratis transfer leadership fails with DIFFERENT_LEADER
> -----------------------------------------------------
>
> Key: HDDS-16259
> URL: https://issues.apache.org/jira/browse/HDDS-16259
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
>
> We encountered an issue where transfer leadership failed with DIFFERENT_LEADER.
> After looking at the audit logs, it was identified that it was due to the METADATA log created in the old leader. The old leader should block non-METADATA Raft log append through the use of RaftServerImpl#isSteppingDown.
> The sequence looks like, say the Raft group consists of peer [A,B,C]
> # User sends a transfer leadership request from Raft peer A to Raft peer B
> # Raft peer A checks that Raft peer B log is up to date through TransferLeadership#isFollowUpToDate, say the Raft peer B log index matched with the Raft peer A for index 5
> # Raft peer A sends a StartLeaderElectionRequest to Raft peer B
> # {*}(Important{*}) Raft peer A creates a METADATA log entry at index 6 due to onFollowerSuccessAppendEntries (note that unlike normal log, METADATA log is not pretected by RaftServerImpl#isSteppingDown
> # {*}(Important{*}) Raft peer A replicates the a METADATA log entry at index 6 to Raft peer C (another follower)
> # Raft peer B starts an election, but Raft peer A and C rejected the election because Raft peer B entry is still at 5 (i.e. not up-to-date)
> # Raft peer A or Raft peer C election timeout is hit and then starts an election, say Raft peer C becomes the new leader instead
> # Raft peer A TransferLeadership#transfer finished, but it recorded DIFFERENT_LEADER (expect Raft peer B, but found Raft peer C to be the new leader)
> We can disable METADATA log to prevent this issue, but we should also try to think how to prevent METADATA log from impacting the transfer leadership process. One easy way is that we ignore the METADATA log creation when transfer leadership is going on, but we need to double check whether this is safe (METADATA log is not in the original Raft paper)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org