You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Siyao Meng (Jira)" <ji...@apache.org> on 2026/10/19 00:27:00 UTC

[jira] [Updated] (HDDS-17125) OM transferLeadership is not idempotent under client failover retry

     [ https://issues.apache.org/jira/browse/HDDS-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDDS-17125:
------------------------------
    Description: 
If the {{transferLeadership}} admin RPC takes longer than {{ozone.om.client.rpc.timeout}} (see HDDS-17124 for why it can take 4-13 s), the OM client fails over and sends the same request to the next OM. After a successful transfer that is the *new* leader, which then runs a second set-priority / transfer / reset-priority cycle while the first one is still resetting priorities.

Observed in a 5-node test (5 s client RPC timeout):
* 17.52 omNode-1 starts transfer to omNode-2; 22.52 client SocketTimeoutException and failover.
* 24.21 transfer to omNode-2 succeeds; 24.55 the retried request arrives at omNode-2: "Start transferring leadership to omNode-2" (to itself).
* 29.83 / 30.10 both handlers reset priorities; 31.60 omNode-2 steps down and omNode-4 is elected. That is an extra leader change nobody asked for.
* Client fails with "Could not determine or connect to OM Leader" after 10 failovers.

*Proposal:* treat "target is already the leader" as success without touching priorities, and/or do not retry {{transferLeadership}} on a different OM after a timeout.

  was:
If the {{transferLeadership}} admin RPC takes longer than {{ozone.om.client.rpc.timeout}} (see the RatisHelper leader hint sub-task for why it can take 4-13 s), the OM client fails over and sends the same request to the next OM. After a successful transfer that is the *new* leader, which then runs a second set-priority / transfer / reset-priority cycle while the first one is still resetting priorities.

Observed in a 5-node test (5 s client RPC timeout):
* 17.52 omNode-1 starts transfer to omNode-2; 22.52 client SocketTimeoutException and failover.
* 24.21 transfer to omNode-2 succeeds; 24.55 the retried request arrives at omNode-2: "Start transferring leadership to omNode-2" (to itself).
* 29.83 / 30.10 both handlers reset priorities; 31.60 omNode-2 steps down and omNode-4 is elected. That is an extra leader change nobody asked for.
* Client fails with "Could not determine or connect to OM Leader" after 10 failovers.

*Proposal:* treat "target is already the leader" as success without touching priorities, and/or do not retry {{transferLeadership}} on a different OM after a timeout.


> OM transferLeadership is not idempotent under client failover retry
> -------------------------------------------------------------------
>
>                 Key: HDDS-17125
>                 URL: https://issues.apache.org/jira/browse/HDDS-17125
>             Project: Apache Ozone
>          Issue Type: Sub-task
>          Components: OM HA
>            Reporter: Siyao Meng
>            Priority: Major
>
> If the {{transferLeadership}} admin RPC takes longer than {{ozone.om.client.rpc.timeout}} (see HDDS-17124 for why it can take 4-13 s), the OM client fails over and sends the same request to the next OM. After a successful transfer that is the *new* leader, which then runs a second set-priority / transfer / reset-priority cycle while the first one is still resetting priorities.
> Observed in a 5-node test (5 s client RPC timeout):
> * 17.52 omNode-1 starts transfer to omNode-2; 22.52 client SocketTimeoutException and failover.
> * 24.21 transfer to omNode-2 succeeds; 24.55 the retried request arrives at omNode-2: "Start transferring leadership to omNode-2" (to itself).
> * 29.83 / 30.10 both handlers reset priorities; 31.60 omNode-2 steps down and omNode-4 is elected. That is an extra leader change nobody asked for.
> * Client fails with "Could not determine or connect to OM Leader" after 10 failovers.
> *Proposal:* treat "target is already the leader" as success without touching priorities, and/or do not retry {{transferLeadership}} on a different OM after a timeout.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org