You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Siyao Meng (Jira)" <ji...@apache.org> on 2026/10/19 00:27:00 UTC

[jira] [Updated] (HDDS-17122) Support and test OM and SCM HA with more than 3 nodes

     [ https://issues.apache.org/jira/browse/HDDS-17122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDDS-17122:
------------------------------
    Description: 
OM HA and SCM HA have no hard limit on the number of nodes: {{ozone.om.nodes.<serviceId>}} / {{ozone.scm.nodes.<serviceId>}} accept any number of node IDs, and every node becomes a Ratis peer. In practice only 3-node rings are tested and documented (SCM-HA.md: "For reliable HA support choose 3 independent nodes"). Operators ask for 5 nodes so a ring keeps HA while one node is down for maintenance (see HDDS-3099).

This umbrella tracks making 5+ node OM and SCM rings a tested, documented configuration.

h3. Findings so far

New MiniOzoneCluster integration tests with 5 OMs and 5 SCMs (HDDS-17123) show that the core Ratis behaviour works: a single Raft group of 5, replication to all nodes, leader transfer, surviving 2 failures including the leader, no leader with 3 of 5 down, catch-up through checkpoint install, bootstrap 3 -> 5 and decommission 5 -> 3, secure 5-SCM failover.

The problems are around it:
* *Blocker:* HDDS-16924. An SCM that has installed a checkpoint can no longer serve checkpoints once it is leader, so lagging followers never catch up. With 3 SCMs this is rare (the primordial SCM usually stays leader); with 5 SCMs and a few failovers the 5-node test hits it reliably.
* Leadership transfer takes 4-13 s and the OM admin RPC is not idempotent under client retry.
* OM clients cannot reach a leader that is not in their configured OM list.
* SCM checkpoint download is retried without backoff.
* Test infrastructure bugs in MiniOzoneHAClusterImpl.
* Missing acceptance coverage and documentation.

Each item is a sub-task.

  was:
OM HA and SCM HA have no hard limit on the number of nodes: {{ozone.om.nodes.<serviceId>}} / {{ozone.scm.nodes.<serviceId>}} accept any number of node IDs, and every node becomes a Ratis peer. In practice only 3-node rings are tested and documented (SCM-HA.md: "For reliable HA support choose 3 independent nodes"). Operators ask for 5 nodes so a ring keeps HA while one node is down for maintenance (see HDDS-3099).

This umbrella tracks making 5+ node OM and SCM rings a tested, documented configuration.

h3. Findings so far

New MiniOzoneCluster integration tests with 5 OMs and 5 SCMs (sub-task "Add integration tests for 5-node OM HA and SCM HA") show that the core Ratis behaviour works: a single Raft group of 5, replication to all nodes, leader transfer, surviving 2 failures including the leader, no leader with 3 of 5 down, catch-up through checkpoint install, bootstrap 3 -> 5 and decommission 5 -> 3, secure 5-SCM failover.

The problems are around it:
* *Blocker:* HDDS-16924. An SCM that has installed a checkpoint can no longer serve checkpoints once it is leader, so lagging followers never catch up. With 3 SCMs this is rare (the primordial SCM usually stays leader); with 5 SCMs and a few failovers the 5-node test hits it reliably.
* Leadership transfer takes 4-13 s and the OM admin RPC is not idempotent under client retry.
* OM clients cannot reach a leader that is not in their configured OM list.
* SCM checkpoint download is retried without backoff.
* Test infrastructure bugs in MiniOzoneHAClusterImpl.
* Missing acceptance coverage and documentation.

Each item is a sub-task.


> Support and test OM and SCM HA with more than 3 nodes
> -----------------------------------------------------
>
>                 Key: HDDS-17122
>                 URL: https://issues.apache.org/jira/browse/HDDS-17122
>             Project: Apache Ozone
>          Issue Type: Improvement
>          Components: OM HA, SCM HA
>            Reporter: Siyao Meng
>            Priority: Major
>
> OM HA and SCM HA have no hard limit on the number of nodes: {{ozone.om.nodes.<serviceId>}} / {{ozone.scm.nodes.<serviceId>}} accept any number of node IDs, and every node becomes a Ratis peer. In practice only 3-node rings are tested and documented (SCM-HA.md: "For reliable HA support choose 3 independent nodes"). Operators ask for 5 nodes so a ring keeps HA while one node is down for maintenance (see HDDS-3099).
> This umbrella tracks making 5+ node OM and SCM rings a tested, documented configuration.
> h3. Findings so far
> New MiniOzoneCluster integration tests with 5 OMs and 5 SCMs (HDDS-17123) show that the core Ratis behaviour works: a single Raft group of 5, replication to all nodes, leader transfer, surviving 2 failures including the leader, no leader with 3 of 5 down, catch-up through checkpoint install, bootstrap 3 -> 5 and decommission 5 -> 3, secure 5-SCM failover.
> The problems are around it:
> * *Blocker:* HDDS-16924. An SCM that has installed a checkpoint can no longer serve checkpoints once it is leader, so lagging followers never catch up. With 3 SCMs this is rare (the primordial SCM usually stays leader); with 5 SCMs and a few failovers the 5-node test hits it reliably.
> * Leadership transfer takes 4-13 s and the OM admin RPC is not idempotent under client retry.
> * OM clients cannot reach a leader that is not in their configured OM list.
> * SCM checkpoint download is retried without backoff.
> * Test infrastructure bugs in MiniOzoneHAClusterImpl.
> * Missing acceptance coverage and documentation.
> Each item is a sub-task.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org