You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Siyao Meng (Jira)" <ji...@apache.org> on 2026/10/11 07:36:00 UTC
[jira] [Updated] (HDDS-16891) After an SCM restart or leader change, SCM has no pending container allocations and grants again the slots reserved for containers not yet written
[ https://issues.apache.org/jira/browse/HDDS-16891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siyao Meng updated HDDS-16891:
------------------------------
Attachment: TestBugMC8PendingAllocationFailover.java
> After an SCM restart or leader change, SCM has no pending container allocations and grants again the slots reserved for containers not yet written
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16891
> URL: https://issues.apache.org/jira/browse/HDDS-16891
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: TestBugMC8PendingAllocationFailover.java
>
>
> h3. Mechanism
> {{PendingContainerTracker}} keeps the reservations in memory, in the {{TwoWindowBucket}} of each {{DatanodeInfo}}. Only the leader fills it: {{ContainerManagerImpl.allocateContainer}} calls {{PipelineManagerImpl.checkSpaceAndRecordAllocation}} before the replicated {{containerStateManager.addContainer}}. Followers apply {{addContainer}}, so they know the container, but they do not record its slot. On restart, {{ContainerStateManagerImpl.initialize}} loads the containers from the SCM DB and nothing records their slots either. Nothing records the slots when an SCM becomes leader. Datanodes create a container on its first write, so until then no storage report counts it as committed. After a leader change or a restart, SCM therefore sees the slots of granted containers that are not written yet as free.
> Every SCM receives the replicated {{addContainer}} and the heartbeats and container reports that datanodes send to all SCMs (followers run the report handlers, which also remove pending entries). The gap is that only the leader records the reservation, before submitting.
> h3. Trigger
> # The leader allocates container C1 on a pipeline whose datanodes have one slot left. C1 is not written yet.
> # Leadership moves to another SCM (transfer, crash, long pause), or the SCM restarts. This includes a single SCM.
> # Once out of safe mode, the leader allocates C2 on the same datanodes.
> h3. Impact
> A leader change or restart drops every unconfirmed reservation on every datanode at once, as if all of them had aged out. Only reservations younger than two roll intervals ({{ozone.scm.pending.container.roll.interval}}, default 5m) and not yet confirmed by a container report are affected, and only datanodes that are down to their last slots can be over granted. The expected effect is that the datanode refuses one of the containers for lack of space when it is created. On a Ratis pipeline the failed create marks the pipeline's {{ContainerStateMachine}} unhealthy, so the pipeline is closed and the in flight writes on it are retried elsewhere. This is argued from the code and was not run. Replica placement after the leader change reads the same empty buckets; that variant was not run either.
> h3. Reproduction
> [^TestBugMC8PendingAllocationFailover.java] ({{hadoop-ozone/integration-test}}) starts a mini cluster with 3 SCMs, 1 OM and 3 datanodes. Each datanode volume uses the existing {{MockSpaceUsageCheckFactory$HalfTera}} (512 GiB) and the container size is 300 GiB, so each datanode has one slot. Only the admin calls {{allocateContainer}} and {{transferLeadership}} are used. The leader grants C1 and holds one pending entry per datanode, and it refuses a second container. After the transfer, once the new leader is ready and out of safe mode, it has C1 OPEN with no replicas, committed space 0 and no pending entry on all three datanodes, and it grants C2 on the same three datanodes. A passing test means the defect is present (under a minute on an unmodified checkout). The restart case was not run.
> h3. Suggested fix
> No patch is attached yet, so that the design can be agreed first. The proposal is a rebuild of the buckets when an SCM becomes the serving leader:
> * Run once the SCM is leader ready and out of safe mode, the point where {{ReplicationManager}} starts. After a restart the datanodes are registered by then, and allocation is refused in safe mode anyway.
> * First drop all pending entries, then seed: for each OPEN container, record a slot on each datanode of its pipeline that has no replica of it.
> * Finish the rebuild before the first allocation is served.
> This covers a leader change, a restart of one SCM or of all of them, a follower that installed a Ratis snapshot, and a newly bootstrapped SCM. The drop also clears stale entries of a returning leader: replication reservations that are not removed when {{ReplicationManager}} clears its pending ops on leader change, and the case below. It keeps the apply path free of node manager calls. Its cost is that it also reserves for OPEN containers that the previous leader had already aged out (for example containers {{getMatchingContainer}} opened ahead of use), so for up to two roll intervals after the rebuild SCM grants fewer containers than before, never more. A datanode that registers after the rebuild starts with an empty bucket, as today.
> The alternative, recording the slot whenever an SCM applies the replicated {{addContainer}}, needs no scan, but it does not cover an SCM restart or a snapshot install, because containers loaded from the SCM DB are not applied again. It also puts a node manager call in the apply path (which Recon runs too), has to avoid counting the leader's own entry twice, and on a lagging follower the container report for a container can arrive before the follower applies its {{addContainer}}, leaving the entry to age out. Waiting for a container report interval after leader ready does not help either, because a container that is not written yet is in no report.
> A related case: when the replicated {{addContainer}} fails after the slot was reserved, the reservation is kept. That is deliberate. {{SCMRatisServerImpl.submitRequest}} gives up after {{ozone.scm.ha.ratis.request.timeout}} while the entry stays in the Raft log, and on step down Ratis (3.3.1) answers the requests already in the leader's log with {{NotLeaderException}} although the next leader may commit them. A timeout never says whether the entry will commit, and a step down reply has the same result code ({{SCM_NOT_LEADER}}) as a definite rejection, so releasing on failure could turn a bounded under grant into an over grant. Such an entry ages out within two roll intervals, or is dropped by the rebuild above when the SCM becomes leader again.
> Found by TLA+ model checking and code review of the SCM pending container allocation tracking under HDDS-15926, on commit b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org