[jira] [Created] (HDDS-17267) Datanode drops container CLOSE actions from heartbeats that SCM did not process, so a full container can stay OPEN after an SCM restart

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created HDDS-17267:
---------------------------------

             Summary: Datanode drops container CLOSE actions from heartbeats that SCM did not process, so a full container can stay OPEN after an SCM restart
                 Key: HDDS-17267
                 URL: https://issues.apache.org/jira/browse/HDDS-17267
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Siyao Meng
         Attachments: MC-2-put-back-container-actions-not-processed-by-scm.patch, TestBugMC2ContainerActionsNotPutBack.java

h3. Mechanism
{{HeartbeatEndpointTask.call}} takes the queued container actions for the SCM out of the per SCM queue ({{StateContext.getPendingContainerAction}} polls them) when it builds the heartbeat. Nothing returns them when SCM does not process that heartbeat:
* *The RPC fails with an IOException.* {{putBackIncrementalReports}} returns only command status reports and incremental container reports to the queue. The container actions are dropped.
* *SCM does not know the datanode*, which is the state of an SCM that has just restarted. {{SCMDatanodeHeartbeatDispatcher.dispatch}} replies with a re-register command and fires no event for the payload (intended since HDDS-601). The RPC succeeded, so the datanode puts nothing back.

The only producer of a CLOSE container action is the write path ({{HddsDispatcher.sendCloseContainerActionIfNeeded}}), which raises it again on each later write to the container. If no write reaches the container after the action was dropped, SCM never receives it. Pipeline actions are not affected: the only pipeline action is CLOSE, and {{StateContext.PipelineActionMap}} keeps it queued until the pipeline is removed from the datanode.

h3. Trigger
An SCM restart while a container is being filled past the datanode close threshold, with the last write to that container landing in one of two windows:
# while SCM is down: the heartbeats that fail drain the action (IOException path);
# after SCM is back but before the datanode's first heartbeat to it: that heartbeat gets the re-register reply.

After re-registration the full container report gives SCM the real size of the container, and {{ContainerStateManagerImpl.findContainerWithSpace}} only places a block in a container while used bytes plus block size fit in the container size. Once the container is past that point no block, and so no write, goes there again. With default sizes (5 GB container, 256 MB block, close threshold 0.9) the datanode asks to close from 4.5 GB and SCM stops placing blocks above 4.75 GB, so the container stays OPEN only if its last write left it above about 4.75 GB; below that SCM may place another block there and the next write raises CLOSE again (argued from the code, not run with default sizes).

With SCM HA the datanode keeps a separate queue per SCM and only the leader acts on a CLOSE action ({{CloseContainerEventHandler}} skips it on followers). A restarted follower therefore loses nothing that matters. The loss matters with a single SCM, or when the restarted SCM is the leader again by the time the datanode re-registers, for example after all SCMs were restarted (argued, not run with SCM HA).

The IOException path alone, without a restart (for example one heartbeat RPC timeout), is usually hidden: SCM learns the size of an open container only from container reports, so it keeps placing blocks there and the next write raises CLOSE again. It can strand the container only if writes stop until the next full container report shows the container past the point above (argued, not run).

h3. Impact
The container stays OPEN in SCM until its pipeline is closed or an administrator closes it. While it is OPEN:
* {{DeletedBlockLogImpl}} skips delete transactions for open containers, so the space of deleted keys in it is not reclaimed;
* ReplicationManager stops at {{OpenContainerHandler}}, which only closes an open container that has no healthy pipeline or a replica in another state;
* the balancer does not move it and the datanode data scanner does not scan it.

No data is lost and writes continue in other containers.

h3. Reproduction
[^TestBugMC2ContainerActionsNotPutBack.java] ({{hadoop-hdds/server-scm}}), deterministic, unmodified source. It connects a real {{StateContext}}, {{EndpointStateMachine}} and {{HeartbeatEndpointTask}} to a real {{SCMNodeManager}} and {{SCMDatanodeHeartbeatDispatcher}} through a mocked RPC proxy that builds the reply like {{SCMDatanodeProtocolServer.sendHeartbeat}}. An SCM restart is a new node manager and dispatcher. The first test shows a CLOSE action sent to a registered SCM arriving, then a CLOSE action in the heartbeat to the restarted SCM never arriving, also after re-registration and three more heartbeats. The second shows the same after one heartbeat that fails with an IOException. A passing test means the defect is present.

A mini cluster run (one datanode, RATIS/ONE, container size equal to block size of 1 MB, 20 s heartbeat interval, public client API, real SCM restart, not attached) showed the end state for both windows: with the last write after the restart, and with the last write while SCM was stopped, the full container was still OPEN in SCM 80 s after re-registration while SCM placed the next key in a new container. Without a restart, and with the write after re-registration, the same container went to CLOSING within seconds. The second window (last write after the restart) depends on restarting SCM right after a heartbeat so that the datanode has not yet contacted the new SCM.

h3. Patch
[^MC-2-put-back-container-actions-not-processed-by-scm.patch], against b10a7d427bdf83807a7e48a1be507d3b02d5a503. It also applies to master at e0de543cbe9.

{{HeartbeatEndpointTask}} returns the container actions of the heartbeat to the queue of that SCM when the RPC fails with an IOException and when the reply contains a re-register command. The new {{StateContext.putBackContainerActions}} adds them only to that SCM's queue and skips an action that is already queued again, like {{addContainerActionIfAbsent}}. The datanode then sends the action in its next heartbeat, after re-registration in the second case. Incremental and command status reports are left as they are on the re-register path: the full container report sent with the registration supersedes the incremental reports, and putting them back there would send them after it (compare HDDS-17062). A CLOSE action that SCM did process before the RPC failed is sent again, and so is one in a heartbeat that Recon processed and answered with a re-register command (Recon does that for a datanode it has not heard from recently, including after its own restart); SCM already receives repeated CLOSE actions for the same container from repeated writes, and {{CloseContainerEventHandler}} does not change the state of a container that is no longer OPEN. While an SCM or Recon is unreachable the actions now stay queued for it instead of being drained, at most one per container and reason.

Covered by {{testContainerActionsPutBackOnHeartbeatFailure}} (which also checks that the queue of a second SCM is left unchanged) and {{testContainerActionsPutBackOnReregisterCommand}} in the existing {{TestHeartbeatEndpointTask}}, and {{testheartbeatWithContainerActions}} now also checks that a processed heartbeat leaves nothing queued. Without the change both new tests fail. With it {{TestHeartbeatEndpointTask}}, {{TestStateContext}}, {{TestHddsDispatcher}}, {{TestEndPoint}} and {{TestSCMDatanodeHeartbeatDispatcher}} pass (64 tests) and checkstyle is clean. With the patch both tests of the reproduction fail, because SCM receives the CLOSE action after re-registration and after the failed heartbeat, and in the mini cluster runs above the container went to CLOSING in both windows. Containers already left OPEN are not repaired by the patch; {{ozone admin container close}} closes them.

Found by TLA+ model checking and code review of the datanode registration and heartbeat paths under HDDS-15926, on commit b10a7d427bdf83807a7e48a1be507d3b02d5a503. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. Related but not duplicates: HDDS-16723 (incremental reports queued for a re-registering SCM make the heartbeat too large), HDDS-17062 (incremental reports put back after a failed heartbeat are sent after the newer full report) and HDDS-17064 (a full container report in a failed heartbeat is not resent). The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org