[jira] [Created] (HDDS-17224) Decommission or maintenance completes on an empty container set after SCM restart or datanode re-registration, before the full container report

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created HDDS-17224:
---------------------------------

             Summary: Decommission or maintenance completes on an empty container set after SCM restart or datanode re-registration, before the full container report
                 Key: HDDS-17224
                 URL: https://issues.apache.org/jira/browse/HDDS-17224
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Siyao Meng
         Attachments: MC-2-wait-for-full-container-report-before-admin-completion.patch, TestBugMC2EmptyContainerSetPassesGate.java

h3. Mechanism
{{DatanodeAdminMonitorImpl.checkContainersReplicatedOnNode}} checks only the containers in {{NodeManager.getContainers}} for the node and passes when that set is empty. The set is filled only by container reports, and it starts empty whenever the node gets a new {{DatanodeEntry}}:
* {{NodeStateMap.addNode}}, when a datanode registers with an SCM that has restarted;
* {{NodeStateMap.updateNode}}, when a datanode registers again with a new address, ports or software version (the container set loss described in HDDS-16632).

{{SCMDatanodeProtocolServer.register}} fires the full container report as an event only after {{SCMNodeManager.register}} has returned, and the report is processed on the container report executor. The monitor runs on its own executor and does not check safe mode or whether a report has been processed. After an SCM restart, the admin workflow is resumed for a node that registers with a persisted DECOMMISSIONING or ENTERING_MAINTENANCE state, by the NEW_NODE handler ({{NodeDecommissionManager.continueAdminForNode}}) or, if the node registered before this SCM became leader, by {{NodeDecommissionManager.onBecomeLeader}}. The other gate conditions do not hold the node: its pipelines were closed when the workflow started, and its persisted state matches. If a monitor run lands before the report is processed, the node completes the workflow. The report that arrives later does not reopen it.

h3. Trigger
Re-registration:
# A datanode is being decommissioned and the monitor is waiting for an under replicated container on it.
# The datanode restarts and registers again with a new software version (rolling upgrade) or a changed port set (seen on a plain datanode restart in a mini cluster in HDDS-17190). Its container set in SCM is now empty.
# A monitor run happens before the full container report from the registration has been processed.
# The node becomes DECOMMISSIONED.

SCM restart:
# A datanode is DECOMMISSIONING or ENTERING_MAINTENANCE and holds an under replicated container. SCM restarts.
# The datanode registers with its persisted state, and the leader starts tracking it again.
# A monitor run happens before its full container report has been processed (after an SCM restart many datanodes register and their reports queue up).
# The node becomes DECOMMISSIONED or IN_MAINTENANCE.

h3. Impact
The node is reported DECOMMISSIONED (safe to remove) or IN_MAINTENANCE while containers on it are still under replicated, and it is no longer tracked. This can hold for every container on the node at once. Once the report is processed, Replication Manager sees the containers as under replicated and copies them from the node while it is still up, so redundancy is lost only if the operator removes or stops the node first, and data only with a further failure. The window is from registration until the report is processed; how long it is on a real cluster was not measured.

h3. Reproduction
PASS on unmodified source, deterministic. [^TestBugMC2EmptyContainerSetPassesGate.java] ({{hadoop-hdds/server-scm}}) uses the real {{SCMNodeManager}} for registration, heartbeats and operational state, the real {{ContainerReportHandler}} for the full container report (with a stub {{ContainerManager}} that knows one CLOSED RATIS/THREE container) and the real {{DatanodeAdminMonitorImpl}}. Replication Manager is mocked as in {{TestDatanodeAdminMonitor}}: the container has the replica on the node and one other copy. The only thing the test controls is whether the monitor runs before or after the report is processed.
* Re-registration: the node is DECOMMISSIONING and held by the container, registers again with version 2.0.0 instead of 1.0.0 (container set empty), the monitor runs, then the report is processed. The node is DECOMMISSIONED, and stays so with the container back in its set.
* SCM restart, persisted DECOMMISSIONING and ENTERING_MAINTENANCE: with the report processed first the node stays in its admin state; with the monitor first it becomes DECOMMISSIONED or IN_MAINTENANCE.

A passing test means the defect is present.

h3. Patch
[^MC-2-wait-for-full-container-report-before-admin-completion.patch], against b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at 20d95ed37fe.

{{DatanodeInfo}} gets a flag that {{ContainerReportHandler}} sets once a full container report has been applied to the node's container set, and {{checkContainersReplicatedOnNode}} does not pass while the node's current {{DatanodeInfo}} lacks it. The flag needs no reset: SCM creates a new {{DatanodeInfo}} in exactly the two cases where the container set starts empty. Incremental reports do not set it. The change is on the gate side, so it is independent of how HDDS-16632 is fixed; carrying the container set over in {{NodeStateMap.updateNode}} alone would not cover the SCM restart case. This patch is meant to cover the admin monitor symptom that HDDS-16632 describes, so a fix for HDDS-16632 does not need to change the monitor. The flag is read after the container set, so a registration between the two reads counts as a node still waiting for its report. Behaviour change: after an SCM restart or a re-registration, a tracked node waits for its first full container report to be processed before it can complete. Normally that is the report sent with the registration; if the handler fails on that one, the node waits for the next full report ({{hdds.container.report.interval}}, 60 minutes by default) or until the datanode registers again, which sends a new report.

Covered by {{testAdminWaitsForContainerReportAfterRegistration}} (decommission and maintenance) in the existing {{TestDatanodeAdminMonitor}}, and {{testFullReportMarksContainerReportProcessed}} in the existing {{TestIncrementalContainerReportHandler}} (an incremental report does not set the flag, a full report does, after re-registering with a new version the node starts without it). Without the gate change the first fails with "expected: <DECOMMISSIONING> but was: <DECOMMISSIONED>" and "expected: <ENTERING_MAINTENANCE> but was: <IN_MAINTENANCE>". With it {{TestDatanodeAdminMonitor}}, {{TestIncrementalContainerReportHandler}}, {{TestContainerReportHandler}}, {{TestSCMNodeManager}}, {{TestNodeDecommissionManager}}, {{TestNodeDecommissionMetrics}} and {{TestDeadNodeHandler}} pass, the mini cluster suite {{TestDecommissionAndMaintenance}} (9 tests, including its SCM restart cases) passes, checkstyle is clean, and the three reproduction tests fail because the node waits for the report and is then held by the under replicated container.

Found by TLA+ model checking and code review of the SCM datanode decommission and maintenance admin paths under HDDS-15926, on commit b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. Related but not duplicates: HDDS-16632 (container set lost on re-registration, mentions this effect without a reproduction), HDDS-16925 (ContainerManager replicas lost on SCM snapshot install). The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org