[jira] [Updated] (HDDS-16906) Pipeline with a DEAD datanode stays OPEN if SCM leadership changes while the node goes STALE and DEAD, and its EC container is not repaired

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-16906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDDS-16906:
------------------------------
    Attachment: TestBugMC7DeadEcPipelineMemberAfterFailovers.java

> Pipeline with a DEAD datanode stays OPEN if SCM leadership changes while the node goes STALE and DEAD, and its EC container is not repaired
> -------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-16906
>                 URL: https://issues.apache.org/jira/browse/HDDS-16906
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Siyao Meng
>            Priority: Minor
>         Attachments: MC-7-scrub-open-pipelines-with-dead-nodes.patch, TestBugMC7DeadEcPipelineMemberAfterFailovers.java
>
>
> h3. Mechanism
> Every SCM, leader or follower, runs its own node health check and fires {{STALE_NODE}} and {{DEAD_NODE}} once per transition. {{StaleNodeHandler}} and {{DeadNodeHandler}} close the node's pipelines through {{PipelineManager.closePipeline}}, which is a Ratis request: on a follower it fails with NotLeaderException. {{StaleNodeHandler}} logs "Could not finalize pipeline" at INFO without the exception, {{DeadNodeHandler}} logs the exception, and both drop it. No SCM fires the events again for a node that is already DEAD, and nothing re-runs the handlers when an SCM becomes leader. The only periodic check, {{PipelineManagerImpl.scrubPipelines}} on the leader, closes an OPEN pipeline only if a member is not registered (HDDS-7341); a DEAD datanode stays registered.
> So if the SCM that becomes leader saw the STALE transition while it was a follower, and the same happens for the DEAD transition, no SCM closes the pipeline. For RATIS/THREE pipelines the remaining datanodes can still trigger the close through pipeline actions (argued, not run). A RATIS/ONE pipeline on the dead datanode and EC pipelines have no such path, because no other member is left or because EC datanodes are not aware of pipelines.
> h3. Trigger
> # An OPEN EC pipeline, three SCMs, and a datanode of the pipeline stops.
> # The datanode goes STALE. A follower sees it first, its handler fails, and that follower becomes leader before the old leader's close has been applied.
> # The datanode goes DEAD. Again the SCM that becomes leader saw it first as a follower.
> Each SCM checks node health on its own timer ({{ozone.scm.heartbeat.thread.interval}}, default 3s), so each leader change has to fall in a window of a few seconds around the transition, at both transitions (5 minutes apart with the default stale and dead intervals). A close that fails on the leader for another reason is lost the same way (argued, not run).
> h3. Impact
> * The EC pipeline stays OPEN with a DEAD member. {{SCMBlockProtocolServer.allocateBlock}} keeps returning it to requests that do not exclude it. The client then fails over: it adds the pipeline to its exclude list and asks for another one (argued from the client code, not run).
> * Its OPEN container keeps 4 of 5 replicas. {{ReplicationManager}} does not reconstruct OPEN containers, and the container cannot fill, so it stays under replicated until the datanode returns, an admin closes the pipeline or container, or an SCM that was restarted after the node died becomes leader (then the node is unregistered there and the existing scrubber rule closes the pipeline).
> * In the same run, two RATIS/THREE pipelines and one RATIS/ONE pipeline with the dead datanode were also still OPEN on the new leader after a scrubber pass. Whether the datanodes close the RATIS/THREE ones later was not observed.
> h3. Reproduction
> PASS, unmodified source. [^TestBugMC7DeadEcPipelineMemberAfterFailovers.java] ({{hadoop-ozone/integration-test}}) starts 3 SCMs, 1 OM and 5 datanodes with the stale and dead intervals shortened to 6s and 40s, and writes one EC 3-2 key with the client. It widens the two windows with the existing {{SCMNodeManager.pauseHealthCheck}} test hook on the current leader and moves leadership with the admin {{transferLeadership}} RPC: followers B and C see STALE, leadership moves A to B, followers A and C see DEAD, leadership moves B to C. On C it then runs one scrubber pass and one ReplicationManager pass. Output on the pin: the EC pipeline is OPEN with a DEAD member, its container is OPEN with 4 replicas, {{allocateBlock}} for EC 3-2 returns that pipeline, and the log shows the stale node handler's "Could not finalize pipeline" lines and the dead node handler failing with NotLeaderException on the followers. As a control, a second member stopped while C is leader gets the pipeline closed. A passing test means the defect is present. With the patch it fails at the first check, because the scrubber pass closes the pipeline and the container moves to CLOSING.
> h3. Patch
> [^MC-7-scrub-open-pipelines-with-dead-nodes.patch], against b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at b9b33851b20.
> {{scrubPipelines}} also closes an OPEN pipeline that has a DEAD member, next to the existing rule for unregistered members. The scrubber runs only on the leader, so the close no longer depends on which SCM saw the transition, and it also covers a close that failed on a stable leader. {{closePipeline}} closes the pipeline's containers too, so ReplicationManager can repair them once they are closed. The scrubber runs only on a leader that is ready and out of safe mode. In normal operation the leader's stale or dead node handler has already closed every pipeline of a DEAD node, of any type, so the rule finds nothing. If such a close was lost, the scrubber does it at most one scrub interval (default 5 minutes) after the node is DEAD. It does not act on STALE members. Covered by {{testScrubOpenWithDeadNodes}} in the existing {{TestPipelineManagerImpl}} (a STALE member keeps the pipeline open, a DEAD member closes it), which fails without the change. With it the SCM pipeline unit suites, {{TestSCMSafeModeManager}}, {{TestSCMNodeManager}} and {{TestDeadNodeHandler}} (359 tests) and {{TestDecommissionAndMaintenance}} (9 tests) pass, checkstyle is clean.
> Related: HDDS-16887. If {{DeadNodeHandler}} deletes the pipeline during a scrubber pass, the new close hits the {{PipelineNotFoundException}} that HDDS-16887 handles (today it ends the pass, as it already can for the existing rules). HDDS-16887 also wraps the loop body in {{scrubPipelines}} in a {{try}} block, so the patches conflict textually and whichever lands second needs a trivial rebase.
> Found by TLA+ model checking and code review of the SCM pipeline lifecycle paths under HDDS-15926, on commit b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org