[jira] [Updated] (HDDS-16903) HealthyPipelineSafeModeRule counts DORMANT and CLOSED pipelines, so SCM can leave safe mode with no healthy OPEN RATIS/THREE pipeline

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-16903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDDS-16903:
------------------------------
    Attachment: MC-4-count-only-open-pipelines-in-healthy-pipeline-rule.patch

> HealthyPipelineSafeModeRule counts DORMANT and CLOSED pipelines, so SCM can leave safe mode with no healthy OPEN RATIS/THREE pipeline
> -------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-16903
>                 URL: https://issues.apache.org/jira/browse/HDDS-16903
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Siyao Meng
>            Priority: Minor
>         Attachments: MC-4-count-only-open-pipelines-in-healthy-pipeline-rule.patch, TestBugMC4NonOpenPipelineCountsAsHealthy.java
>
>
> h3. Mechanism
> While SCM is in safe mode, {{PipelineReportHandler.processPipelineReport}} fires {{OPEN_PIPELINE}} for every reported pipeline that is healthy (all members reported, leader known), whatever its state in SCM. {{HealthyPipelineSafeModeRule.process}}, the only subscriber, checks the replication config, whether the pipeline was already counted, the member count and the member health, but not the state. A DORMANT or CLOSED RATIS/THREE pipeline whose datanodes still host the Ratis group is therefore counted, although the threshold in {{initializeRule}} is computed from OPEN pipelines only. The count never goes down, and once every rule has passed SCM does not return to safe mode.
> The other checks in the same area count OPEN pipelines only: {{OneReplicaPipelineSafeModeRule.process}} checks {{isOpen()}}, and so does the report free path {{validateHealthyPipelineSafeModeRuleUsingPipelineManager}} (HDDS-13590). HDDS-4399 ("Safe mode rule for pipelines should only consider open pipelines") moved the {{fireEvent}} out of the ALLOCATED branch so that pipelines reloaded as OPEN are counted, and the state check was lost with that move.
> h3. Trigger
> Datanodes keep a pipeline's Ratis group, and keep reporting it, until SCM sends a close command. For a pipeline that was deactivated, or closed by a pipeline action, an admin close or {{StaleNodeHandler}}, SCM sends it only when the pipeline scrubber deletes the CLOSED pipeline, and the scrubber runs only outside safe mode. Two ways to get there:
> * DORMANT: an admin runs {{ozone admin pipeline deactivate}}. Only the SCM state changes, and the state is persisted, so the pipeline counts at every later SCM restart for as long as its datanodes are up.
> * CLOSED: during safe mode a member asks SCM to close the pipeline (pipeline action, for example after a Ratis failure), an admin runs {{ozone admin pipeline close}}, or a member goes stale for a while (for example a long GC pause) and {{StaleNodeHandler}} closes its pipelines before it recovers. The members keep reporting the pipeline for the rest of safe mode. A pipeline closed shortly before an SCM restart is loaded as CLOSED and behaves the same (argued, not run).
> SCM leaves safe mode on such a pipeline once the other rules pass, for example after a restart where an OPEN pipeline has only one live member: that one member satisfies {{OneReplicaPipelineSafeModeRule}}, but the pipeline cannot become healthy.
> h3. Impact
> SCM leaves safe mode although no OPEN RATIS/THREE pipeline is healthy. That is the situation this rule exists to prevent (HDDS-1178). Block allocation is then allowed, and new RATIS/THREE writes can only proceed once a new pipeline is created and opened. Where the datanodes have room for a new pipeline, this mostly makes the exit earlier. Where they have none (for example, the slots are held by the DORMANT pipeline), writes keep failing until an operator acts (argued from the code, not run). No data is lost, and the container rules still gate the exit. The state check is missing since 1.1.0 (HDDS-4399).
> h3. Reproduction
> [^TestBugMC4NonOpenPipelineCountsAsHealthy.java] ({{hadoop-hdds/server-scm}}) uses the real {{PipelineManagerImpl}}, {{PipelineReportHandler}}, {{PipelineActionHandler}} and {{SCMSafeModeManager}} with all its rules, driven only by datanode registration, pipeline report and pipeline action events. Pipeline P1 is OPEN with one live member. Pipeline P2 is on three other live datanodes and is deactivated before the start (first test), or closed by a member's close action during safe mode (second test). Before P2's members report it, only {{HealthyPipelineSafeModeRule}} is unmet. After their reports, SCM leaves safe mode with P1 as the only OPEN RATIS/THREE pipeline and P1 not healthy. Both tests pass on unmodified source, which means the defect is present. With the patch, both fail because SCM stays in safe mode. The reproduction does not model datanodes opening pipelines that the background pipeline creator allocates.
> h3. Patch
> [^MC-4-count-only-open-pipelines-in-healthy-pipeline-rule.patch], against b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at a5aeb764d80.
> {{HealthyPipelineSafeModeRule.process}} reads the pipeline's current state from {{PipelineManager}} and skips it unless it is OPEN, as {{OneReplicaPipelineSafeModeRule}} does. The check is in the rule rather than in {{PipelineReportHandler}} for two reasons. The {{Pipeline}} object the handler fires is the copy it read before calling {{openPipeline}}, so it still says ALLOCATED for a pipeline that the same report has just opened. And the event is processed asynchronously. {{PipelineReportHandler}} is unchanged.
> One behaviour change: if the only fully reported RATIS/THREE pipelines are DORMANT or CLOSED and no OPEN one can become healthy, SCM now stays in safe mode until one does or an operator runs {{ozone admin safemode exit}}, instead of leaving it without a writable pipeline. With the default {{hdds.scm.safemode.pipeline.creation=true}} a CLOSED pipeline only delays the exit while three healthy datanodes remain: CLOSED pipelines do not count against the datanode pipeline limit, so the background creator builds a new one. The case that changes is DORMANT pipelines taking every free pipeline slot, for example an admin who deactivates all pipelines and then restarts SCM. SCM then stays in safe mode, so {{ReplicationManager}} does not run either, until the admin activates a pipeline or runs {{ozone admin safemode exit}}.
> Covered by {{testPipelineNotOpenIsIgnored}} in the existing {{TestHealthyPipelineSafeModeRule}}: with one OPEN, one DORMANT and one CLOSED pipeline, all fully reported, the rule must not count the DORMANT or the CLOSED pipeline, and must pass on the OPEN one. The events carry stale copies, as the handler's may: an OPEN copy of the DORMANT and of the CLOSED pipeline, and the ALLOCATED copy of the OPEN pipeline, so a check on the event's own copy fails the test as well. Without the change it fails ({{expected: 0 but was: 1}}). With it, the {{org.apache.hadoop.hdds.scm.safemode}} unit tests together with {{TestPipelineManagerImpl}} and {{TestPipelineActionHandler}} (108 tests) pass, as do {{TestSCMSafeModeWithPipelineRules}} and {{TestSafeModeCheckSubcommandHA}} on a mini cluster (8 tests), and checkstyle is clean.
> Found by TLA+ model checking and code review of the SCM safe mode rules under HDDS-15926, on commit b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. This issue also covers the same defect found by code review of the SCM pipeline lifecycle under HDDS-16538, which approached it from {{PipelineReportHandler}} and the DORMANT state. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. Related but not duplicates: HDDS-4399 (introduced the state independent event), HDDS-13590 (OPEN filter for the report free path only), HDDS-11904 (datanode health check in the same method) and HDDS-11844 (open, proposes not waiting for all pipelines). The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org