[jira] [Created] (HDDS-16894) OneReplicaPipelineSafeModeRule still counts reports of pipelines closed during safe mode after its refresh lowers the threshold

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created HDDS-16894:
---------------------------------

             Summary: OneReplicaPipelineSafeModeRule still counts reports of pipelines closed during safe mode after its refresh lowers the threshold
                 Key: HDDS-16894
                 URL: https://issues.apache.org/jira/browse/HDDS-16894
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Siyao Meng
         Attachments: MC-3-drop-closed-pipeline-reports-on-one-replica-rule-refresh.patch, TestBugMC3OneReplicaRuleCountsClosedPipeline.java

h3. Mechanism
While the rule is not met, {{OneReplicaPipelineSafeModeRule.refresh(false)}} calls {{initializeRule(true)}}, which rebuilds {{oldPipelineIDSet}} and {{thresholdCount}} from the RATIS/THREE pipelines that are open now. {{currentReportedPipelineCount}} and {{reportedPipelineIDSet}} are kept as they are, so reports of pipelines that have closed since still count towards a threshold that only covers open pipelines. With R reported pipelines of which C have closed, the rule passes once R reaches ceil(0.9 x open), so it covers C fewer open pipelines than configured. Once {{SCMSafeModeManager}} records the rule as validated it is never checked again.

The class Javadoc says the rule ensures at least one reported datanode per open pipeline, so that open containers have a replica to read when SCM leaves safe mode. The container rules only count CLOSED and QUASI_CLOSED containers.

h3. Trigger
# SCM starts in safe mode with open RATIS/THREE pipelines, and the datanodes of some of them report.
# One of the reported pipelines is closed while SCM is in safe mode: by {{StaleNodeHandler}} or {{DeadNodeHandler}} after a registered datanode goes silent, by a Ratis pipeline action from a datanode, or by {{ozone admin pipeline close}}.
# The periodic refresh ({{hdds.scm.safemode.rule.refresh.interval}}, default 5s) runs. In releases before the periodic refresh thread, the refresh runs on {{SCMStateMachine.applyTransaction}} in safe mode, so the transaction that closes the pipeline triggers it (code reading, not run on a release).

h3. Impact
* Smallest case, run: two open pipelines p1 and p2 (threshold 2). Only p1's three datanodes register and report, and p1 is reported healthy. p1 is closed. The refresh sets the threshold to 1 with the count still 1, the rule validates with 0 of 1 open pipelines reported, and in the same refresh cycle SCM leaves safe mode, because every other rule already passes (3 datanodes registered, one healthy pipeline, no closed containers).
* SCM leaves safe mode only if the other rules pass. If the unreported datanodes hold closed containers with keys, the container rule still holds SCM in safe mode unless those containers are under 1% of the total. The rule's own bound already allows 10% of open pipelines to stay unreported. The bug loosens that by one pipeline per reported pipeline closed during safe mode.
* After an early exit, {{getContainerWithPipeline}} no longer refuses open containers without reported replicas, so reads of those containers go to datanodes that have not reported until they come back, and block allocation can pick the unreported open pipeline (code reading, not run). No data loss is expected.
* The refresh path dates from HDDS-5263 (1.2.0). Related: HDDS-11844 proposes changing what this rule requires before SCM leaves safe mode. This report and patch keep the current requirement.

h3. Reproduction
[^TestBugMC3OneReplicaRuleCountsClosedPipeline.java] ({{hadoop-hdds/server-scm}}) runs the smallest case above with the real {{PipelineManagerImpl}}, the real {{SCMSafeModeManager}} with all its rules and its refresh thread (interval lowered to 100ms), real registration, pipeline report and open pipeline events, and {{PipelineManagerImpl.closePipeline}}. On the unmodified commit it fails every time with {{SCM left safe mode with 0 of 1 open pipelines reported}}, and the log shows {{OneReplicaPipelineSafeModeRule rule is successfully validated}} followed by {{SCM exiting safe mode.}} With the patch it passes. Not run on a real cluster.

h3. Patch
[^MC-3-drop-closed-pipeline-reports-on-one-replica-rule-refresh.patch], against b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at 6818707698c.

On refresh, {{initializeRule(true)}} keeps only the reported pipelines that are still open ({{reportedPipelineIDSet.retainAll(oldPipelineIDSet)}}) and lowers {{currentReportedPipelineCount}} by the number removed. Reports of pipelines that are still open are kept. The count is lowered rather than set to the set size because {{cleanup()}} empties the set after the rule has passed. From the code, this also covers the report free validation path ({{updateReportedPipelineSet}}), which uses the same set and count. That path is not tested. Nothing changes when no reported pipeline closes during safe mode. The metric {{currentPipelinesWithAtleastOneReplicaReportedCount}} is a counter and still includes the closed pipelines.

Covered by {{testRefreshDropsReportsOfPipelinesNoLongerOpen}} in the existing {{TestOneReplicaPipelineSafeModeRule}}: three open pipelines, two report, one of those closes, and after the refresh the rule expects threshold 2 and count 1 and is not met. A report from the third pipeline then satisfies it. Without the change it fails ({{expected: <1> but was: <2>}}). With it, the {{org.apache.hadoop.hdds.scm.safemode}} unit tests (75) and {{TestSCMSafeModeWithPipelineRules}} on a mini cluster (2) pass, and checkstyle is clean.

Found by TLA+ model checking and code review of the SCM safe mode rules under HDDS-15926, on commit b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org