[jira] [Commented] (HDDS-17071) Recon PipelineSyncTask stops after one failed cycle and ContainerHealthTask retries a failed sweep without waiting

["Andy Huang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDDS-17071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18129155#comment-18129155 ] 

Andy Huang commented on HDDS-17071:
-----------------------------------

PR #11593 was closed because its fixes are now covered by PR #11585, which handles failed runs centrally in ReconScmTask for all three tasks.

The regression tests from #11593 were carried over to #11585 in commit 13d2c6f1b2, covering pipeline sync recovery after an SCM failure and the wait after a failed container health sweep. Both tests failed against the pre-fix implementation and passed with #11585.

This issue will be addressed by https://github.com/apache/ozone/pull/11585.

> Recon PipelineSyncTask stops after one failed cycle and ContainerHealthTask retries a failed sweep without waiting
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-17071
>                 URL: https://issues.apache.org/jira/browse/HDDS-17071
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Siyao Meng
>            Assignee: Andy Huang
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: MC-6-keep-recon-scm-tasks-running-after-failed-cycle.patch, TestBugMC6ReconTaskLoopAfterFailedCycle.java
>
>
> h3. Mechanism
> Both are periodic Recon tasks built on {{ReconScmTask}}, whose {{initializeAndRunTask}} passes every exception up to the task's {{run}} loop.
> * {{PipelineSyncTask.run}} has its {{catch (Throwable)}} outside the {{while (canRun())}} loop. When one cycle fails, the thread logs "Exception in Pipeline sync Thread." and ends. The task's running flag stays true, so {{isRunning()}} keeps reporting the task as running and {{start()}} would not start it again; nothing else does either. A cycle fails when {{scmClient.getPipelines()}} or {{getNodes()}} throws: Recon's SCM client retries are bounded ({{ozone.recon.scmclient.max.retry.timeout}} 6s, {{ozone.recon.scmclient.failover.max.retry}} 3), so an SCM outage longer than a few seconds reaches the task as an {{IOException}}. It also fails when a write to Recon's own SCM DB fails; HDDS-10017 shows this exception from a real cluster, where {{ReconPipelineManager.initializePipelines}} failed on such a write.
> * {{ContainerHealthTask.run}} catches per cycle, but its interval sleep is inside the {{try}} after the sweep. When the sweep throws, for example because the Recon SQL database rejects the write of {{UNHEALTHY_CONTAINERS}}, the next full sweep starts at once instead of after the interval (at least 60s).
> The {{PipelineSyncTask}} loop has this shape in every release since ozone-1.0.0 (read from the tagged source). The health task hot loop came with HDDS-13891 (ozone-2.2.0 and 2.2.1); in earlier releases that thread instead ended on its first failure, like the pipeline sync.
> h3. Trigger
> * Pipeline sync: any cycle that fails (default every 5 minutes), such as during an SCM restart, upgrade or failover, or on an error from Recon's local SCM DB.
> * Health task: any error from the sweep, for example an outage of an external Recon SQL database ({{ozone.recon.sql.db.jdbc.url}}).
> h3. Impact
> * After the pipeline sync thread has ended, Recon no longer refreshes the operational state of DEAD datanodes from SCM, and no longer closes or removes pipelines that SCM has closed or removed, until Recon restarts. The sync still runs when some datanode goes STALE or DEAD (those handlers call it directly), so a busy cluster may hide the problem while a quiet one does not. Live datanodes are not affected because they report their state in heartbeats.
> * A datanode that is DEAD while in maintenance keeps its replicas in Recon ({{DeadNodeHandler}} skips the removal for nodes in maintenance). When SCM ends the maintenance, the periodic sync is what copies IN_SERVICE to Recon, which fires DEAD_NODE again and removes the replicas. With the thread ended, Recon keeps showing the node IN_MAINTENANCE and keeps its replicas. In the attached run with real {{ReconNodeManager}}, {{PipelineSyncTask}} and {{ReconDeadNodeHandler}}, after 25 sync intervals SCM had IN_SERVICE, Recon had IN_MAINTENANCE, the task had made no further SCM call while {{isRunning()}} returned true, and the replica was never removed. That the kept replica then hides the container from Recon's under replicated and missing reports is argued from the code, not run.
> * A failing health sweep runs back to back, each failure logging a stack trace, for as long as the failure lasts. The attached run counted about 20000 failing sweeps (mocked, failing at once) in about one second. On a real cluster each failed sweep still walks every container before the database write fails, so the sweeps run back to back at the speed of a full sweep.
> h3. Reproduction
> [^TestBugMC6ReconTaskLoopAfterFailedCycle.java] goes in {{hadoop-ozone/recon}} (target path and Maven command in the class comment) and passes on unmodified source when the bugs are present. The first test uses a real {{ReconNodeManager}} with short stale and dead intervals, a real {{PipelineSyncTask}} with a 200ms interval and a real {{ReconDeadNodeHandler}}; only the SCM RPC client is mocked (its first {{getPipelines()}} throws {{IOException}}) and the container manager is a mock so the replica removal can be observed. The second runs the real {{ContainerHealthTask}} thread over a mocked {{ReconReplicationManager}} whose {{processAll}} throws a jOOQ {{DataAccessException}}.
> h3. Patch
> [^MC-6-keep-recon-scm-tasks-running-after-failed-cycle.patch], against 916aa1d9949cbb10f2750a40c8704340893131a2. It also applies to master at d77edaa6c15.
> {{PipelineSyncTask.run}} now catches {{Exception}} per cycle, logs the same error, records the failed run in the task status and sleeps the interval before the next cycle. {{ContainerHealthTask.run}} moves its existing catch directly after the sweep so the interval sleep follows a failed sweep too. In both, an interrupt still ends the loop: the per cycle catch restores the interrupt flag of an {{InterruptedException}}, so the sleep that follows throws and the loop exits as before. An {{Error}} still ends the thread, as it does in {{ContainerHealthTask}} today. This is the per cycle pattern the SCM container sync and OM sync schedulers in Recon already use.
> Covered by the new {{TestPipelineSyncTask.keepsRunningAfterFailedSync}} (first {{getPipelines()}} fails, a later run must succeed) and by {{TestContainerHealthTask.testFailedRunWaitsForNextInterval}} (a failing sweep must run once in the first second). Without the change the first sees only the failed run and the second counted about 20000 sweeps. With it the Recon {{scm}} and {{fsck}} unit tests pass, checkstyle is clean, and both attached tests no longer show the bugs.
> {{ContainerSizeCountTask}} has the same catch outside its loop; it is tracked in HDDS-17065.
> Found by TLA+ model checking and code review of the datanode container report and Recon container tracking paths under HDDS-15926, on commit 916aa1d9949cbb10f2750a40c8704340893131a2. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. Related but not duplicates: HDDS-5331 (resolved, added the stale and dead node triggers for this task), HDDS-10017 (Cannot Reproduce, a field log of this exception after a failed write to Recon's SCM DB), HDDS-5814 (resolved, interrupt handling in these loops), HDDS-13891 (resolved, introduced the current {{ContainerHealthTask}} loop). The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org