[jira] [Assigned] (HDDS-17065) Recon ContainerSizeCountTask stops for good after one failed run, freezing the container size distribution until restart

["Andy Huang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-17065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andy Huang reassigned HDDS-17065:
---------------------------------

    Assignee: Andy Huang

> Recon ContainerSizeCountTask stops for good after one failed run, freezing the container size distribution until restart
> ------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-17065
>                 URL: https://issues.apache.org/jira/browse/HDDS-17065
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Siyao Meng
>            Assignee: Andy Huang
>            Priority: Minor
>         Attachments: CR-5-keep-container-size-count-task-running-and-recount-after-failed-run.patch, TestBugCR5ContainerSizeCountTaskStopsAfterFailedRun.java
>
>
> h3. Mechanism
> {{ContainerSizeCountTask.run}} has its {{catch (Throwable)}} outside the {{while (canRun())}} loop, and {{ReconScmTask.initializeAndRunTask}} passes every exception up. When one run fails, the thread logs "Exception in Container Size Distribution task Thread." at DEBUG and ends. The task's running flag stays true, so {{isRunning()}} keeps reporting the task as running and {{start()}} would not start it again; nothing else does either.
> What fails in practice is the Recon SQL database: the {{truncate}} in {{runTask}}, or {{findById}}, {{insert}} or {{update}} in {{writeCountsToDB}}, all of which throw a jOOQ {{DataAccessException}} when the database rejects the call. Listing the containers is in memory, and errors for a single container are already caught per container.
> Letting the loop simply continue is not enough. {{processContainers}} updates its in memory {{processedContainers}} before {{writeCountsToDB}}, and that write is an insert followed by a separate update. After a failed write the next run computes its delta from the already updated map, so the failed change is never applied; if the insert went through and the update did not, the table also keeps half of the change. Either way the counts stay wrong until Recon restarts, which recounts from scratch.
> h3. Trigger
> Any failure of the Recon SQL database during one run of the task (default every 60s, {{ozone.recon.task.containercounttask.interval}}), for example an outage of an external database configured with {{ozone.recon.sql.db.jdbc.url}}, or a full disk under the embedded Derby database.
> h3. Impact
> * After the thread has ended, {{GET /api/v1/utilization/containerCount}} and the "Container Size Distribution" chart on Recon's Insights page keep showing the counts of the last successful run until Recon restarts. If the failed run was the first one after startup, they show nothing, or the rows the previous Recon process left, since the table persists across restarts. The task logs the failure only at DEBUG. In a real database outage {{ReconTaskStatusUpdater}} also logs an ERROR "Failed to update table for task" without the cause, but nothing reports that the thread has ended. {{/api/v1/task/status}} shows the task's last run as failed, with a timestamp that no longer advances.
> * Only this statistic is affected; the endpoint is its only reader. SCM, datanodes and data are not affected.
> h3. Reproduction
> [^TestBugCR5ContainerSizeCountTaskStopsAfterFailedRun.java] goes in {{hadoop-ozone/recon}} (target path and Maven command in the class comment) and passes on unmodified source when the bug is present. It runs the real {{ContainerSizeCountTask}} on Recon's real SQL schema (embedded Derby) and reads the result through the real {{UtilizationEndpoint}}; the container manager is a mock and the DAO is a spy of the real DAO whose first write throws {{DataAccessException}}.
> * {{oneFailedRunStopsTheTask}} (200ms interval): the first run fails, then the cluster gains a second container. After 12 intervals the task had listed the containers once, the endpoint returned no rows for two containers, {{isRunning()}} returned true, and the task logged nothing at INFO or above (the failure is injected at the size count DAO only, so the task status writes succeed).
> * {{retryAfterPartialWriteKeepsWrongCounts}} shows why a plain retry is not enough: runs are called directly, a container moves from the 2GB bin to the 4GB bin, the new 4GB row is inserted and the update of the 2GB row fails. After the next successful run the endpoint still returns one container in each bin for a single 2.5GB container.
> What was not exercised: a real outage of an external database or a full disk; the failure is injected at the DAO.
> h3. Patch
> [^CR-5-keep-container-size-count-task-running-and-recount-after-failed-run.patch], against 916aa1d9949cbb10f2750a40c8704340893131a2. It also applies to master at 14e0611fdd5.
> {{ContainerSizeCountTask.run}} now catches a failed run inside the loop, logs it at ERROR, records the failed run in the task status and waits the interval before the next run. {{processContainers}} clears {{processedContainers}} when the run fails, so the next run truncates the table and recounts all containers, as Recon does at startup. A recount is used instead of rolling back the in memory state because the table may already hold part of the failed write. The recount costs what Recon already does on its first run after startup: a truncate and one row per non empty size bin, on top of the pass over the container list that every run makes. A reader may see an empty table for that moment, as after a restart.
> Covered by {{TestContainerSizeCountTask.testTaskKeepsRunningAfterFailedRun}} (the first write fails, the container must still reach the table) and {{TestContainerSizeCountTask.testCountsRecoverAfterFailedWrite}} (after a partial write and one more run only the new bin may be counted). Without the change the first times out and the second sees both bins. With it the size count, endpoint, SCM sync helper and SCM facade tests pass, checkstyle is clean, and the attached test no longer shows the bug.
> The same death pattern in {{PipelineSyncTask}} is a separate issue with its own patch; the two patches touch different files and apply together.
> Found by code review of the datanode container report and Recon container tracking paths, as part of the TLA+ verification effort under HDDS-15926, on commit 916aa1d9949cbb10f2750a40c8704340893131a2. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. Related but not duplicates: HDDS-12227 (resolved, lowered this log line to DEBUG), HDDS-10293 (resolved, negative container size, caught per container), HDDS-7632 (resolved, added the task). The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org