[jira] [Assigned] (HDDS-16455) Stale scrub task demotes a CLOSED reconstructed EC replica to UNHEALTHY, forcing SCM re reconstruction

["Chi-Hsuan Huang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-16455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chi-Hsuan Huang reassigned HDDS-16455:
--------------------------------------

    Assignee: Chi-Hsuan Huang

> Stale scrub task demotes a CLOSED reconstructed EC replica to UNHEALTHY, forcing SCM re reconstruction
> ------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-16455
>                 URL: https://issues.apache.org/jira/browse/HDDS-16455
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Siyao Meng
>            Assignee: Chi-Hsuan Huang
>            Priority: Major
>         Attachments: HDDS-16455.001.patch
>
>
> h3. Finding
> A stale scrub task silently demotes a successfully-CLOSED reconstructed EC replica to UNHEALTHY with no ICR; on the next full container report SCM deletes and re-reconstructs a valid replica. Reachable via the normal scrub background thread racing the reconstruction/close path; harm is persistent on the datanode but bounded (SCM re-replicates), so High rather than Critical.
> h3. Classification
> * Verdict: REPRODUCED
> * Severity: High
> * Source: Specula TLA+ model checking and confirmation debate, finding MC-1
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula:      v1.1.0 (commit c6aa3dfa)
> Target:       dn-ec-reconstruction
> Guidance:     campaigns/ozone-9fbf9ee/targets/022-dn-ec-reconstruction/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=high --keep-original --max-parallel=2 \
>   --enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8 \
>   "dn-ec-reconstruction|apache/ozone|Java|Use the target-specific .prompt-extra.md"
> {code}
> Discovered under HDDS-16433 (Specula TLA+ verification effort). The TLA+ specification, counterexample, and confirmation debate live in the Specula run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> 1. An EC reconstruction target is created RECOVERING; its {{recoveringContainerMap}} entry carries a deadline (create time plus recovering timeout).
> 2. Reconstruction stalls past the deadline, so {{StaleRecoveringContainerScrubbingService.getTasks}} enqueues a {{RecoveringContainerScrubbingTask}} and removes the map entry, all while the container is still RECOVERING.
> 3. Reconstruction then completes: the replica is moved RECOVERING to CLOSING to CLOSED through the normal close path.
> 4. The queued scrub task fires on the background pool. It re-checks only the deadline (now null, so it proceeds) and calls {{markContainerUnhealthy()}}, which has no state guard, so the successfully CLOSED replica is silently demoted to UNHEALTHY. No ICR is sent, so SCM only learns of the UNHEALTHY replica on the next full container report and then deletes and re-reconstructs a valid replica.
> h3. Root cause
> {{StaleRecoveringContainerScrubbingService}} splits its "decide stale" step ({{getTasks}}) from its "act" step ({{RecoveringContainerScrubbingTask.call}}) across two threads with no state re-check between them. {{call()}} re-reads only the deadline (added earlier for the deadline-extension race) and never re-checks the container state before calling {{Container.markContainerUnhealthy}}. {{KeyValueContainer.markContainerUnhealthy}} unconditionally sets state to UNHEALTHY, so a container that reached a terminal CLOSED (or CLOSING) state between scheduling and execution is degraded anyway.
> h3. Suggested fix
> Attached patch [^HDDS-16455.001.patch] re-checks the container state under the container write lock inside {{RecoveringContainerScrubbingTask.call}} and marks the container UNHEALTHY only if it is still RECOVERING, mirroring the write-lock discipline that {{KeyValueHandler.markContainerForClose}} already uses. A container that has moved out of RECOVERING (closed or deleted) is left untouched. A unit test in {{TestStaleRecoveringContainerScrubbingService}} drives the schedule then close then execute interleaving and asserts the replica stays CLOSED.
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full test, and review before merge.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org