[jira] [Resolved] (HDDS-15791) EC reconstructed RECOVERING container while still idle can be deleted by SCM and recreated as new partial OPEN container

["Wei-Chiu Chuang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-15791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wei-Chiu Chuang resolved HDDS-15791.
------------------------------------
    Fix Version/s: 2.3.0
       Resolution: Fixed

> EC reconstructed RECOVERING container while still idle can be deleted by SCM and recreated as new partial OPEN container
> ------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-15791
>                 URL: https://issues.apache.org/jira/browse/HDDS-15791
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Aswin Shakil
>            Assignee: Aswin Shakil
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 2.3.0
>
>
> This issue has been identified by [~deveshsingh] and [~umamahesh] 
> During EC offline reconstruction, the coordinator creates a *RECOVERING* replica on a spare DataNode and writes blocks over time. If reconstruction stalls (e.g. slow block-group processing) longer than `ozone.recovering.container.timeout` (~20m, measured from creation, not last write), the DN scrubber marks the replica *UNHEALTHY* and SCM force-deletes it.
> The coordinator may still be running. The next `WriteChunk` finds no container; `HddsDispatcher` auto-creates a new OPEN container instead of failing. The coordinator continues and closes that replica shortly after, leaving a partial container (fewer blocks/chunks than the reference) that SCM can accept as *CLOSED* — silent data loss.
> *Logic flow*
> {code:java}
> Coordinator Phase 1
>   createRecoveringContainer  →  RECOVERING on spare DN
>   (slow / sleep — no writes for extended period)
> StaleRecoveringContainerScrubbingService
>   creationTime + timeout exceeded  →  mark UNHEALTHY
> SCM / Replication Manager
>   force-delete UNHEALTHY replica  →  DELETED on spare DN
> Coordinator resumes writes
>   container == null
>   HddsDispatcher auto-create      →  OPEN (not RECOVERING)
> Coordinator completes
>   closeContainer                  →  CLOSING / CLOSED
> Result: CLOSED replica with partial blocks/chunks at EC index → data loss
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org