[jira] [Created] (HDDS-15791) EC reconstructed RECOVERING container while still idle can be deleted by SCM and recreated as new partial OPEN container

["Aswin Shakil (Jira)" <ji...@apache.org>]

Aswin Shakil created HDDS-15791:
-----------------------------------

             Summary: EC reconstructed RECOVERING container while still idle can be deleted by SCM and recreated as new partial OPEN container
                 Key: HDDS-15791
                 URL: https://issues.apache.org/jira/browse/HDDS-15791
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Aswin Shakil
            Assignee: Aswin Shakil


This issue has been identified by [~deveshsingh] and [~umamahesh] 

During EC offline reconstruction, the coordinator creates a *RECOVERING* replica on a spare DataNode and writes blocks over time. If reconstruction stalls (e.g. slow block-group processing) longer than `ozone.recovering.container.timeout` (~20m, measured from creation, not last write), the DN scrubber marks the replica *UNHEALTHY* and SCM force-deletes it.

The coordinator may still be running. The next `WriteChunk` or `PutBlock` finds no container; `HddsDispatcher` auto-creates a new OPEN container instead of failing. The coordinator continues and closes that replica shortly after, leaving a partial container (fewer blocks/chunks than the reference) that SCM can accept as *CLOSED* — silent data loss.

*Logic flow*

{code:java}
Coordinator Phase 1
  createRecoveringContainer  →  RECOVERING on spare DN
  (slow / sleep — no writes for extended period)

StaleRecoveringContainerScrubbingService
  creationTime + timeout exceeded  →  mark UNHEALTHY

SCM / Replication Manager
  force-delete UNHEALTHY replica  →  DELETED on spare DN

Coordinator resumes writes
  container == null
  HddsDispatcher auto-create      →  OPEN (not RECOVERING)

Coordinator completes
  closeContainer                  →  CLOSING / CLOSED

Result: CLOSED replica with partial blocks/chunks at EC index → data loss
{code}


 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org