[jira] [Updated] (HDDS-16504) Datanode startup can hang indefinitely when Ratis group recovery stalls (add a timeout/watchdog around container initialization)

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-16504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated HDDS-16504:
----------------------------------
    Labels: pull-request-available  (was: )

> Datanode startup can hang indefinitely when Ratis group recovery stalls (add a timeout/watchdog around container initialization)
> --------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-16504
>                 URL: https://issues.apache.org/jira/browse/HDDS-16504
>             Project: Apache Ozone
>          Issue Type: Improvement
>          Components: Ozone Datanode
>            Reporter: Devesh Kumar Singh
>            Assignee: Devesh Kumar Singh
>            Priority: Major
>              Labels: pull-request-available
>
> [HDDS-16425|https://issues.apache.org/jira/browse/HDDS-16425] makes datanode startup fail cleanly when container initialization throws (e.g., a corrupt Raft log with raft.server.log.corruption.policy = EXCEPTION): the state is marked FAILED, waiting callers fail fast, and the datanode shuts down.
> However, it does not cover the case where initialization hangs without throwing. If OzoneContainer.start() → writeChannel.start() → XceiverServerRatis.start() → Ratis server.start() blocks while recovering a Raft group — for example a slow or failing volume where a disk read never returns — then:
> initializeContainerServices(...) never returns, so the thread holding initializationLock holds it indefinitely.
> OzoneContainer.start() never reaches INITIALIZED or FAILED.
> VersionEndpointTask never advances the endpoint past GETVERSION, so the datanode never registers or heartbeats.
> RunningDatanodeState logs an endpoint‑task timeout every heartbeat interval, forever, with no forward progress and no clean shutdown.
> Net effect: the datanode is a "zombie" — the process is up but never registers, and (unlike the throw path) it neither recovers nor fails cleanly. Only an external restart clears it, and on restart the same group recovery is attempted again, reproducing the hang.
> *Current behavior*
> * Throw during init → handled by HDDS‑16425 (FAILED + clean shutdown). 
> * Indefinite block during init (I/O stall in group recovery) → infinite hold of initializationLock, no FAILED, no shutdown. 
> *Proposed change*
> Add a bounded timeout/watchdog around container initialization so a stalled startup is converted into an initialization failure that reuses the HDDS‑16425 clean‑shutdown path:
> * Run initializeContainerServices(...) (or at minimum the writeChannel.start() group‑recovery step) with a configurable timeout.
> * If it does not complete within the timeout, mark initializingStatus = FAILED, record a descriptive cause (e.g., TimeoutException("Container initialization did not complete within <N>s")), release waiting callers with that failure, and trigger the datanode shutdown path — the same terminal outcome as a thrown initialization error.
> * Because a thread blocked in a native/disk read generally cannot be interrupted reliably, the watchdog should not depend on unblocking the stuck thread; marking FAILED and initiating shutdown is sufficient (the process will terminate).
> *Configuration*
> New config key (e.g., hdds.datanode.container.init.timeout) with a generous default (group recovery on large logs can legitimately take time; the goal is to catch true stalls, not slow‑but‑healthy startups). Timeout of 0/negative disables the watchdog (preserves current behavior).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org