[jira] [Created] (HDDS-17107) Show startup progress in Datanode web UI

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created HDDS-17107:
---------------------------------

             Summary: Show startup progress in Datanode web UI
                 Key: HDDS-17107
                 URL: https://issues.apache.org/jira/browse/HDDS-17107
             Project: Apache Ozone
          Issue Type: Improvement
          Components: Ozone Datanode
            Reporter: Siyao Meng
            Assignee: Siyao Meng


h3. Problem
Unlike the HDFS NameNode ("Startup Progress" tab), Ozone services expose no startup progress. A datanode with many containers can take a long time to start, and until it does, operators can only follow the logs. They cannot tell which phase the datanode is in, how long each phase has taken, or how long startup will still take.

h3. Proposal
Track datanode startup in phases and show it in the Datanode web UI:
# Initializing
# Waiting for SCM (until the version response is received)
# Preparing volumes (formatting volumes and loading volume DBs)
# Loading containers (one step per volume, with done / total containers)
# Checking volumes
# Starting container services (including Ratis pipeline recovery)

For each phase, show its status (pending / running / complete / failed) and elapsed time. Container loading counts the container directories of each volume up front, so done / total and an estimated remaining time can be shown per volume. If startup fails, the running phase and any failed volume are marked as failed.

The progress is exposed through a {{StartupProgress}} MXBean ({{Hadoop:service=HddsDatanode,name=StartupProgress}}), so it is also available via {{/jmx}}. A new "Startup Progress" page in the Datanode web UI renders it.

The datanode HTTP server already starts before the slow phases, so their progress is visible while the datanode is starting.

h3. Follow-ups (out of scope)
The tracker lives in hdds-server-framework so that it can be reused for OM and SCM. Those services start their HTTP server near the end of startup, so they would first need the web UI to be available earlier.

h3. Related
* HDDS-12391, HDDS-13331, HDDS-13332: reduce datanode startup time (container set loading). This page makes the effect of such changes visible.
* HDDS-13896, HDDS-13897: datanode startup can hang on a slow or stuck volume without logging. Per-volume loading progress shows which volume is stuck.
* HDDS-16425, HDDS-16504: fail datanode startup cleanly, and add a watchdog for container initialization.
* HDDS-17078: OM startup can take hours with many snapshots (motivation for the OM follow-up).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org