[jira] [Created] (HDDS-17114) Show startup progress and readiness for OM, SCM and Datanode

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created HDDS-17114:
---------------------------------

             Summary: Show startup progress and readiness for OM, SCM and Datanode
                 Key: HDDS-17114
                 URL: https://issues.apache.org/jira/browse/HDDS-17114
             Project: Apache Ozone
          Issue Type: New Feature
          Components: SCM, Ozone Manager, Ozone Datanode
            Reporter: Siyao Meng


h3. Problem
OM, SCM and datanodes can take a long time to start on large clusters: opening RocksDB, loading containers and pipelines, the OM snapshot chain and prefixes, replaying the Raft log, and waiting for SCM safe mode exit. During that time operators only have the logs. The services expose no startup phase, progress or estimated remaining time, unlike the HDFS NameNode's "Startup Progress" page.

OM and SCM also start their HTTP server near the end of startup, so their web UI, /jmx and /prom are unavailable for most of it. There is no endpoint that says whether a service is ready, so scripts and Kubernetes probes use an open port as a stand-in:
* The Kubernetes liveness probes use TCP on the OM and SCM RPC ports (9862, 9861) with a 30s initial delay. OM binds 9862 only after loading its DB, and an HA SCM follower binds 9861 only after catching up with the leader, so a slow start can be killed by the probe.
* The rolling-upgrade test waits on the SCM HTTP port (9876) before restarting the next SCM.

h3. Proposal
# Track startup in phases with a reusable StartupProgress MXBean ({{Hadoop:service=<service>,name=StartupProgress}}). Report status, elapsed time, done / total and the estimated remaining time per phase and per step.
# Show it on a "Startup Progress" page in each service's web UI.
# Start the OM and SCM HTTP servers right after Kerberos login. Until the service is ready, servlets that need service state (/dbCheckpoint, /v2/dbCheckpoint, /serviceList, /snapshotList) return 503.
# Startup is complete once the Raft log has been replayed (OM and SCM) and SCM has left safe mode.
# Add /health/live and /health/ready to OM, SCM and the datanode, following the S3 Gateway endpoints from HDDS-16360, and use them in the Kubernetes examples and the rolling-upgrade test.

h3. Related
* HDDS-16360: S3 Gateway health check endpoints (pattern to reuse)
* HDDS-17078: OM startup can take hours with many snapshots
* HDDS-16299, HDDS-16301: guard leadership transfer to an OM or SCM that is not ready



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org