You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Siyao Meng (Jira)" <ji...@apache.org> on 2026/10/19 01:52:00 UTC
[jira] [Updated] (HDDS-17119) Use health endpoints for OM, SCM and Datanode probes in Kubernetes examples
[ https://issues.apache.org/jira/browse/HDDS-17119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siyao Meng updated HDDS-17119:
------------------------------
Description:
Replace the TCP liveness probes on the OM and SCM RPC ports with HTTP probes on /health/live, so that a slow start (DB load, Raft log replay) is not killed. If readiness probes are added, the headless services need publishNotReadyAddresses: true. Otherwise pods that are not ready drop out of DNS, datanodes cannot reach SCM, and SCM can never leave safe mode.
h3. Implementation notes
Changes are in hadoop-ozone/dist/src/main/k8s/definitions/ozone; the examples are regenerated with flekszible 2.3.0. The unrelated config-configmap.yaml drift that regeneration produces on master is left out.
* OM and SCM liveness probes: httpGet /health/live on 9874 / 9876 instead of tcpSocket 9862 / 9861. In SCM HA, 9861 only opens once the SCM has caught up with the leader, so the old probe could kill an SCM during replay.
* Datanode: new liveness probe, httpGet /health/live on 9882.
* OM and Recon WAITFOR: scm-0.scm:9860 instead of scm-0.scm:9876. With the early HTTP server (HDDS-17116), 9876 opens long before SCM can serve RPC. 9860 used to open just before 9876 in SCM start(), so the wait stays the same.
* k8s tests (examples/testlib.sh): wait for OM /health/ready instead of the "HTTP server of ozoneManager listening" log line, which is now logged early in the OM constructor (HDDS-17117).
* No readiness probes for OM, SCM and datanodes. Besides the DNS problem above, the statefulsets use OrderedReady pod management, so in ozone-ha scm-1 would not be created until scm-0 has left safe mode, which needs datanodes. A readiness probe would also remove a starting pod from the *-public services and hide the startup progress page. Adding them would need publishNotReadyAddresses: true on the headless services (and podManagementPolicy: Parallel for SCM HA); S3 Gateway keeps its existing readiness probe.
h3. Known limitations
* WAITFOR and om --init run in the main container, so the liveness probe already runs while OM waits for SCM. If SCM takes longer than about 50s (initialDelaySeconds 30 + 3 failed probes) to open 9860, OM is restarted. This was the same before. The datanode HTTP server starts after the volume DBs are loaded, so a very slow datanode start could be killed the same way. A startupProbe on /health/live would cover both.
* An HTTP server start failure, which these services treat as non-fatal, or ozone.http.policy=HTTPS_ONLY now fails the liveness probe.
h3. Dependencies
*Depends on HDDS-17118 (and so on HDDS-17117, HDDS-17116, HDDS-17115 and HDDS-17107).* HDDS-17119.001.diff applies on top of HDDS-17118.001.diff, HDDS-17117.001.diff, HDDS-17116.001.diff, HDDS-17115.001.diff and the HDDS-17107 change (https://github.com/apache/ozone/pull/11611), in that order.
Testing: regenerated examples match flekszible output for all example dirs; bash -n on testlib.sh. Not run on a Kubernetes cluster.
was:Replace the TCP liveness probes on the OM and SCM RPC ports with HTTP probes on /health/live, so that a slow start (DB load, Raft log replay) is not killed. If readiness probes are added, the headless services need publishNotReadyAddresses: true. Otherwise pods that are not ready drop out of DNS, datanodes cannot reach SCM, and SCM can never leave safe mode.
> Use health endpoints for OM, SCM and Datanode probes in Kubernetes examples
> ---------------------------------------------------------------------------
>
> Key: HDDS-17119
> URL: https://issues.apache.org/jira/browse/HDDS-17119
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: kubernetes
> Reporter: Siyao Meng
> Priority: Major
> Attachments: HDDS-17119.001.diff
>
>
> Replace the TCP liveness probes on the OM and SCM RPC ports with HTTP probes on /health/live, so that a slow start (DB load, Raft log replay) is not killed. If readiness probes are added, the headless services need publishNotReadyAddresses: true. Otherwise pods that are not ready drop out of DNS, datanodes cannot reach SCM, and SCM can never leave safe mode.
> h3. Implementation notes
> Changes are in hadoop-ozone/dist/src/main/k8s/definitions/ozone; the examples are regenerated with flekszible 2.3.0. The unrelated config-configmap.yaml drift that regeneration produces on master is left out.
> * OM and SCM liveness probes: httpGet /health/live on 9874 / 9876 instead of tcpSocket 9862 / 9861. In SCM HA, 9861 only opens once the SCM has caught up with the leader, so the old probe could kill an SCM during replay.
> * Datanode: new liveness probe, httpGet /health/live on 9882.
> * OM and Recon WAITFOR: scm-0.scm:9860 instead of scm-0.scm:9876. With the early HTTP server (HDDS-17116), 9876 opens long before SCM can serve RPC. 9860 used to open just before 9876 in SCM start(), so the wait stays the same.
> * k8s tests (examples/testlib.sh): wait for OM /health/ready instead of the "HTTP server of ozoneManager listening" log line, which is now logged early in the OM constructor (HDDS-17117).
> * No readiness probes for OM, SCM and datanodes. Besides the DNS problem above, the statefulsets use OrderedReady pod management, so in ozone-ha scm-1 would not be created until scm-0 has left safe mode, which needs datanodes. A readiness probe would also remove a starting pod from the *-public services and hide the startup progress page. Adding them would need publishNotReadyAddresses: true on the headless services (and podManagementPolicy: Parallel for SCM HA); S3 Gateway keeps its existing readiness probe.
> h3. Known limitations
> * WAITFOR and om --init run in the main container, so the liveness probe already runs while OM waits for SCM. If SCM takes longer than about 50s (initialDelaySeconds 30 + 3 failed probes) to open 9860, OM is restarted. This was the same before. The datanode HTTP server starts after the volume DBs are loaded, so a very slow datanode start could be killed the same way. A startupProbe on /health/live would cover both.
> * An HTTP server start failure, which these services treat as non-fatal, or ozone.http.policy=HTTPS_ONLY now fails the liveness probe.
> h3. Dependencies
> *Depends on HDDS-17118 (and so on HDDS-17117, HDDS-17116, HDDS-17115 and HDDS-17107).* HDDS-17119.001.diff applies on top of HDDS-17118.001.diff, HDDS-17117.001.diff, HDDS-17116.001.diff, HDDS-17115.001.diff and the HDDS-17107 change (https://github.com/apache/ozone/pull/11611), in that order.
> Testing: regenerated examples match flekszible output for all example dirs; bash -n on testlib.sh. Not run on a Kubernetes cluster.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org