You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "shuan1026 (via GitHub)" <gi...@apache.org> on 2026/07/16 09:22:13 UTC
[PR] HDDS-15874. Adding readinessProbe for helm HA deployments [ozone-helm-charts]
shuan1026 opened a new pull request, #43:
URL: https://github.com/apache/ozone-helm-charts/pull/43
## What changes were proposed in this pull request?
This change adds a `readinessProbe` for HA OM/SCM deployments so pods are removed from Service endpoints until they have a stable Ratis role (`LEADER` or `FOLLOWER`).
The chart already has (or will have via [[HDDS-14349](https://issues.apache.org/jira/browse/HDDS-14349)]) an HTTP `livenessProbe` and a roles-based `startupProbe`. **Startup** answers “still initializing do not treat liveness failures as fatal.” **Readiness** answers “should this pod receive client traffic via Service endpoints now?”
A roles exec `readinessProbe` matches the [[reviewer sample](https://github.com/apache/ozone-helm-charts/pull/20#discussion_r2631983830)] and is stricter than UI `httpGet`: a process can respond on the UI port before OM/SCM has joined the HA ring in a follower/leader role.
### In scope
* OM HA (`replicas > 1`): `readinessProbe` via `ozone admin om roles`, requiring `LEADER|FOLLOWER` on the line that matches `$(hostname)`
* SCM HA (`replicas > 1`): `readinessProbe` via `ozone admin scm roles`, with `$(hostname -f)` to match SCM roles output format
* Probe timings from kind testing: `timeoutSeconds: 20` (sample `5` is too low while peers/DNS/Ratis are still forming)
* **SCM HA headless DNS (if not already present):** `publishNotReadyAddresses: true` on `scm-service-headless` when `scm.replicas > 1` — same peer-DNS requirement as OM during bootstrap/readiness
### Why add readiness (Things to confirm)
[[HDDS-1668](https://issues.apache.org/jira/browse/HDDS-1668)] warned against readiness on the OM HA example: Ready-gated endpoints can fight peer DNS needed during startup (`OzoneManager.loadOMHAConfigs`). That concern is may still valid if readiness is added **without** companions.
This PR follows the reviewer sample and addresses former by:
1. Relying on **14349** for startup + OM `Parallel` + OM headless `publishNotReadyAddresses` so peers are discoverable while pods are not Ready
2. Using readiness only to gate **Service endpoints** (traffic), not to block headless peer DNS once `publishNotReadyAddresses` is set
**Trade-off:** `ozone admin * roles` as a **recurring** readiness check has ongoing cost (fork JVM/CLI + RPC). Acceptable for HA correctness in this chart; a lighter `httpGet` readiness remains a possible follow-up if maintainers prefer.
### Probe shape (illustrative; OM)
```yaml
readinessProbe:
exec:
command:
- sh
- -c
- |
ozone admin om roles -id {{ .Values.clusterId }} 2>/dev/null \
| grep $(hostname) | grep -qE 'LEADER|FOLLOWER' && exit 0 || exit 1
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 20
failureThreshold: 3
```
SCM HA uses `ozone admin scm roles` with `$(hostname -f)` and the same `LEADER|FOLLOWER` grep.
Non-HA (`replicas == 1`) does not add roles exec readiness.
Startup ([[HDDS-14349](https://issues.apache.org/jira/browse/HDDS-14349)]) only checks membership in the roles list; readiness additionally requires `LEADER|FOLLOWER`.
## What is the link to the Apache JIRA
* This issue (readiness): https://issues.apache.org/jira/browse/HDDS-15874
* Startup (prerequisite): https://issues.apache.org/jira/browse/HDDS-14349
* Parent: https://issues.apache.org/jira/browse/HDDS-14382
Review discussion: https://github.com/apache/ozone-helm-charts/pull/20#discussion_r2631983830
## How was this patch tested?
### 1. kind — default install (OM HA)
On a fresh kind cluster:
```bash
helm upgrade --install ozone charts/ozone --wait --timeout 15m
```
Results:
* Release deployed; all pods Ready with no restarts
* During cold start, OM may show 1–2 Startup `timed out after 20s` events while peers/roles are not ready; within `failureThreshold` the probe succeeds and Ready is reached **without** repeated liveness Killing
* Green CI run and manually verified leader transfer and basic ozone commands
### 2. kind — SCM HA (`scm.replicas=3`, fresh install)
On a **new** release/cluster (do not scale an existing `scm.replicas=1` in place):
```bash
helm upgrade --install ozone charts/ozone \
--set scm.replicas=3 \
--wait --timeout 20m
```
Same pass criteria: Ready, no restart storm.
**Out of scope for this issue/PR:** `kubectl scale` or `helm upgrade --set scm.replicas=3` migrating an existing **scm 1→3** deployment (known STS revision / `addSCM` issues; track separately).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15874. Adding readinessProbe for helm HA deployments [ozone-helm-charts]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #43:
URL: https://github.com/apache/ozone-helm-charts/pull/43#issuecomment-5661101898
@Tejaskriya would you like to take another look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15874. Adding readinessProbe for helm HA deployments [ozone-helm-charts]
Posted by "Tejaskriya (via GitHub)" <gi...@apache.org>.
Tejaskriya merged PR #43:
URL: https://github.com/apache/ozone-helm-charts/pull/43
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15874. Adding readinessProbe for helm HA deployments [ozone-helm-charts]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #43:
URL: https://github.com/apache/ozone-helm-charts/pull/43#issuecomment-5011505510
@dnskr please review if you have time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15874. Adding readinessProbe for helm HA deployments [ozone-helm-charts]
Posted by "shuan1026 (via GitHub)" <gi...@apache.org>.
shuan1026 commented on PR #43:
URL: https://github.com/apache/ozone-helm-charts/pull/43#issuecomment-5169521846
> Thanks for working on this @shuan1026 , similar to your other PR, I believe there will be merge-conflicts resolution needed here too. Also please consider making the constants parameterized using values.yaml
Done, thanks @Tejaskriya!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15874. Adding readinessProbe for helm HA deployments [ozone-helm-charts]
Posted by "Tejaskriya (via GitHub)" <gi...@apache.org>.
Tejaskriya commented on PR #43:
URL: https://github.com/apache/ozone-helm-charts/pull/43#issuecomment-5677432327
Thanks for the patch @shuan1026 , and reviews @chungen0126
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org