You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "shuan1026 (via GitHub)" <gi...@apache.org> on 2026/07/21 14:34:53 UTC
[PR] HDDS-14351. Add a preStop hook or graceful termination period for OM … [ozone-helm-charts]
shuan1026 opened a new pull request, #48:
URL: https://github.com/apache/ozone-helm-charts/pull/48
## What changes were proposed in this pull request?
Follow-up to [PR #20 discussion](https://github.com/apache/ozone-helm-charts/pull/20#discussion_r2632054499): add a preStop hook / longer graceful termination so OM/SCM do not stop while still holding Ratis leadership.
* Add `lifecycle.preStop` on OM/SCM StatefulSets (`replicas > 1`): if this pod is leader, run `ozone admin om|scm transfer -r` before shutdown.
* Add `terminationGracePeriodSeconds` for HA OM/SCM (default path used **150s**) so a slow transfer + short sleep still finishes before SIGKILL.
* The preStop script only does the leader transfer; it does **not** send `kill -TERM 1` as in the [PR #20 discussion](https://github.com/apache/ozone-helm-charts/pull/20#discussion_r2632054499) sample. kubelet sends SIGTERM to the container automatically once preStop returns, and the image's PID 1 (`dumb-init`) forwards it to the OM/SCM process, so an explicit `kill` would just duplicate that and adds a dependency on the entrypoint's PID 1.
* Complements the existing pre-upgrade `om-leader-transfer-job` (scale-down + persistence); this covers ordinary pod delete / roll / drain.
* Set `publishNotReadyAddresses: true` on the OM/SCM headless Services. During termination a terminating endpoint always has `ready: false`, so by default it is dropped from the headless Service DNS ([Kubernetes docs](https://kubernetes.io/docs/tutorials/services/pods-and-endpoint-termination-flow/)). The transferee then cannot resolve the outgoing leader's address to confirm the handoff, and `transfer -r` blocks until `TIMED_OUT(60s)`. Publishing not-ready addresses keeps the terminating pod resolvable so the transfer completes quickly (see [discussion](https://github.com/apache/ozone-helm-charts/pull/20#discussion_r3619734581)).
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-14351
## How was this patch tested?
- Green CI run, and tested manually with leader transfer and basic ozone commands.
* kind, OM HA (`om.replicas=3`): over repeated leader pod-delete cycles, preStop transfers leadership and quorum consistently recovers to a single LEADER.
* kind, SCM HA (`scm.replicas=3`): same repeated leader pod-delete cycles, with the same result.
* Compared with `main` (both OM and SCM): no preStop / default 30s grace; deleting the leader does **not** attempt transfer (failover is only Raft election after SIGTERM).
Manually Test:
```bash
kind create cluster --name ozone-helm-prestop
# OM HA (default)
helm upgrade --install ozone charts/ozone --wait --timeout 15m
kubectl exec ozone-om-0 -c om -- ozone admin om roles -id cluster1
kubectl delete pod ozone-om-1 # delete current LEADER
kubectl get events --sort-by='.lastTimestamp' | grep -iE 'preStop|Killing'
kubectl exec ozone-om-0 -c om -- ozone admin om roles -id cluster1
# expect: single LEADER; transfer done in preStop (no TIMED_OUT)
# SCM HA
helm upgrade --install ozone charts/ozone --set scm.replicas=3 --wait --timeout 20m
kubectl exec ozone-scm-0 -c scm -- ozone admin scm roles -id cluster1
kubectl delete pod ozone-scm-1 # delete current LEADER
kubectl get events --sort-by='.lastTimestamp' | grep -iE 'preStop|Killing'
kubectl exec ozone-scm-0 -c scm -- ozone admin scm roles -id cluster1
# expect: single LEADER; transfer done in preStop
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14351. Add preStop and graceful termination for OM and SCM [ozone-helm-charts]
Posted by "Tejaskriya (via GitHub)" <gi...@apache.org>.
Tejaskriya merged PR #48:
URL: https://github.com/apache/ozone-helm-charts/pull/48
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14351. Add preStop and graceful termination for OM and SCM [ozone-helm-charts]
Posted by "shuan1026 (via GitHub)" <gi...@apache.org>.
shuan1026 commented on PR #48:
URL: https://github.com/apache/ozone-helm-charts/pull/48#issuecomment-5061344623
PTAL @adoroszlai @chungen0126 Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14351. Add preStop and graceful termination for OM and SCM [ozone-helm-charts]
Posted by "Tejaskriya (via GitHub)" <gi...@apache.org>.
Tejaskriya commented on PR #48:
URL: https://github.com/apache/ozone-helm-charts/pull/48#issuecomment-5165141341
Thanks for the patch @shuan1026 , and for the review @chungen0126
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org