You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@hertzbeat.apache.org by "orangeCatDeveloper (via GitHub)" <gi...@apache.org> on 2026/08/09 10:51:45 UTC
[PR] [fix] add startup probes so manager survives slow boot [hertzbeat-helm-chart]
orangeCatDeveloper opened a new pull request, #23:
URL: https://github.com/apache/hertzbeat-helm-chart/pull/23
## What's changed?
Fixes apache/hertzbeat#3849: `helm install` leaves the manager pod in permanent CrashLoopBackOff on many clusters.
Two compounding causes, reproduced on a kind cluster with chart 1.7.3:
1. The manager starts before postgres is ready, fails with `Connection to hertzbeat-database:5432 refused` and exits. This part self-heals via restarts once the database is up.
2. The bare `livenessProbe` (tcpSocket 1157, no `initialDelaySeconds`, no `startupProbe`) kills the container ~30s after start with k8s defaults, while the manager needs well over 30s to boot on typical nodes. Every retry gets killed mid-boot, so the pod never recovers — `kubectl describe` shows `Liveness probe failed: dial tcp ...:1157: connect: connection refused`.
The crash is invisible in `kubectl logs` because logback routes everything to file appenders inside the container, which is why the issue reports "no useful log".
Fix: add a `startupProbe` (10s × 30 = 5 min boot budget) to the manager and collector deployments; liveness only takes over after startup succeeds. Verified on kind: before — manager restarts forever; after — one restart while postgres boots, then all pods reach `1/1 Running`.
## Checklist
- [x] I have read the [Contributing Guide](https://hertzbeat.apache.org/docs/community/code_style_and_quality_guide)
- [x] I have written the necessary doc or comment.
- [ ] I have added the necessary unit tests and all cases have passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@hertzbeat.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@hertzbeat.apache.org
For additional commands, e-mail: notifications-help@hertzbeat.apache.org
Re: [PR] [fix] add startup probes so manager survives slow boot [hertzbeat-helm-chart]
Posted by "orangeCatDeveloper (via GitHub)" <gi...@apache.org>.
orangeCatDeveloper commented on PR #23:
URL: https://github.com/apache/hertzbeat-helm-chart/pull/23#issuecomment-5231438891
Note on CI: every workflow run in this repo currently dies with `startup_failure` — the `ubuntu-22.04` hosted runner was retired by GitHub, so no workflow can start (all runs since late 2025 are affected, including pushes to `main`). #24 fixes the pipeline (supported runner + `validate-maintainers: false`, since the "hertzbeat" maintainer entry is not a GitHub user). Once #24 is merged, re-running checks here should go green — this branch already passes `ct lint` locally against the fixed config (chart version bumped to 1.8.1 as ct requires).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@hertzbeat.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@hertzbeat.apache.org
For additional commands, e-mail: notifications-help@hertzbeat.apache.org
Re: [PR] [fix] add startup probes so manager survives slow boot [hertzbeat-helm-chart]
Posted by "Aias00 (via GitHub)" <gi...@apache.org>.
Aias00 merged PR #23:
URL: https://github.com/apache/hertzbeat-helm-chart/pull/23
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@hertzbeat.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@hertzbeat.apache.org
For additional commands, e-mail: notifications-help@hertzbeat.apache.org