You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@skywalking.apache.org by wu...@apache.org on 2026/06/10 02:02:11 UTC
(skywalking-helm) branch master updated: Run OAP init job in the main phase to fix `helm --wait` deadlock (#190)
This is an automated email from the ASF dual-hosted git repository.
wu-sheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/skywalking-helm.git
The following commit(s) were added to refs/heads/master by this push:
new 3aee62a Run OAP init job in the main phase to fix `helm --wait` deadlock (#190)
3aee62a is described below
commit 3aee62ac0b0a8c9626abbd386e9728c859cd21be
Author: 吴晟 Wu Sheng <wu...@foxmail.com>
AuthorDate: Wed Jun 10 10:02:06 2026 +0800
Run OAP init job in the main phase to fix `helm --wait` deadlock (#190)
The OAP init job was a `post-install,post-upgrade,post-rollback` hook. Under
`helm upgrade --install --wait`, Helm waits for all release resources to become
Ready before running post-* hooks, but the OAP Deployment runs in `-Dmode=no-init`
and never becomes Ready until the init job creates the storage schema. The hook
therefore never runs and the install deadlocks until it times out (hits new
users on a fresh install/storage).
Hooks cannot fix this with embedded storage subcharts: a pre-* hook init job
cannot reach main-phase storage, and a post-* hook deadlocks under `--wait`.
So the init job now runs as a normal main-phase resource alongside storage and
the OAP Deployment, which blocks in no-init mode until the schema appears.
To avoid `spec.template is immutable` failures on upgrade (a Job's pod template
cannot be patched), the Job name carries an 8-char hash of the chart values, so
a changed spec yields a new Job and Helm prunes the previous one. A new optional
`oapInit.ttlSecondsAfterFinished` can auto-clean finished Jobs (off by default;
left off for GitOps tools that would otherwise recreate the Job).
The OAP Deployment startupProbe default failureThreshold is raised 9 -> 30
(90s -> 300s) so the pod waits for the init job during a cold start instead of
being restarted.
Docs (values.yaml, chart README, root README) updated accordingly.
---
README.md | 27 ++++++++++++++------------
chart/skywalking/README.md | 3 ++-
chart/skywalking/templates/oap-deployment.yaml | 5 ++++-
chart/skywalking/templates/oap-init.job.yaml | 19 ++++++++++++++----
chart/skywalking/values.yaml | 14 ++++++++++---
5 files changed, 47 insertions(+), 21 deletions(-)
diff --git a/README.md b/README.md
index 011c576..6698a29 100644
--- a/README.md
+++ b/README.md
@@ -316,22 +316,25 @@ You can set those environment variables by `--set oap.env.<ENV_NAME>=<ENV_VALUE>
> The environment variables take priority over the overrode configuration files.
-## Rerun OAP init job
+## OAP init job
-Kubernetes Job cannot be rerun by default, if you want to rerun the OAP init
-job, you need to delete the Job and recreate it.
+The OAP storage schema (Elasticsearch indices / SQL tables / BanyanDB groups) is created by a
+one-shot `*-oap-init-*` Job that runs OAP in `-Dmode=init`. The main OAP Deployment runs in
+`-Dmode=no-init` and blocks (its `12800` port stays closed, so it is not Ready) until that schema
+exists. The init Job is a **normal release resource** that runs in the main install/upgrade phase,
+so `helm upgrade --install --wait` works: the Job creates the schema while OAP waits for it. To get
+Helm to surface init-Job failures directly (instead of only seeing OAP fail to become Ready), add
+`--wait-for-jobs` alongside `--wait`.
+
+The Job name carries a hash of the chart values, so any `helm upgrade` that changes a value
+re-creates the Job and re-runs init automatically (Helm prunes the previous one).
+
+To **force a rerun** without changing any value — delete the Job and re-run `helm upgrade`; Helm
+recreates the (now missing) Job and init runs again:
```shell
-# Make sure to export the Job manifest to a file before deleting it.
-kubectl get job -n "${SKYWALKING_RELEASE_NAMESPACE}" -l release=$SKYWALKING_RELEASE_NAME -o yaml > oap-init.job.yaml
-# Trim the Job manifest to keep only the Job part, you can either download yq from https://github.com/mikefarah/yq or
-# manually remove the fields that are not needed.
-yq 'del(.items[0].metadata.creationTimestamp,.items[0].metadata.resourceVersion,.items[0].metadata.uid,.items[0].status,.items[0].spec.template.metadata.labels."batch.kubernetes.io/controller-uid",.items[0].spec.template.metadata.labels."controller-uid",.items[0].spec.selector.matchLabels."batch.kubernetes.io/controller-uid")' oap-init.job.yaml > oap-init.job.trimmed.yaml
-# Check the file oap-init.job.trimmed.yaml to make sure it has correct content
-# Delete the Job
kubectl delete job -n "${SKYWALKING_RELEASE_NAMESPACE}" -l release=$SKYWALKING_RELEASE_NAME
-# Create the Job
-kubectl -n "${SKYWALKING_RELEASE_NAMESPACE}" apply -f oap-init.job.trimmed.yaml
+helm upgrade "$SKYWALKING_RELEASE_NAME" <chart> -n "${SKYWALKING_RELEASE_NAMESPACE}" --reuse-values
```
# Contact Us
diff --git a/chart/skywalking/README.md b/chart/skywalking/README.md
index 21c838b..89be301 100644
--- a/chart/skywalking/README.md
+++ b/chart/skywalking/README.md
@@ -68,7 +68,7 @@ The following table lists the configurable parameters of the Skywalking chart an
| `oap.nodeSelector` | OAP labels for master pod assignment | `{}` |
| `oap.tolerations` | OAP tolerations | `[]` |
| `oap.resources` | OAP node resources requests & limits | `{} - cpu limit must be an integer` |
-| `oap.startupProbe` | Configuration fields for the [startupProbe](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/) | `tcpSocket.port: 12800` <br> `failureThreshold: 9` <br> `periodSeconds: 10`
+| `oap.startupProbe` | Configuration fields for the [startupProbe](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/). The default budget (`failureThreshold` * `periodSeconds` = 300s) is large enough for OAP to wait in no-init mode while the OAP init Job creates the storage schema. | `tcpSocket.port: 12800` <br> `failureThreshold: 30` <br> `periodSeconds: 10`
| `oap.livenessProbe` | Configuration fields for the [livenessProbe](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/) | `tcpSocket.port: 12800` <br> `initialDelaySeconds: 5` <br> `periodSeconds: 10`
| `oap.readinessProbe` | Configuration fields for the [readinessProbe](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/) | `tcpSocket.port: 12800` <br> `initialDelaySeconds: 5` <br> `periodSeconds: 10`
| `oap.env` | OAP environment variables | `[]` |
@@ -109,6 +109,7 @@ The following table lists the configurable parameters of the Skywalking chart an
| `oapInit.nodeSelector` | OAP init job labels for master pod assignment | `{}` |
| `oapInit.tolerations` | OAP init job tolerations | `[]` |
| `oapInit.extraPodLabels` | OAP init job metadata labels | `[]` |
+| `oapInit.ttlSecondsAfterFinished` | Seconds after which the finished OAP init Job (and its Pod) is auto-deleted by the Kubernetes TTL-after-finished controller. Empty keeps the Job. Leave empty with GitOps tools (Argo CD/Flux), which would recreate it after deletion. | `""` |
| `satellite.name` | Satellite deployment name | `satellite` |
| `satellite.replicas` | Satellite k8s deployment replicas | `1` |
| `satellite.enabled` | Is enable Satellite | `false` |
diff --git a/chart/skywalking/templates/oap-deployment.yaml b/chart/skywalking/templates/oap-deployment.yaml
index d88c6ef..8ab826d 100644
--- a/chart/skywalking/templates/oap-deployment.yaml
+++ b/chart/skywalking/templates/oap-deployment.yaml
@@ -105,7 +105,10 @@ spec:
{{ else }}
tcpSocket:
port: 12800
- failureThreshold: 9
+ # In no-init mode OAP blocks (port 12800 stays closed) until the init Job has created
+ # the storage schema. Give it a generous budget (30 * 10s = 300s) so the pod waits for
+ # the init Job instead of being restarted during a cold start.
+ failureThreshold: 30
periodSeconds: 10
{{- end }}
readinessProbe:
diff --git a/chart/skywalking/templates/oap-init.job.yaml b/chart/skywalking/templates/oap-init.job.yaml
index 2bf1874..8b4f05d 100644
--- a/chart/skywalking/templates/oap-init.job.yaml
+++ b/chart/skywalking/templates/oap-init.job.yaml
@@ -18,17 +18,28 @@
apiVersion: batch/v1
kind: Job
metadata:
- name: "{{ template "skywalking.oap.fullname" . }}-init"
+ # NOTE: This Job is intentionally a normal release resource, NOT a Helm hook.
+ # Running it as a post-install/post-upgrade hook deadlocks `helm upgrade --install --wait`:
+ # Helm waits for every release resource to become Ready before it runs post-* hooks, but the
+ # OAP Deployment runs in `-Dmode=no-init` and never becomes Ready until this Job has created
+ # the storage schema -- so the hook (and therefore the schema) would never run. As a main-phase
+ # resource the Job runs alongside the OAP Deployment, which blocks in no-init mode until the
+ # schema appears, so `--wait` resolves instead of deadlocking.
+ #
+ # The name carries a hash of the chart values: a Job's `spec.template` is immutable, so a stable
+ # name would make `helm upgrade` fail with "field is immutable" whenever the pod template changes.
+ # Hashing yields a fresh Job whenever a relevant value changes; Helm prunes the previous one.
+ name: "{{ printf "%s-init-%s" (include "skywalking.oap.fullname" . | trunc 40 | trimSuffix "-") (.Values | toYaml | sha256sum | trunc 8) }}"
labels:
app: {{ template "skywalking.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version }}
component: "{{ template "skywalking.fullname" . }}-job"
heritage: {{ .Release.Service }}
release: {{ .Release.Name }}
- annotations:
- "helm.sh/hook": post-install,post-upgrade,post-rollback
- "helm.sh/hook-weight": "1"
spec:
+ {{- if .Values.oapInit.ttlSecondsAfterFinished }}
+ ttlSecondsAfterFinished: {{ .Values.oapInit.ttlSecondsAfterFinished }}
+ {{- end }}
template:
metadata:
name: "{{ .Release.Name }}-oap-init"
diff --git a/chart/skywalking/values.yaml b/chart/skywalking/values.yaml
index e83947d..5681cc2 100644
--- a/chart/skywalking/values.yaml
+++ b/chart/skywalking/values.yaml
@@ -75,11 +75,13 @@ oap:
# initialDelaySeconds: 5
# periodSeconds: 20
startupProbe: {}
- # Time to boot the application is set to:
- # 9 (failureThreshold) * 10 (periodSeconds) = 90 seconds in this case.
+ # Boot budget defaults to 30 (failureThreshold) * 10 (periodSeconds) = 300 seconds.
+ # In no-init mode OAP keeps port 12800 closed until the OAP init Job has created the storage
+ # schema, so the budget must be large enough to cover storage startup + schema creation;
+ # otherwise the pod is restarted while it is legitimately waiting for the init Job.
# tcpSocket:
# port: 12800
- # failureThreshold: 9
+ # failureThreshold: 30
# periodSeconds: 10
readinessProbe: {}
# tcpSocket:
@@ -301,6 +303,12 @@ oapInit:
tolerations: []
extraPodLabels: {}
# sidecar.istio.io/inject: false
+ # Auto-delete the completed init Job (and its Pod) this many seconds after it finishes, via the
+ # Kubernetes TTL-after-finished controller. Leave empty to keep the completed Job around.
+ # NOTE: leave this empty when using GitOps tools (e.g. Argo CD, Flux) -- they would recreate the
+ # Job after the TTL controller deletes it, re-running init on every reconcile. The Job name is
+ # value-hashed, so upgrades already work without TTL; this is only for tidying finished Jobs.
+ ttlSecondsAfterFinished: ""
# Elasticsearch managed by ECK (eck-elasticsearch chart)
# When enabled, the ECK operator is also installed as a dependency.