You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/09/15 15:33:42 UTC
(spark-kubernetes-operator) branch main updated: [SPARK-59504] Use `enabled` key convention in Helm `values.yaml` while honoring legacy `enable`
This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new df96061 [SPARK-59504] Use `enabled` key convention in Helm `values.yaml` while honoring legacy `enable`
df96061 is described below
commit df96061b17552cd25049b67b903c077982a826e0
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Tue Sep 15 08:33:33 2026 -0700
[SPARK-59504] Use `enabled` key convention in Helm `values.yaml` while honoring legacy `enable`
### What changes were proposed in this pull request?
This PR aims to use the `enabled` key convention in the Helm chart `values.yaml` while still honoring the legacy `enable` keys.
- `operatorDeployment.networkPolicy.enable` → `operatorDeployment.networkPolicy.enabled`
- `operatorConfiguration.dynamicConfig.enable` → `operatorConfiguration.dynamicConfig.enabled`
Two helpers in `_helpers.tpl` resolve each toggle to `true` when either the new or the legacy key is `true`, and all templates use them. The legacy keys stay valid in `values.schema.json`, marked as deprecated, and will be removed in chart `2.0.0`.
### Why are the changes needed?
`<feature>.enabled` is the common Helm convention, and the operator's runtime property is already `spark.kubernetes.operator.dynamicConfig.enabled`. Resolving with `or` keeps existing `enable: true` users working because the chart default `enabled: false` is always merged and cannot be distinguished from a user-supplied value.
### Does this PR introduce _any_ user-facing change?
Yes, compared to [v1.0.0 (2026-07-23)](https://github.com/apache/spark-kubernetes-operator/releases/tag/1.0.0). New users should use `enabled`. Existing values files with `enable` keep working without modification.
### How was this patch tested?
- `helm lint --strict` passes.
- Rendered with `helm template` for default, `enabled=true`, legacy `enable=true`, and legacy `enable=false` and confirmed the expected resources and properties.
- Added a CI step in the `lint` job that renders the chart with the legacy `enable` keys and asserts they are still honored.
- Existing Helm and E2E tests cover the new `enabled` keys.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5.1
Closes #825 from dongjoon-hyun/SPARK-59504.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.github/workflows/build_and_test.yml | 24 ++++++++++++++++++++++
.../templates/_helpers.tpl | 22 +++++++++++++++++++-
.../templates/network-policy.yaml | 2 +-
.../templates/operator-rbac.yaml | 2 +-
.../templates/spark-operator.yaml | 4 ++--
.../templates/tests/test-network-policy.yaml | 2 +-
.../spark-kubernetes-operator/values.schema.json | 16 +++++++++++----
.../helm/spark-kubernetes-operator/values.yaml | 10 ++++++---
docs/operations.md | 11 ++++++++--
tests/e2e/helm/dynamic-config-values-2.yaml | 2 +-
tests/e2e/helm/dynamic-config-values-file.yaml | 2 +-
tests/e2e/helm/dynamic-config-values.yaml | 2 +-
.../helm-test-values/network-policy/values.yaml | 2 +-
13 files changed, 82 insertions(+), 19 deletions(-)
diff --git a/.github/workflows/build_and_test.yml b/.github/workflows/build_and_test.yml
index 815c144..96debfc 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -304,3 +304,27 @@ jobs:
- name: Validate helm chart linting
run: |
helm lint --strict build-tools/helm/spark-kubernetes-operator
+ - name: Validate deprecated helm values are still honored
+ run: |
+ helm template spark build-tools/helm/spark-kubernetes-operator \
+ --set operatorDeployment.networkPolicy.enable=true \
+ | grep -q 'kind: NetworkPolicy'
+ helm template spark build-tools/helm/spark-kubernetes-operator \
+ --set operatorConfiguration.dynamicConfig.enable=true \
+ | grep -q 'spark.kubernetes.operator.dynamicConfig.enabled=true'
+ - name: Validate helm values resolve to disabled
+ run: |
+ if helm template spark build-tools/helm/spark-kubernetes-operator \
+ | grep -q 'kind: NetworkPolicy'; then
+ echo "NetworkPolicy rendered with the default values"; exit 1
+ fi
+ if helm template spark build-tools/helm/spark-kubernetes-operator \
+ --set operatorDeployment.networkPolicy.enable=false \
+ | grep -q 'kind: NetworkPolicy'; then
+ echo "NetworkPolicy rendered with enable=false"; exit 1
+ fi
+ if helm template spark build-tools/helm/spark-kubernetes-operator \
+ --set operatorConfiguration.dynamicConfig.enable=false \
+ | grep -q 'spark.kubernetes.operator.dynamicConfig.enabled=true'; then
+ echo "dynamicConfig enabled with enable=false"; exit 1
+ fi
diff --git a/build-tools/helm/spark-kubernetes-operator/templates/_helpers.tpl b/build-tools/helm/spark-kubernetes-operator/templates/_helpers.tpl
index 94452ca..9bf397c 100644
--- a/build-tools/helm/spark-kubernetes-operator/templates/_helpers.tpl
+++ b/build-tools/helm/spark-kubernetes-operator/templates/_helpers.tpl
@@ -106,6 +106,26 @@ List of Spark workload namespaces. If not provied in values, use the same namesp
{{- end }}
{{- end }}
+{{/*
+Whether the operator pod NetworkPolicy is enabled. The legacy key
+{operatorDeployment.networkPolicy.enable} is deprecated but still honored: the feature is
+enabled when either key is true.
+*/}}
+{{- define "spark-operator.networkPolicy.enabled" -}}
+{{- $np := .Values.operatorDeployment.networkPolicy -}}
+{{- if or $np.enabled $np.enable }}true{{ else }}false{{ end -}}
+{{- end }}
+
+{{/*
+Whether dynamic config (hot properties loading) is enabled. The legacy key
+{operatorConfiguration.dynamicConfig.enable} is deprecated but still honored: the feature is
+enabled when either key is true.
+*/}}
+{{- define "spark-operator.dynamicConfig.enabled" -}}
+{{- $dc := .Values.operatorConfiguration.dynamicConfig -}}
+{{- if or $dc.enabled $dc.enable }}true{{ else }}false{{ end -}}
+{{- end }}
+
{{/*
Default property overrides
*/}}
@@ -113,7 +133,7 @@ Default property overrides
# Runtime resolved properties
spark.kubernetes.operator.namespace={{ .Release.Namespace }}
spark.kubernetes.operator.name={{- include "spark-operator.name" . }}
-spark.kubernetes.operator.dynamicConfig.enabled={{ .Values.operatorConfiguration.dynamicConfig.enable }}
+spark.kubernetes.operator.dynamicConfig.enabled={{ include "spark-operator.dynamicConfig.enabled" . }}
spark.kubernetes.operator.dynamicConfig.source={{ .Values.operatorConfiguration.dynamicConfig.source }}
spark.kubernetes.operator.metrics.port={{ include "spark-operator.metricsPort" . }}
spark.kubernetes.operator.health.probePort={{ include "spark-operator.probePort" . }}
diff --git a/build-tools/helm/spark-kubernetes-operator/templates/network-policy.yaml b/build-tools/helm/spark-kubernetes-operator/templates/network-policy.yaml
index cf557ca..74a79e9 100644
--- a/build-tools/helm/spark-kubernetes-operator/templates/network-policy.yaml
+++ b/build-tools/helm/spark-kubernetes-operator/templates/network-policy.yaml
@@ -13,7 +13,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-{{- if .Values.operatorDeployment.networkPolicy.enable }}
+{{- if eq (include "spark-operator.networkPolicy.enabled" .) "true" }}
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
diff --git a/build-tools/helm/spark-kubernetes-operator/templates/operator-rbac.yaml b/build-tools/helm/spark-kubernetes-operator/templates/operator-rbac.yaml
index ec43357..50d0511 100644
--- a/build-tools/helm/spark-kubernetes-operator/templates/operator-rbac.yaml
+++ b/build-tools/helm/spark-kubernetes-operator/templates/operator-rbac.yaml
@@ -173,7 +173,7 @@ metadata:
{{- template "spark-operator.operatorRbacRules" $ }}
---
{{- end }}
-{{- if and .Values.operatorConfiguration.dynamicConfig.enable (eq .Values.operatorConfiguration.dynamicConfig.source "configMap") }}
+{{- if and (eq (include "spark-operator.dynamicConfig.enabled" .) "true") (eq .Values.operatorConfiguration.dynamicConfig.source "configMap") }}
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
diff --git a/build-tools/helm/spark-kubernetes-operator/templates/spark-operator.yaml b/build-tools/helm/spark-kubernetes-operator/templates/spark-operator.yaml
index 046464f..a49a88a 100644
--- a/build-tools/helm/spark-kubernetes-operator/templates/spark-operator.yaml
+++ b/build-tools/helm/spark-kubernetes-operator/templates/spark-operator.yaml
@@ -162,7 +162,7 @@ spec:
mountPath: /opt/spark-operator/logs
- name: tmp-volume
mountPath: /tmp
- {{- if and .Values.operatorConfiguration.dynamicConfig.enable (eq .Values.operatorConfiguration.dynamicConfig.source "file") }}
+ {{- if and (eq (include "spark-operator.dynamicConfig.enabled" .) "true") (eq .Values.operatorConfiguration.dynamicConfig.source "file") }}
- name: spark-operator-dynamic-config-volume
mountPath: /opt/spark-operator/dynamic-conf
readOnly: true
@@ -191,7 +191,7 @@ spec:
emptyDir: { }
- name: tmp-volume
emptyDir: { }
- {{- if and .Values.operatorConfiguration.dynamicConfig.enable (eq .Values.operatorConfiguration.dynamicConfig.source "file") }}
+ {{- if and (eq (include "spark-operator.dynamicConfig.enabled" .) "true") (eq .Values.operatorConfiguration.dynamicConfig.source "file") }}
- name: spark-operator-dynamic-config-volume
configMap:
name: spark-kubernetes-operator-dynamic-configuration
diff --git a/build-tools/helm/spark-kubernetes-operator/templates/tests/test-network-policy.yaml b/build-tools/helm/spark-kubernetes-operator/templates/tests/test-network-policy.yaml
index 9ba24b8..cfbf784 100644
--- a/build-tools/helm/spark-kubernetes-operator/templates/tests/test-network-policy.yaml
+++ b/build-tools/helm/spark-kubernetes-operator/templates/tests/test-network-policy.yaml
@@ -13,7 +13,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-{{- if .Values.operatorDeployment.networkPolicy.enable }}
+{{- if eq (include "spark-operator.networkPolicy.enabled" .) "true" }}
apiVersion: v1
kind: Pod
metadata:
diff --git a/build-tools/helm/spark-kubernetes-operator/values.schema.json b/build-tools/helm/spark-kubernetes-operator/values.schema.json
index 9d85fb8..60fa41f 100644
--- a/build-tools/helm/spark-kubernetes-operator/values.schema.json
+++ b/build-tools/helm/spark-kubernetes-operator/values.schema.json
@@ -396,13 +396,17 @@
"type": "object",
"description": "NetworkPolicy for the operator pod",
"required": [
- "enable"
+ "enabled"
],
"properties": {
- "enable": {
+ "enabled": {
"type": "boolean",
"description": "Whether to create a NetworkPolicy for the operator pod"
},
+ "enable": {
+ "type": "boolean",
+ "description": "Deprecated, use 'enabled' instead; still honored (the NetworkPolicy is created when either key is true) and will be removed in chart 2.0.0 (SPARK-59533)"
+ },
"metricsIngress": {
"type": ["array", "null"],
"description": "List of NetworkPolicyPeer(s) allowed to reach the metrics port; when empty, ingress to the metrics port is denied",
@@ -818,16 +822,20 @@
"type": "object",
"description": "Dynamic configuration",
"required": [
- "enable",
+ "enabled",
"create",
"annotations",
"data"
],
"properties": {
- "enable": {
+ "enabled": {
"type": "boolean",
"description": "Enable dynamic configuration"
},
+ "enable": {
+ "type": "boolean",
+ "description": "Deprecated, use 'enabled' instead; still honored (dynamic configuration is enabled when either key is true) and will be removed in chart 2.0.0 (SPARK-59533)"
+ },
"create": {
"type": "boolean",
"description": "Create ConfigMap for dynamic configuration"
diff --git a/build-tools/helm/spark-kubernetes-operator/values.yaml b/build-tools/helm/spark-kubernetes-operator/values.yaml
index e33fb4d..e9fdb47 100644
--- a/build-tools/helm/spark-kubernetes-operator/values.yaml
+++ b/build-tools/helm/spark-kubernetes-operator/values.yaml
@@ -94,7 +94,9 @@ operatorDeployment:
# operator pod is denied. Note that this requires a CNI plugin with NetworkPolicy support,
# and that egress traffic is not restricted.
networkPolicy:
- enable: false
+ # The legacy key `enable` is deprecated and will be removed in chart 2.0.0 (SPARK-59533).
+ # It is still honored: the NetworkPolicy is created when either `enabled` or `enable` is true.
+ enabled: false
# List of NetworkPolicyPeer(s) allowed to reach the metrics port, e.g. the Prometheus
# scraper. When empty, ingress to the metrics port is denied.
metricsIngress: [ ]
@@ -218,8 +220,10 @@ operatorConfiguration:
metrics.properties: |+
# Metrics Properties Overrides
dynamicConfig:
- # Enable this for hot properties loading.
- enable: false
+ # Enable this for hot properties loading. The legacy key `enable` is deprecated and will be
+ # removed in chart 2.0.0 (SPARK-59533). It is still honored: hot properties loading is
+ # enabled when either `enabled` or `enable` is true.
+ enabled: false
# Source of the dynamic config overrides when enabled. Supported values:
# configMap - (default) watch a ConfigMap via a Kubernetes informer. Requires the operator
# to have RBAC to read ConfigMaps (created by this chart).
diff --git a/docs/operations.md b/docs/operations.md
index 4697d78..66dd7e1 100644
--- a/docs/operations.md
+++ b/docs/operations.md
@@ -133,7 +133,8 @@ following table:
| operatorConfiguration.spark-operator.properties | The default operator configuration. | |
| operatorConfiguration.metrics.properties | The default operator metrics (sink) configuration. | |
| operatorConfiguration.dynamicConfig.create | If set to true, a config map would be created & watched by operator as source of truth for hot properties loading. | false |
-| operatorConfiguration.dynamicConfig.enable | If set to true, operator would honor the created config map as source of truth for hot properties loading. | false |
+| operatorConfiguration.dynamicConfig.enabled | If set to true, operator would honor the created config map as source of truth for hot properties loading. | false |
+| operatorConfiguration.dynamicConfig.enable | Deprecated, use `operatorConfiguration.dynamicConfig.enabled`. Still honored: `enable: true` wins over `enabled: false`. Removed in chart 2.0.0 (SPARK-59533). | |
| operatorConfiguration.dynamicConfig.annotations | Annotations to be applied for the dynamicConfig resources. | `"helm.sh/resource-policy": keep` |
| operatorConfiguration.dynamicConfig.data | Data field (key-value pairs) that acts as hot properties in the config map. | `spark.kubernetes.operator.reconciler.intervalSeconds: "60"` |
@@ -154,7 +155,7 @@ for the operator pod:
```yaml
operatorDeployment:
networkPolicy:
- enable: true
+ enabled: true
metricsIngress:
- namespaceSelector:
matchLabels:
@@ -174,6 +175,12 @@ Note that this requires a CNI plugin that enforces NetworkPolicy; on clusters wi
a plugin the policy is silently ignored. Egress traffic of the operator (Kubernetes API
server, DNS) is not restricted by this policy.
+The legacy key `operatorDeployment.networkPolicy.enable` is deprecated in favor of `enabled`
+and will be removed in chart `2.0.0` ([SPARK-59533](https://issues.apache.org/jira/browse/SPARK-59533)).
+It is still honored: the NetworkPolicy is created when either key is `true`, so a stale
+`enable: true` in a base values file wins over `enabled: false` and must be removed to turn
+the feature off. The same rule applies to `operatorConfiguration.dynamicConfig.enable`.
+
## Operator Health(Liveness) Probe with Sentinel Resource
Learning
diff --git a/tests/e2e/helm/dynamic-config-values-2.yaml b/tests/e2e/helm/dynamic-config-values-2.yaml
index b6410cb..38891b4 100644
--- a/tests/e2e/helm/dynamic-config-values-2.yaml
+++ b/tests/e2e/helm/dynamic-config-values-2.yaml
@@ -27,7 +27,7 @@ workloadResources:
operatorConfiguration:
dynamicConfig:
- enable: true
+ enabled: true
create: true
data:
spark.kubernetes.operator.watchedNamespaces: "spark-3"
diff --git a/tests/e2e/helm/dynamic-config-values-file.yaml b/tests/e2e/helm/dynamic-config-values-file.yaml
index 568477a..8a7296b 100644
--- a/tests/e2e/helm/dynamic-config-values-file.yaml
+++ b/tests/e2e/helm/dynamic-config-values-file.yaml
@@ -33,7 +33,7 @@ operatorDeployment:
operatorConfiguration:
dynamicConfig:
- enable: true
+ enabled: true
source: file
create: true
data:
diff --git a/tests/e2e/helm/dynamic-config-values.yaml b/tests/e2e/helm/dynamic-config-values.yaml
index aac7cd2..d0d0569 100644
--- a/tests/e2e/helm/dynamic-config-values.yaml
+++ b/tests/e2e/helm/dynamic-config-values.yaml
@@ -33,7 +33,7 @@ operatorDeployment:
operatorConfiguration:
dynamicConfig:
- enable: true
+ enabled: true
create: true
data:
spark.kubernetes.operator.watchedNamespaces: "default"
diff --git a/tests/e2e/helm/helm-test-values/network-policy/values.yaml b/tests/e2e/helm/helm-test-values/network-policy/values.yaml
index cef0a92..62ab1d8 100644
--- a/tests/e2e/helm/helm-test-values/network-policy/values.yaml
+++ b/tests/e2e/helm/helm-test-values/network-policy/values.yaml
@@ -21,7 +21,7 @@
operatorDeployment:
networkPolicy:
- enable: true
+ enabled: true
metricsIngress:
- namespaceSelector:
matchLabels:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org