You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/09/24 19:41:25 UTC
(spark-kubernetes-operator) branch main updated: [SPARK-59770] Add `ValidatingAdmissionPolicy` to keep the Kueue queue of started `Spark(Application|Cluster)`
This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new b17bb353 [SPARK-59770] Add `ValidatingAdmissionPolicy` to keep the Kueue queue of started `Spark(Application|Cluster)`
b17bb353 is described below
commit b17bb353b44babe2d78136ccd1965911beb65c2b
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Thu Sep 24 12:41:19 2026 -0700
[SPARK-59770] Add `ValidatingAdmissionPolicy` to keep the Kueue queue of started `Spark(Application|Cluster)`
### What changes were proposed in this pull request?
This PR aims to add a `ValidatingAdmissionPolicy` to the Helm chart, so that the `kueue.x-k8s.io/queue-name` label of a started `SparkApplication` or `SparkCluster` cannot change.
- It is installed with `operatorRbac.kueue.enabled` and matches the watched namespaces.
- Like Kueue, the label can change only before the resource starts (`Submitted` or `ScheduledToRestart`) or while a `SparkCluster` is `Suspended`. The operator follows such a change since SPARK-59766.
- Update the documentation and the `kueue` E2E test.
### Why are the changes needed?
Currently, the label of a running resource can change while its admitted `Workload` keeps the quota in the old queue. Kueue prevents this for its built-in integrations with a webhook, but `spark.apache.org` resources are not among them. A CRD validation rule cannot access `metadata.labels`.
### Does this PR introduce _any_ user-facing change?
Yes, but only when `operatorRbac.kueue.enabled` is set. The Kueue integration is not released yet. The latest release is [1.0.0](https://github.com/apache/spark-kubernetes-operator/releases/tag/1.0.0) (2026-07-23).
### How was this patch tested?
Pass the CIs with the updated `kueue` E2E test. I also verified the policy manually on a `kind` cluster (K8s 1.37.0).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #884 from dongjoon-hyun/SPARK-59770.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.../kueue-validating-admission-policy.yaml | 81 ++++++++++++++++++++++
.../helm/spark-kubernetes-operator/values.yaml | 6 +-
docs/operations.md | 9 ++-
docs/spark_custom_resources.md | 18 +++++
tests/e2e/kueue/chainsaw-test.yaml | 38 ++++++++++
5 files changed, 147 insertions(+), 5 deletions(-)
diff --git a/build-tools/helm/spark-kubernetes-operator/templates/kueue-validating-admission-policy.yaml b/build-tools/helm/spark-kubernetes-operator/templates/kueue-validating-admission-policy.yaml
new file mode 100644
index 00000000..f89e3908
--- /dev/null
+++ b/build-tools/helm/spark-kubernetes-operator/templates/kueue-validating-admission-policy.yaml
@@ -0,0 +1,81 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{/*
+Keep the queue-name label of a SparkApplication or SparkCluster from changing once it started,
+until the application is scheduled to restart or the cluster is Suspended. See the Kueue section
+of docs/spark_custom_resources.md.
+*/}}
+{{- if .Values.operatorRbac.kueue.enabled }}
+{{- $name := printf "%s-spark-kueue-queue-name" .Release.Name }}
+---
+apiVersion: admissionregistration.k8s.io/v1
+kind: ValidatingAdmissionPolicy
+metadata:
+ name: {{ $name }}
+{{- template "spark-operator.rbacLabelsAnnotations" $ }}
+spec:
+ failurePolicy: Fail
+ matchConstraints:
+ {{- if .Values.workloadResources.namespaces.overrideWatchedNamespaces }}
+ namespaceSelector:
+ matchExpressions:
+ - key: kubernetes.io/metadata.name
+ operator: In
+ values:
+ {{- range (include "spark-operator.workloadNamespacesStr" . | trim | splitList ",") }}
+ - {{ . | quote }}
+ {{- end }}
+ {{- end }}
+ resourceRules:
+ - apiGroups: ["spark.apache.org"]
+ apiVersions: ["*"]
+ operations: ["UPDATE"]
+ resources: ["sparkapplications", "sparkclusters"]
+ variables:
+ - name: oldQueue
+ expression: >-
+ has(oldObject.metadata.labels) && 'kueue.x-k8s.io/queue-name' in oldObject.metadata.labels
+ ? oldObject.metadata.labels['kueue.x-k8s.io/queue-name'] : ''
+ - name: newQueue
+ expression: >-
+ has(object.metadata.labels) && 'kueue.x-k8s.io/queue-name' in object.metadata.labels
+ ? object.metadata.labels['kueue.x-k8s.io/queue-name'] : ''
+ - name: oldState
+ expression: >-
+ has(oldObject.status) && has(oldObject.status.currentState)
+ && has(oldObject.status.currentState.currentStateSummary)
+ ? oldObject.status.currentState.currentStateSummary : ''
+ validations:
+ # The states in which ApplicationStateSummary or ClusterStateSummary#isInitializing holds, i.e.
+ # before an attempt starts, and ClusterStateSummary.Suspended. Keep them in sync.
+ - expression: >-
+ variables.oldQueue == variables.newQueue
+ || variables.oldState in ['', 'Submitted', 'ScheduledToRestart', 'Suspended']
+ message: >-
+ The kueue.x-k8s.io/queue-name label can be changed only before the resource starts, i.e.
+ while it is Submitted or ScheduledToRestart, or while a SparkCluster is Suspended, since
+ the Kueue Workload admitted for it keeps its quota in that queue.
+ reason: Forbidden
+---
+apiVersion: admissionregistration.k8s.io/v1
+kind: ValidatingAdmissionPolicyBinding
+metadata:
+ name: {{ $name }}
+{{- template "spark-operator.rbacLabelsAnnotations" $ }}
+spec:
+ policyName: {{ $name }}
+ validationActions: ["Deny"]
+{{- end }}
diff --git a/build-tools/helm/spark-kubernetes-operator/values.yaml b/build-tools/helm/spark-kubernetes-operator/values.yaml
index ecc5ba8a..05077eea 100644
--- a/build-tools/helm/spark-kubernetes-operator/values.yaml
+++ b/build-tools/helm/spark-kubernetes-operator/values.yaml
@@ -143,8 +143,10 @@ operatorRbac:
# granted through the ClusterRole only, so {operatorRbac.clusterRole.create} is required
# for the operator to read them. Kueue itself must be installed separately.
# Enabling this also enables the Kueue integration of the operator
- # (spark.kubernetes.operator.kueue.enabled), which requires Kueue to be installed.
- # Otherwise, the operator ignores the kueue.x-k8s.io/queue-name label.
+ # (spark.kubernetes.operator.kueue.enabled), which requires Kueue to be installed, and
+ # installs a ValidatingAdmissionPolicy to reject a change of the kueue.x-k8s.io/queue-name
+ # label of a started SparkApplication or SparkCluster, see the Kueue section of
+ # docs/spark_custom_resources.md. Otherwise, the operator ignores the label.
enabled: false
labels:
"app.kubernetes.io/component": "operator-rbac"
diff --git a/docs/operations.md b/docs/operations.md
index 9f619d34..97eea019 100644
--- a/docs/operations.md
+++ b/docs/operations.md
@@ -52,8 +52,11 @@ under the License.
`resourceflavors`, `workloadpriorityclasses` and `priorityclasses` are granted only by
`clusterRole.create`. Without the permission, the priority of a `Workload` is skipped with a
warning, while the `ResourceFlavor`s which Kueue assigned cannot be read at all, so an admitted
- resource is held until the permission is granted. See
- [Kueue](spark_custom_resources.md#kueue) for how resources opt into queueing.
+ resource is held until the permission is granted. `operatorRbac.kueue.enabled` also installs a
+ cluster-scoped `ValidatingAdmissionPolicy` and its binding, so whoever installs or upgrades the
+ chart needs access to the `validatingadmissionpolicies` and `validatingadmissionpolicybindings`
+ of `admissionregistration.k8s.io`. See [Kueue](spark_custom_resources.md#kueue) for how
+ resources opt into queueing and for the policy.
## Spark Application Namespaces
@@ -134,7 +137,7 @@ following table:
| operatorRbac.configManagement.create | Enable this to create a Role for operator configuration management (hot property loading from ConfigMap). Requires `dynamicConfig` with the `configMap` source. | true |
| operatorRbac.configManagement.roleName | Role name for operator configuration management. | `"spark-operator-config-monitor"` |
| operatorRbac.configManagement.roleBindingName | RoleBinding name for operator configuration management. | `"spark-operator-config-monitor-role-binding"` |
-| operatorRbac.kueue.enabled | Grant the operator access to Kueue `workloads`, `resourceflavors`, `workloadpriorityclasses` and to `priorityclasses`, and enable the Kueue integration (`spark.kubernetes.operator.kueue.enabled`). The cluster-scoped ones need `clusterRole.create`. See [Optional Prerequisites](#optional-prerequisites). | false |
+| operatorRbac.kueue.enabled | Grant the operator access to Kueue `workloads`, `resourceflavors`, `workloadpriorityclasses` and to `priorityclasses`, and enable the Kueue integration (`spark.kubernetes.operator.kueue.enabled`). The cluster-scoped ones need `clusterRole.create`. It also installs a `ValidatingAdmissionPolicy` which keeps the `kueue.x-k8s.io/queue-name` label of a started resource. See [Optional Prerequisites](#optional-prerequisites). [...]
| operatorRbac.labels | Labels to be applied on all created `operatorRbac` resources. | `"app.kubernetes.io/component": "operator-rbac"` |
| operatorRbac.annotations | Annotations to be applied on all created `operatorRbac` resources. | |
| workloadResources.namespaces.create | Whether to create dedicated namespaces for Spark workload. | true |
diff --git a/docs/spark_custom_resources.md b/docs/spark_custom_resources.md
index a238f568..59d3815b 100644
--- a/docs/spark_custom_resources.md
+++ b/docs/spark_custom_resources.md
@@ -680,6 +680,24 @@ spec:
terminating pods still occupy the quota. The `Workload` is deleted even if the
`kueue.x-k8s.io/queue-name` label was removed after the admission. The resource is queued again
when it is resumed with the `kueue.x-k8s.io/queue-name` label.
+* Like the webhooks of Kueue built-in integrations, which let only a suspended job change its queue,
+ the Helm chart installs a `ValidatingAdmissionPolicy` with `operatorRbac.kueue.enabled`. It
+ rejects an update that adds, changes or removes the `kueue.x-k8s.io/queue-name` label of a
+ `SparkApplication` or a `SparkCluster` once its status says that it started, since the `Workload`
+ admitted for it keeps its quota in that queue. The label may change again when a
+ `SparkApplication` is `ScheduledToRestart`, whose next attempt is queued with a new `Workload`, or
+ when a `SparkCluster` is `Suspended`. So the label of a terminated resource, e.g. `Failed`, stays
+ as it was. To move a running `SparkCluster` to another queue, set `spec.suspend` to `true`, change
+ the label once the cluster is `Suspended`, and then resume it, while a running `SparkApplication`
+ moves with its next attempt. Before a resource starts, the operator follows the label as described
+ above. The policy reads only the status, so a resource whose first status is not persisted yet,
+ e.g. after the update following the request of its driver or master failed, looks like it has not
+ started. Its label may still change then, while the operator keeps the admitted `Workload` of such
+ a resource. The policy is cluster-scoped, so it is named `<release name>-spark-kueue-queue-name`
+ and matches only the namespaces which the chart sets as the watched namespaces, or all namespaces
+ if `workloadResources.namespaces.overrideWatchedNamespaces` is disabled. Whoever installs the
+ chart needs access to the `validatingadmissionpolicies` and `validatingadmissionpolicybindings` of
+ `admissionregistration.k8s.io`, while the operator does not.
* Once the integration is disabled or the access of the operator to `Workload`s is revoked, e.g.
by disabling `operatorRbac.kueue.enabled`, the operator no longer deletes them, and a `Workload`
left behind keeps its quota until its owner is deleted. While the integration stays enabled
diff --git a/tests/e2e/kueue/chainsaw-test.yaml b/tests/e2e/kueue/chainsaw-test.yaml
index 40bbcb07..7acf8f81 100644
--- a/tests/e2e/kueue/chainsaw-test.yaml
+++ b/tests/e2e/kueue/chainsaw-test.yaml
@@ -168,6 +168,17 @@ spec:
type: Normal
reason: KueueAdmissionPending
(contains(message || '', 'sparkapplication-spark-job-kueue-queued-test')): true
+ # The ValidatingAdmissionPolicy of the chart lets the queued application move to another
+ # queue, here and back again
+ - script:
+ timeout: 30s
+ env:
+ - name: LOCAL_QUEUE
+ value: ($LOCAL_QUEUE)
+ content: |
+ set -e
+ kubectl label sparkapplication spark-job-kueue-queued-test -n default --overwrite kueue.x-k8s.io/queue-name=another-queue
+ kubectl label sparkapplication spark-job-kueue-queued-test -n default --overwrite "kueue.x-k8s.io/queue-name=$LOCAL_QUEUE"
- script:
timeout: 30s
env:
@@ -237,6 +248,14 @@ spec:
metadata:
name: sparkapplication-spark-job-kueue-queued-test
namespace: default
+ # Like a finished Kueue job, the terminated application keeps its queue
+ - script:
+ timeout: 30s
+ content: |
+ kubectl label sparkapplication spark-job-kueue-queued-test -n default kueue.x-k8s.io/queue-name-
+ check:
+ ($error != null): true
+ (contains($stderr, 'ValidatingAdmissionPolicy')): true
catch:
- describe:
apiVersion: spark.apache.org/v1
@@ -539,6 +558,15 @@ spec:
type: Normal
reason: KueueAdmitted
(contains(message || '', 'sparkcluster-spark-cluster-kueue-queued-test')): true
+ # The ValidatingAdmissionPolicy of the chart keeps the running cluster in its queue, since its
+ # Workload holds the quota until the cluster is Suspended
+ - script:
+ timeout: 30s
+ content: |
+ kubectl label sparkcluster spark-cluster-kueue-queued-test -n default kueue.x-k8s.io/queue-name-
+ check:
+ ($error != null): true
+ (contains($stderr, 'ValidatingAdmissionPolicy')): true
# Suspending the running cluster releases its master and workers, and then its quota
- script:
timeout: 30s
@@ -555,6 +583,16 @@ spec:
status:
currentState:
currentStateSummary: Suspended
+ # The Suspended cluster may move to another queue, here and back again
+ - script:
+ timeout: 30s
+ env:
+ - name: LOCAL_QUEUE
+ value: ($LOCAL_QUEUE)
+ content: |
+ set -e
+ kubectl label sparkcluster spark-cluster-kueue-queued-test -n default --overwrite kueue.x-k8s.io/queue-name=another-queue
+ kubectl label sparkcluster spark-cluster-kueue-queued-test -n default --overwrite "kueue.x-k8s.io/queue-name=$LOCAL_QUEUE"
- error:
timeout: 120s
resource:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org