You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/09/15 15:34:47 UTC
(spark-kubernetes-operator) branch main updated: [SPARK-59519] Add opt-in `Kueue` RBAC rules to Helm chart
This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new ac5b0ee [SPARK-59519] Add opt-in `Kueue` RBAC rules to Helm chart
ac5b0ee is described below
commit ac5b0eea024346d8d9e579dbe4de1ccfad2bcd75
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Tue Sep 15 08:34:40 2026 -0700
[SPARK-59519] Add opt-in `Kueue` RBAC rules to Helm chart
### What changes were proposed in this pull request?
This PR adds an opt-in Helm value `operatorRbac.kueue.enabled` (default `false`). When enabled, the operator is granted the RBAC rules that [Kueue](https://kueue.sigs.k8s.io/docs/tasks/dev/integrate_a_custom_job/) requires from an external framework integration:
| Group / Resource | Verbs | Granted via |
|---|---|---|
| `kueue.x-k8s.io` / `workloads` | get, list, watch, create, update, patch, delete | ClusterRole and Role |
| `kueue.x-k8s.io` / `workloads/status` | get, update, patch | ClusterRole and Role |
| `kueue.x-k8s.io` / `workloads/finalizers` | update | ClusterRole and Role |
| `kueue.x-k8s.io` / `resourceflavors`, `workloadpriorityclasses` | get, list, watch | ClusterRole only |
| `scheduling.k8s.io` / `priorityclasses` | get, list, watch | ClusterRole only |
The namespaced `workloads` rules live in the shared `operatorRbacRules` block under `{{- if }}`, like the existing `leases` rule. The cluster-scoped resources go into a new `operatorClusterRbacRules` wrapper used only by the ClusterRole, since a namespaced Role cannot grant them. `events.k8s.io/events` from the Kueue doc is intentionally left out: the operator only emits core `events`, which are already granted.
Also included: `helm test` assertions for the new grants, a `helm-tests` CI group `kueue` that installs Kueue v0.19.4 and runs them, a check that `workloads` create is denied with the default values, and a `docs/operations.md` entry.
### Why are the changes needed?
This is the deployment-side preparation for the Kueue integration (SPARK-59486, SPARK-59490, SPARK-59503). Keeping the grant opt-in avoids widening the operator ClusterRole for users who do not run Kueue. The operator runtime changes and the Kueue-side `integrations.externalFrameworks` configuration are out of scope.
### Does this PR introduce _any_ user-facing change?
Yes, a new Helm value `operatorRbac.kueue.enabled` (default `false`). With the default, the rendered manifests are identical to the current chart.
### How was this patch tested?
- `helm lint --strict` passes.
- `helm template` output with the default values is identical to `main`; with `operatorRbac.kueue.enabled=true` the ClusterRole gets all five rules and, when `operatorRbac.role.create=true`, each workload-namespace Role gets only the three `workloads` rules.
- `--set operatorRbac.kueue=null` is now rejected by the values schema like every other `operatorRbac` block.
- New `helm-tests / kueue` CI job: installs Kueue, runs `helm test` with the value enabled, then upgrades to the default values and asserts the operator service account is denied `create` on `workloads.kueue.x-k8s.io`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5.1
Closes #827 from dongjoon-hyun/SPARK-59519.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.github/workflows/build_and_test.yml | 17 +++++++
.../templates/operator-rbac.yaml | 57 +++++++++++++++++++++-
.../templates/tests/test-rbac.yaml | 16 ++++++
.../spark-kubernetes-operator/values.schema.json | 16 +++++-
.../helm/spark-kubernetes-operator/values.yaml | 6 +++
docs/operations.md | 7 +++
tests/e2e/helm/helm-test-values/kueue/values.yaml | 25 ++++++++++
7 files changed, 142 insertions(+), 2 deletions(-)
diff --git a/.github/workflows/build_and_test.yml b/.github/workflows/build_and_test.yml
index 96debfc..2c2b0ab 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -238,6 +238,7 @@ jobs:
- "1.37.0"
test-group:
- configmap-metadata
+ - kueue
- network-policy
- operator-command
steps:
@@ -267,6 +268,13 @@ jobs:
run: |
kubectl get pods -A
kubectl describe node
+ - name: Install Kueue
+ if: matrix.test-group == 'kueue'
+ run: |
+ kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.19.4/manifests.yaml
+ # Kueue installs webhooks for pods, deployments and statefulsets with failurePolicy=Fail,
+ # so wait for the controller (whose readyz covers the webhook server) before helm install.
+ kubectl wait deploy/kueue-controller-manager -n kueue-system --for=condition=Available --timeout=5m
- name: Run Spark K8s Operator Helm Tests
run: |
eval $(minikube docker-env)
@@ -277,6 +285,15 @@ jobs:
build-tools/helm/spark-kubernetes-operator/
minikube docker-env --unset
helm test spark
+ - name: Verify Kueue RBAC is denied by default
+ if: matrix.test-group == 'kueue'
+ run: |
+ # The same impersonated check must answer "yes" first, otherwise a wrong subject
+ # would make the denial below vacuous.
+ kubectl auth can-i create workloads.kueue.x-k8s.io --as=system:serviceaccount:default:spark-operator
+ helm upgrade spark -f build-tools/helm/spark-kubernetes-operator/values.yaml \
+ build-tools/helm/spark-kubernetes-operator/
+ if kubectl auth can-i create workloads.kueue.x-k8s.io --as=system:serviceaccount:default:spark-operator; then exit 1; fi
lint:
name: "Linter and documentation"
runs-on: ubuntu-26.04
diff --git a/build-tools/helm/spark-kubernetes-operator/templates/operator-rbac.yaml b/build-tools/helm/spark-kubernetes-operator/templates/operator-rbac.yaml
index 50d0511..bcf8a4d 100644
--- a/build-tools/helm/spark-kubernetes-operator/templates/operator-rbac.yaml
+++ b/build-tools/helm/spark-kubernetes-operator/templates/operator-rbac.yaml
@@ -122,6 +122,61 @@ rules:
- patch
- delete
{{- end }}
+{{- if .Values.operatorRbac.kueue.enabled }}
+ - apiGroups:
+ - "kueue.x-k8s.io"
+ resources:
+ - workloads
+ verbs:
+ - get
+ - list
+ - watch
+ - create
+ - update
+ - patch
+ - delete
+ - apiGroups:
+ - "kueue.x-k8s.io"
+ resources:
+ - workloads/status
+ verbs:
+ - get
+ - update
+ - patch
+ - apiGroups:
+ - "kueue.x-k8s.io"
+ resources:
+ - workloads/finalizers
+ verbs:
+ - update
+{{- end }}
+{{- end }}
+
+{{/*
+RBAC rules used to create the operator clusterrole: the shared rules above plus
+the rules for cluster-scoped resources, which a namespaced role cannot grant
+*/}}
+{{- define "spark-operator.operatorClusterRbacRules" }}
+{{- include "spark-operator.operatorRbacRules" . }}
+{{- if .Values.operatorRbac.kueue.enabled }}
+ - apiGroups:
+ - "kueue.x-k8s.io"
+ resources:
+ - resourceflavors
+ - workloadpriorityclasses
+ verbs:
+ - get
+ - list
+ - watch
+ - apiGroups:
+ - "scheduling.k8s.io"
+ resources:
+ - priorityclasses
+ verbs:
+ - get
+ - list
+ - watch
+{{- end }}
{{- end }}
{{/*
@@ -170,7 +225,7 @@ kind: ClusterRole
metadata:
name: {{ .Values.operatorRbac.clusterRole.name }}
{{- template "spark-operator.rbacLabelsAnnotations" $ }}
-{{- template "spark-operator.operatorRbacRules" $ }}
+{{- template "spark-operator.operatorClusterRbacRules" $ }}
---
{{- end }}
{{- if and (eq (include "spark-operator.dynamicConfig.enabled" .) "true") (eq .Values.operatorConfiguration.dynamicConfig.source "configMap") }}
diff --git a/build-tools/helm/spark-kubernetes-operator/templates/tests/test-rbac.yaml b/build-tools/helm/spark-kubernetes-operator/templates/tests/test-rbac.yaml
index 9647769..7bd63d9 100644
--- a/build-tools/helm/spark-kubernetes-operator/templates/tests/test-rbac.yaml
+++ b/build-tools/helm/spark-kubernetes-operator/templates/tests/test-rbac.yaml
@@ -77,6 +77,22 @@ spec:
kubectl auth can-i watch grpcroutes.gateway.networking.k8s.io --all-namespaces
kubectl auth can-i create grpcroutes.gateway.networking.k8s.io --all-namespaces
fi
+ {{- if .Values.operatorRbac.kueue.enabled }}
+
+ # The Kueue grant is opt-in and, like Gateway API, can only be asserted
+ # where the Kueue CRDs are installed.
+ if kubectl api-resources --api-group=kueue.x-k8s.io --no-headers -o name | grep -q workloads; then
+ kubectl auth can-i watch workloads.kueue.x-k8s.io --all-namespaces
+ kubectl auth can-i create workloads.kueue.x-k8s.io --all-namespaces
+ kubectl auth can-i patch workloads.kueue.x-k8s.io --subresource=status --all-namespaces
+ kubectl auth can-i update workloads.kueue.x-k8s.io --subresource=finalizers --all-namespaces
+ kubectl auth can-i watch resourceflavors.kueue.x-k8s.io
+ kubectl auth can-i watch workloadpriorityclasses.kueue.x-k8s.io
+ fi
+
+ # PriorityClass is built in, so this one needs no CRD-presence guard.
+ kubectl auth can-i watch priorityclasses.scheduling.k8s.io
+ {{- end }}
# The operator must not be cluster-admin. These also prove the suite is
# able to observe a denial at all. `set -e` ignores a failing `!`
diff --git a/build-tools/helm/spark-kubernetes-operator/values.schema.json b/build-tools/helm/spark-kubernetes-operator/values.schema.json
index 60fa41f..21e347a 100644
--- a/build-tools/helm/spark-kubernetes-operator/values.schema.json
+++ b/build-tools/helm/spark-kubernetes-operator/values.schema.json
@@ -428,7 +428,8 @@
"clusterRoleBinding",
"role",
"roleBinding",
- "configManagement"
+ "configManagement",
+ "kueue"
],
"properties": {
"serviceAccount": {
@@ -567,6 +568,19 @@
}
}
},
+ "kueue": {
+ "type": "object",
+ "description": "Kueue integration RBAC",
+ "required": [
+ "enabled"
+ ],
+ "properties": {
+ "enabled": {
+ "type": "boolean",
+ "description": "Whether to grant the operator access to Kueue resources"
+ }
+ }
+ },
"labels": {
"type": "object",
"description": "Labels for RBAC resources",
diff --git a/build-tools/helm/spark-kubernetes-operator/values.yaml b/build-tools/helm/spark-kubernetes-operator/values.yaml
index e9fdb47..62d3bb4 100644
--- a/build-tools/helm/spark-kubernetes-operator/values.yaml
+++ b/build-tools/helm/spark-kubernetes-operator/values.yaml
@@ -137,6 +137,12 @@ operatorRbac:
create: true
roleName: "spark-operator-config-monitor"
roleBindingName: "spark-operator-config-monitor-role-binding"
+ kueue:
+ # If enabled, the operator (cluster)role would be granted access to Kueue workloads.
+ # The cluster-scoped resourceflavors, workloadpriorityclasses and priorityclasses are
+ # granted through the ClusterRole only, so {operatorRbac.clusterRole.create} is required
+ # for the operator to read them. Kueue itself must be installed separately.
+ enabled: false
labels:
"app.kubernetes.io/component": "operator-rbac"
diff --git a/docs/operations.md b/docs/operations.md
index 66dd7e1..11a4d09 100644
--- a/docs/operations.md
+++ b/docs/operations.md
@@ -37,6 +37,12 @@ under the License.
implementation such as Istio) before submitting SparkApplications that request Gateway API
routing. If the CRDs are missing, reconciliation of such SparkApplications will fail with a
`no matches for kind "HTTPRoute"` error from the Kubernetes API.
+- **Kueue** (`workloads.kueue.x-k8s.io`, `resourceflavors.kueue.x-k8s.io`,
+ `workloadpriorityclasses.kueue.x-k8s.io`) — required only when `operatorRbac.kueue.enabled` is
+ set. Kueue is not bundled with the operator; install it from
+ [kueue.sigs.k8s.io](https://kueue.sigs.k8s.io/docs/installation/), and register
+ `SparkApplication.v1.spark.apache.org` and `SparkCluster.v1.spark.apache.org` in Kueue's
+ `integrations.externalFrameworks`.
## Spark Application Namespaces
@@ -108,6 +114,7 @@ following table:
| operatorRbac.configManagement.create | Enable this to create a Role for operator configuration management (hot property loading and leader election). | true |
| operatorRbac.configManagement.roleName | Role name for operator configuration management. | `spark-operator-config-role` |
| operatorRbac.configManagement.roleBinding | RoleBinding name for operator configuration management. | `"spark-operator-config-monitor-role-binding"` |
+| operatorRbac.kueue.enabled | Grant the operator access to Kueue `workloads`, `resourceflavors`, `workloadpriorityclasses` and to `priorityclasses`. The cluster-scoped ones need `clusterRole.create`. See [Optional Prerequisites](#optional-prerequisites). | false |
| operatorRbac.labels | Labels to be applied on all created `operatorRbac` resources. | `"app.kubernetes.io/component": "operator-rbac"` |
| workloadResources.namespaces.create | Whether to create dedicated namespaces for Spark workload. | true |
| workloadResources.namespaces.overrideWatchedNamespaces | When enabled, operator would by default only watch namespace(s) provided in data field. | true |
diff --git a/tests/e2e/helm/helm-test-values/kueue/values.yaml b/tests/e2e/helm/helm-test-values/kueue/values.yaml
new file mode 100644
index 0000000..23ab072
--- /dev/null
+++ b/tests/e2e/helm/helm-test-values/kueue/values.yaml
@@ -0,0 +1,25 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Test values for validating the opt-in Kueue RBAC rules. The Kueue CRDs must be installed
+# on the cluster for the `helm test` assertions to run.
+#
+# Usage:
+# helm install spark-operator . -f tests/e2e/helm/helm-test-values/kueue/values.yaml
+# helm test spark-operator
+
+operatorRbac:
+ kueue:
+ enabled: true
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org