You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/09/15 15:34:47 UTC

(spark-kubernetes-operator) branch main updated: [SPARK-59519] Add opt-in `Kueue` RBAC rules to Helm chart

This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git


The following commit(s) were added to refs/heads/main by this push:
     new ac5b0ee  [SPARK-59519] Add opt-in `Kueue` RBAC rules to Helm chart
ac5b0ee is described below

commit ac5b0eea024346d8d9e579dbe4de1ccfad2bcd75
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Tue Sep 15 08:34:40 2026 -0700

    [SPARK-59519] Add opt-in `Kueue` RBAC rules to Helm chart
    
    ### What changes were proposed in this pull request?
    
    This PR adds an opt-in Helm value `operatorRbac.kueue.enabled` (default `false`). When enabled, the operator is granted the RBAC rules that [Kueue](https://kueue.sigs.k8s.io/docs/tasks/dev/integrate_a_custom_job/) requires from an external framework integration:
    
    | Group / Resource | Verbs | Granted via |
    |---|---|---|
    | `kueue.x-k8s.io` / `workloads` | get, list, watch, create, update, patch, delete | ClusterRole and Role |
    | `kueue.x-k8s.io` / `workloads/status` | get, update, patch | ClusterRole and Role |
    | `kueue.x-k8s.io` / `workloads/finalizers` | update | ClusterRole and Role |
    | `kueue.x-k8s.io` / `resourceflavors`, `workloadpriorityclasses` | get, list, watch | ClusterRole only |
    | `scheduling.k8s.io` / `priorityclasses` | get, list, watch | ClusterRole only |
    
    The namespaced `workloads` rules live in the shared `operatorRbacRules` block under `{{- if }}`, like the existing `leases` rule. The cluster-scoped resources go into a new `operatorClusterRbacRules` wrapper used only by the ClusterRole, since a namespaced Role cannot grant them. `events.k8s.io/events` from the Kueue doc is intentionally left out: the operator only emits core `events`, which are already granted.
    
    Also included: `helm test` assertions for the new grants, a `helm-tests` CI group `kueue` that installs Kueue v0.19.4 and runs them, a check that `workloads` create is denied with the default values, and a `docs/operations.md` entry.
    
    ### Why are the changes needed?
    
    This is the deployment-side preparation for the Kueue integration (SPARK-59486, SPARK-59490, SPARK-59503). Keeping the grant opt-in avoids widening the operator ClusterRole for users who do not run Kueue. The operator runtime changes and the Kueue-side `integrations.externalFrameworks` configuration are out of scope.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, a new Helm value `operatorRbac.kueue.enabled` (default `false`). With the default, the rendered manifests are identical to the current chart.
    
    ### How was this patch tested?
    
    - `helm lint --strict` passes.
    - `helm template` output with the default values is identical to `main`; with `operatorRbac.kueue.enabled=true` the ClusterRole gets all five rules and, when `operatorRbac.role.create=true`, each workload-namespace Role gets only the three `workloads` rules.
    - `--set operatorRbac.kueue=null` is now rejected by the values schema like every other `operatorRbac` block.
    - New `helm-tests / kueue` CI job: installs Kueue, runs `helm test` with the value enabled, then upgrades to the default values and asserts the operator service account is denied `create` on `workloads.kueue.x-k8s.io`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Fable 5.1
    
    Closes #827 from dongjoon-hyun/SPARK-59519.
    
    Authored-by: Dongjoon Hyun <do...@apache.org>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 .github/workflows/build_and_test.yml               | 17 +++++++
 .../templates/operator-rbac.yaml                   | 57 +++++++++++++++++++++-
 .../templates/tests/test-rbac.yaml                 | 16 ++++++
 .../spark-kubernetes-operator/values.schema.json   | 16 +++++-
 .../helm/spark-kubernetes-operator/values.yaml     |  6 +++
 docs/operations.md                                 |  7 +++
 tests/e2e/helm/helm-test-values/kueue/values.yaml  | 25 ++++++++++
 7 files changed, 142 insertions(+), 2 deletions(-)

diff --git a/.github/workflows/build_and_test.yml b/.github/workflows/build_and_test.yml
index 96debfc..2c2b0ab 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -238,6 +238,7 @@ jobs:
           - "1.37.0"
         test-group:
           - configmap-metadata
+          - kueue
           - network-policy
           - operator-command
     steps:
@@ -267,6 +268,13 @@ jobs:
         run: |
           kubectl get pods -A
           kubectl describe node
+      - name: Install Kueue
+        if: matrix.test-group == 'kueue'
+        run: |
+          kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.19.4/manifests.yaml
+          # Kueue installs webhooks for pods, deployments and statefulsets with failurePolicy=Fail,
+          # so wait for the controller (whose readyz covers the webhook server) before helm install.
+          kubectl wait deploy/kueue-controller-manager -n kueue-system --for=condition=Available --timeout=5m
       - name: Run Spark K8s Operator Helm Tests
         run: |
           eval $(minikube docker-env)
@@ -277,6 +285,15 @@ jobs:
           build-tools/helm/spark-kubernetes-operator/
           minikube docker-env --unset
           helm test spark
+      - name: Verify Kueue RBAC is denied by default
+        if: matrix.test-group == 'kueue'
+        run: |
+          # The same impersonated check must answer "yes" first, otherwise a wrong subject
+          # would make the denial below vacuous.
+          kubectl auth can-i create workloads.kueue.x-k8s.io --as=system:serviceaccount:default:spark-operator
+          helm upgrade spark -f build-tools/helm/spark-kubernetes-operator/values.yaml \
+          build-tools/helm/spark-kubernetes-operator/
+          if kubectl auth can-i create workloads.kueue.x-k8s.io --as=system:serviceaccount:default:spark-operator; then exit 1; fi
   lint:
     name: "Linter and documentation"
     runs-on: ubuntu-26.04
diff --git a/build-tools/helm/spark-kubernetes-operator/templates/operator-rbac.yaml b/build-tools/helm/spark-kubernetes-operator/templates/operator-rbac.yaml
index 50d0511..bcf8a4d 100644
--- a/build-tools/helm/spark-kubernetes-operator/templates/operator-rbac.yaml
+++ b/build-tools/helm/spark-kubernetes-operator/templates/operator-rbac.yaml
@@ -122,6 +122,61 @@ rules:
       - patch
       - delete
 {{- end }}
+{{- if .Values.operatorRbac.kueue.enabled }}
+  - apiGroups:
+      - "kueue.x-k8s.io"
+    resources:
+      - workloads
+    verbs:
+      - get
+      - list
+      - watch
+      - create
+      - update
+      - patch
+      - delete
+  - apiGroups:
+      - "kueue.x-k8s.io"
+    resources:
+      - workloads/status
+    verbs:
+      - get
+      - update
+      - patch
+  - apiGroups:
+      - "kueue.x-k8s.io"
+    resources:
+      - workloads/finalizers
+    verbs:
+      - update
+{{- end }}
+{{- end }}
+
+{{/*
+RBAC rules used to create the operator clusterrole: the shared rules above plus
+the rules for cluster-scoped resources, which a namespaced role cannot grant
+*/}}
+{{- define "spark-operator.operatorClusterRbacRules" }}
+{{- include "spark-operator.operatorRbacRules" . }}
+{{- if .Values.operatorRbac.kueue.enabled }}
+  - apiGroups:
+      - "kueue.x-k8s.io"
+    resources:
+      - resourceflavors
+      - workloadpriorityclasses
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups:
+      - "scheduling.k8s.io"
+    resources:
+      - priorityclasses
+    verbs:
+      - get
+      - list
+      - watch
+{{- end }}
 {{- end }}
 
 {{/*
@@ -170,7 +225,7 @@ kind: ClusterRole
 metadata:
   name: {{ .Values.operatorRbac.clusterRole.name }}
 {{- template "spark-operator.rbacLabelsAnnotations" $ }}
-{{- template "spark-operator.operatorRbacRules" $ }}
+{{- template "spark-operator.operatorClusterRbacRules" $ }}
 ---
 {{- end }}
 {{- if and (eq (include "spark-operator.dynamicConfig.enabled" .) "true") (eq .Values.operatorConfiguration.dynamicConfig.source "configMap") }}
diff --git a/build-tools/helm/spark-kubernetes-operator/templates/tests/test-rbac.yaml b/build-tools/helm/spark-kubernetes-operator/templates/tests/test-rbac.yaml
index 9647769..7bd63d9 100644
--- a/build-tools/helm/spark-kubernetes-operator/templates/tests/test-rbac.yaml
+++ b/build-tools/helm/spark-kubernetes-operator/templates/tests/test-rbac.yaml
@@ -77,6 +77,22 @@ spec:
             kubectl auth can-i watch  grpcroutes.gateway.networking.k8s.io --all-namespaces
             kubectl auth can-i create grpcroutes.gateway.networking.k8s.io --all-namespaces
           fi
+          {{- if .Values.operatorRbac.kueue.enabled }}
+
+          # The Kueue grant is opt-in and, like Gateway API, can only be asserted
+          # where the Kueue CRDs are installed.
+          if kubectl api-resources --api-group=kueue.x-k8s.io --no-headers -o name | grep -q workloads; then
+            kubectl auth can-i watch  workloads.kueue.x-k8s.io --all-namespaces
+            kubectl auth can-i create workloads.kueue.x-k8s.io --all-namespaces
+            kubectl auth can-i patch  workloads.kueue.x-k8s.io --subresource=status --all-namespaces
+            kubectl auth can-i update workloads.kueue.x-k8s.io --subresource=finalizers --all-namespaces
+            kubectl auth can-i watch  resourceflavors.kueue.x-k8s.io
+            kubectl auth can-i watch  workloadpriorityclasses.kueue.x-k8s.io
+          fi
+
+          # PriorityClass is built in, so this one needs no CRD-presence guard.
+          kubectl auth can-i watch  priorityclasses.scheduling.k8s.io
+          {{- end }}
 
           # The operator must not be cluster-admin. These also prove the suite is
           # able to observe a denial at all. `set -e` ignores a failing `!`
diff --git a/build-tools/helm/spark-kubernetes-operator/values.schema.json b/build-tools/helm/spark-kubernetes-operator/values.schema.json
index 60fa41f..21e347a 100644
--- a/build-tools/helm/spark-kubernetes-operator/values.schema.json
+++ b/build-tools/helm/spark-kubernetes-operator/values.schema.json
@@ -428,7 +428,8 @@
         "clusterRoleBinding",
         "role",
         "roleBinding",
-        "configManagement"
+        "configManagement",
+        "kueue"
       ],
       "properties": {
         "serviceAccount": {
@@ -567,6 +568,19 @@
             }
           }
         },
+        "kueue": {
+          "type": "object",
+          "description": "Kueue integration RBAC",
+          "required": [
+            "enabled"
+          ],
+          "properties": {
+            "enabled": {
+              "type": "boolean",
+              "description": "Whether to grant the operator access to Kueue resources"
+            }
+          }
+        },
         "labels": {
           "type": "object",
           "description": "Labels for RBAC resources",
diff --git a/build-tools/helm/spark-kubernetes-operator/values.yaml b/build-tools/helm/spark-kubernetes-operator/values.yaml
index e9fdb47..62d3bb4 100644
--- a/build-tools/helm/spark-kubernetes-operator/values.yaml
+++ b/build-tools/helm/spark-kubernetes-operator/values.yaml
@@ -137,6 +137,12 @@ operatorRbac:
     create: true
     roleName: "spark-operator-config-monitor"
     roleBindingName: "spark-operator-config-monitor-role-binding"
+  kueue:
+    # If enabled, the operator (cluster)role would be granted access to Kueue workloads.
+    # The cluster-scoped resourceflavors, workloadpriorityclasses and priorityclasses are
+    # granted through the ClusterRole only, so {operatorRbac.clusterRole.create} is required
+    # for the operator to read them. Kueue itself must be installed separately.
+    enabled: false
   labels:
     "app.kubernetes.io/component": "operator-rbac"
 
diff --git a/docs/operations.md b/docs/operations.md
index 66dd7e1..11a4d09 100644
--- a/docs/operations.md
+++ b/docs/operations.md
@@ -37,6 +37,12 @@ under the License.
   implementation such as Istio) before submitting SparkApplications that request Gateway API
   routing. If the CRDs are missing, reconciliation of such SparkApplications will fail with a
   `no matches for kind "HTTPRoute"` error from the Kubernetes API.
+- **Kueue** (`workloads.kueue.x-k8s.io`, `resourceflavors.kueue.x-k8s.io`,
+  `workloadpriorityclasses.kueue.x-k8s.io`) — required only when `operatorRbac.kueue.enabled` is
+  set. Kueue is not bundled with the operator; install it from
+  [kueue.sigs.k8s.io](https://kueue.sigs.k8s.io/docs/installation/), and register
+  `SparkApplication.v1.spark.apache.org` and `SparkCluster.v1.spark.apache.org` in Kueue's
+  `integrations.externalFrameworks`.
 
 ## Spark Application Namespaces
 
@@ -108,6 +114,7 @@ following table:
 | operatorRbac.configManagement.create                             | Enable this to create a Role for operator configuration management (hot property loading and leader election).                                                                 | true                                                                                                    |
 | operatorRbac.configManagement.roleName                           | Role name for operator configuration management.                                                                                                                               | `spark-operator-config-role`                                                                            |
 | operatorRbac.configManagement.roleBinding                        | RoleBinding name for operator configuration management.                                                                                                                        | `"spark-operator-config-monitor-role-binding"`                                                          |
+| operatorRbac.kueue.enabled                                       | Grant the operator access to Kueue `workloads`, `resourceflavors`, `workloadpriorityclasses` and to `priorityclasses`. The cluster-scoped ones need `clusterRole.create`. See [Optional Prerequisites](#optional-prerequisites). | false                                                                                                   |
 | operatorRbac.labels                                              | Labels to be applied on all created `operatorRbac` resources.                                                                                                                  | `"app.kubernetes.io/component": "operator-rbac"`                                                        |
 | workloadResources.namespaces.create                              | Whether to create dedicated namespaces for Spark workload.                                                                                                                     | true                                                                                                    |
 | workloadResources.namespaces.overrideWatchedNamespaces           | When enabled, operator would by default only watch namespace(s) provided in data field.                                                                                        | true                                                                                                    |
diff --git a/tests/e2e/helm/helm-test-values/kueue/values.yaml b/tests/e2e/helm/helm-test-values/kueue/values.yaml
new file mode 100644
index 0000000..23ab072
--- /dev/null
+++ b/tests/e2e/helm/helm-test-values/kueue/values.yaml
@@ -0,0 +1,25 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Test values for validating the opt-in Kueue RBAC rules. The Kueue CRDs must be installed
+# on the cluster for the `helm test` assertions to run.
+#
+# Usage:
+#   helm install spark-operator . -f tests/e2e/helm/helm-test-values/kueue/values.yaml
+#   helm test spark-operator
+
+operatorRbac:
+  kueue:
+    enabled: true


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org