You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dongjoon-hyun (via GitHub)" <gi...@apache.org> on 2026/09/18 22:15:14 UTC
[PR] [SPARK-59655] Fill `priorityClassRef` and `priority` of Kueue `Workload` [spark-kubernetes-operator]
dongjoon-hyun opened a new pull request, #851:
URL: https://github.com/apache/spark-kubernetes-operator/pull/851
### What changes were proposed in this pull request?
This PR aims to fill `spec.priorityClassRef` and `spec.priority` of Kueue `Workload`s. The priority is decided like Kueue's built-in integrations ([Kueue v0.19.4](https://github.com/kubernetes-sigs/kueue/releases/tag/v0.19.4) (2026-09-10)), in this order:
1. The `WorkloadPriorityClass` named by the `kueue.x-k8s.io/priority-class` label.
2. The `PriorityClass` of the first pod set that has one: `driver` or `master` first.
3. The `globalDefault` `PriorityClass`.
If the named class does not exist, the admission is retried. If the operator cannot read the classes (`403 Forbidden`), it creates the `Workload` without priority. If the label changes while the `Workload` is pending, the `Workload` is updated in place.
### Why are the changes needed?
Kueue orders the queue and picks preemption targets by `spec.priority`, and the integration has to fill it. Currently, all Spark `Workload`s have the same priority.
### Does this PR introduce _any_ user-facing change?
No, `Kueue` feature is not released yet.
### How was this patch tested?
Pass the CIs with the newly added unit tests and the new Chainsaw step in `tests/e2e/kueue`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59655] Fill `priorityClassRef` and `priority` of Kueue `Workload` [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #851:
URL: https://github.com/apache/spark-kubernetes-operator/pull/851#issuecomment-5751983442
Thanks, @peter-toth! Rebased on `main` (which now carries #847) and addressed 9 and 10.
- **10.** Took the split. `setPriority`, `getPodPriorityClass` and `isPriorityClassChangeAllowed` moved to a new `KueueWorkloadPriority` class in the same package, and `pmdMain` passes with no suppression on either class, so the `@SuppressWarnings("PMD.GodClass")` I had added is gone. I kept the priority tests in `KueueWorkloadUtilsTest`, since most of them exercise the behavior through `requestAdmission` and share its helpers; only the direct `setPriority` calls moved to the new class. Happy to split the test class too if you prefer.
- **9.** Added `pendingWorkloadWithoutPriorityClassFollowsAnAddedLabel` as written, and confirmed it fails when the quota-reserved short-circuit is removed.
- **2.** Noted, thank you for catching that #847 removed the split. The follow-up will introduce the get/create split itself and use both parts of the gate.
One rebase note: #847's new `kueueTransientApiFailurePublishesNoEvent` tests needed the same `priorityclasses().list()` stub as the other deep-stub clients in those files, because Mockito cannot deep-stub the generic list. Only the mock setup changed there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59655] Fill `priorityClassRef` and `priority` of Kueue `Workload` [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #851:
URL: https://github.com/apache/spark-kubernetes-operator/pull/851#issuecomment-5752257323
Merged to main
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59655] Fill `priorityClassRef` and `priority` of Kueue `Workload` [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #851:
URL: https://github.com/apache/spark-kubernetes-operator/pull/851#issuecomment-5745997189
Thank you for the detailed review, @peter-toth! I addressed 1, 3 and 5.
- **1.** Added `pendingWorkloadKeepsItsPriorityWithoutPermission`. I confirmed it fails when `desired.getSpec().getPriority() != null` is removed and passes as written.
- **3.** Added `isPriorityClassChangeAllowed`, which mirrors the CEL rules: while quota is reserved, the presence, the group and the kind of the ref are frozen, and so is the name of a `scheduling.k8s.io` `PriorityClass`. A `WorkloadPriorityClass` name change now goes through, so a Workload waiting for its admission checks can still be raised. `quotaReservedWorkloadKeepsPriorityClass` is split into a frozen case and a mutable one.
- **5.** Replaced the private constant with `import static java.net.HttpURLConnection.HTTP_FORBIDDEN`.
- **2.** Agreed that the resolution can be gated on a class-name comparison, but it needs `getOrCreateSecondaryResource` split into a get and a create here, which overlaps with #850. Let me do it in a follow-up ticket once #850 is merged.
- **4.** I would like to keep the current behavior. Kueue returns the `Get` error for both the `WorkloadPriorityClass` and the pod `PriorityClass` and retries with backoff, so failing one of them with `SchedulingFailure` would deviate from the integration this PR mirrors. The missing user-facing signal is real, and I think the Kueue admission events in #847 are the right place for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59655] Fill `priorityClassRef` and `priority` of Kueue `Workload` [spark-kubernetes-operator]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #851:
URL: https://github.com/apache/spark-kubernetes-operator/pull/851#discussion_r4057064988
##########
docs/spark_custom_resources.md:
##########
@@ -586,6 +586,13 @@ spec:
See [Optional Prerequisites](operations.md#optional-prerequisites).
* The `Workload` is named `<lower-cased kind>-<resource name>` and is owned by the Spark resource,
so it is garbage collected along with it.
+* Like Kueue built-in integrations, the `Workload` gets the priority of the `WorkloadPriorityClass`
+ named by the `kueue.x-k8s.io/priority-class` label. Without the label, the `priorityClassName` of
+ the driver (or master) pod template is used, then the one of the executor (or worker) pod
+ template, and then the `globalDefault` `PriorityClass`. The resource keeps waiting without a
+ `Workload` while the named priority class does not exist. Changing the label before the
Review Comment:
**Finding 8.** "before the `Workload` is admitted" is now broader than what `isPriorityClassChangeAllowed` permits.
Once quota is reserved but before admission, two label changes are silently dropped by the presence and group/kind freeze: adding the label to a `Workload` that has no ref at all, and adding it to one backed by a `globalDefault` `PriorityClass`. Both fall inside the window this sentence promises. The previous "while the `Workload` waits for quota" was accurate, so the new wording needs the exception spelled out:
```
`Workload` while the named priority class does not exist. Changing the label before the
`Workload` reserves quota updates its priority in place. After that Kueue freezes the
presence, group and kind of the priority class, so only the name of a
`WorkloadPriorityClass` still changes. A changed value of the same class does not affect
the existing `Workload`.
```
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/kueue/KueueWorkloadUtils.java:
##########
@@ -124,9 +143,121 @@ public static AdmissionResult requestAdmission(
deleteWorkload(client, workload);
return AdmissionResult.STALE;
}
+ WorkloadSpec spec = workload.getSpec();
+ PriorityClassRef desiredPriorityClassRef = desired.getSpec().getPriorityClassRef();
+ if (desired.getSpec().getPriority() != null
+ && !Objects.equals(spec.getPriorityClassRef(), desiredPriorityClassRef)
+ && isPriorityClassChangeAllowed(workload, desiredPriorityClassRef)) {
+ // Like Kueue, a changed priority class is applied in place so that the Workload keeps its
+ // position in the queue.
+ log.info(
+ "Updating the priority class of the pending Kueue Workload {}.",
+ workload.getMetadata().getName());
+ spec.setPriorityClassRef(desiredPriorityClassRef);
+ spec.setPriority(desired.getSpec().getPriority());
+ client.resource(workload).update();
+ }
return AdmissionResult.PENDING;
}
+ /**
+ * Checks whether Kueue accepts the priority class change of the given Workload. Like the Workload
+ * CEL rules, the presence, the group and the kind of the priorityClassRef are frozen once the
+ * quota is reserved, and so is the name of a Kubernetes PriorityClass. The name of a
+ * WorkloadPriorityClass stays mutable, which is what Kueue relies on to raise the priority of a
+ * Workload waiting for its admission checks.
+ */
+ private static boolean isPriorityClassChangeAllowed(
+ final Workload workload, final PriorityClassRef desired) {
+ if (workload.getStatus() == null || !workload.getStatus().isQuotaReserved()) {
+ return true;
+ }
+ PriorityClassRef current = workload.getSpec().getPriorityClassRef();
+ return current != null
Review Comment:
**Finding 6.** Three of this helper's four conditions have no test, and every mutant survives the whole suite.
I removed each in turn and ran `KueueWorkloadUtilsTest`:
| removed | result |
|---|---|
| the `getGroup()` **and** `getKind()` equality checks | suite green |
| `!SCHEDULING_API_GROUP.equals(desired.getGroup())` | suite green |
| `current != null` | suite green |
Group and kind are not separable, since the type's own CEL rule makes `group == 'scheduling.k8s.io'` imply `kind == 'PriorityClass'`, so either one alone still blocks. `quotaReservedWorkloadKeepsFrozenPriorityClass` only covers the presence freeze in the ref-to-none direction.
All three are reachable. These three tests pass as written and kill all three mutants — I ran both arms:
```java
@Test
void quotaReservedWorkloadKeepsItsPodPriorityClassName() {
createPriorityClass("default-a", 50, true);
KueueWorkloadUtils.requestAdmission(kubernetesClient, workload("owner-uid-1", 1));
reserveQuota();
// The cluster-wide default changed, which is a PriorityClass name change
kubernetesClient.resource(priorityClass("default-a", 50, false)).update();
createPriorityClass("default-b", 70, true);
Assertions.assertEquals(
AdmissionResult.PENDING,
KueueWorkloadUtils.requestAdmission(kubernetesClient, workload("owner-uid-1", 1)));
Assertions.assertEquals("default-a", getWorkload().getSpec().getPriorityClassRef().getName());
Assertions.assertEquals(50, getWorkload().getSpec().getPriority());
}
@Test
void quotaReservedWorkloadDoesNotSwitchPriorityClassGroup() {
createPriorityClass("default-a", 50, true);
createWorkloadPriorityClass("high", 1000);
KueueWorkloadUtils.requestAdmission(kubernetesClient, workload("owner-uid-1", 1));
reserveQuota();
// Adding the label would switch the group and the kind of the ref
Assertions.assertEquals(
AdmissionResult.PENDING,
KueueWorkloadUtils.requestAdmission(kubernetesClient, workloadWithPriorityClass("high")));
Assertions.assertEquals(
"scheduling.k8s.io", getWorkload().getSpec().getPriorityClassRef().getGroup());
Assertions.assertEquals(50, getWorkload().getSpec().getPriority());
}
@Test
void quotaReservedWorkloadWithoutPriorityClassDoesNotGainOne() {
createWorkloadPriorityClass("high", 1000);
// The Workload is created while the operator cannot read the classes, so it has no ref
KueueWorkloadUtils.requestAdmission(forbiddenClient(), workloadWithPriorityClass("high"));
Assertions.assertNull(getWorkload().getSpec().getPriorityClassRef());
reserveQuota();
// The permission is back, but Kueue no longer accepts adding a priority class
Assertions.assertEquals(
AdmissionResult.PENDING,
KueueWorkloadUtils.requestAdmission(kubernetesClient, workloadWithPriorityClass("high")));
Assertions.assertNull(getWorkload().getSpec().getPriorityClassRef());
}
```
Two helpers to extract: `priorityClass(name, value, globalDefault)` out of `createPriorityClass`, and `forbiddenClient()` out of `pendingWorkloadKeepsItsPriorityWithoutPermission`.
The last one matters most. Without `current != null` it does not merely allow a bad write, it throws `NullPointerException` out of `requestAdmission` on `current.getGroup()`. `holdForKueueAdmission` catches only `IllegalStateException` and `KubernetesClientException`, so the resource ends up in `SchedulingFailure`.
One note on reachability while writing these: a pod template's `priorityClassName` change can never reach this helper, because it changes the pod sets hash and so takes the STALE/recreate path. A `scheduling.k8s.io` ref only changes name through a `globalDefault` swap, which is what the first test uses.
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/kueue/KueueWorkloadUtils.java:
##########
@@ -124,9 +143,121 @@ public static AdmissionResult requestAdmission(
deleteWorkload(client, workload);
return AdmissionResult.STALE;
}
+ WorkloadSpec spec = workload.getSpec();
+ PriorityClassRef desiredPriorityClassRef = desired.getSpec().getPriorityClassRef();
+ if (desired.getSpec().getPriority() != null
+ && !Objects.equals(spec.getPriorityClassRef(), desiredPriorityClassRef)
+ && isPriorityClassChangeAllowed(workload, desiredPriorityClassRef)) {
+ // Like Kueue, a changed priority class is applied in place so that the Workload keeps its
+ // position in the queue.
+ log.info(
+ "Updating the priority class of the pending Kueue Workload {}.",
+ workload.getMetadata().getName());
+ spec.setPriorityClassRef(desiredPriorityClassRef);
+ spec.setPriority(desired.getSpec().getPriority());
+ client.resource(workload).update();
+ }
return AdmissionResult.PENDING;
}
+ /**
+ * Checks whether Kueue accepts the priority class change of the given Workload. Like the Workload
+ * CEL rules, the presence, the group and the kind of the priorityClassRef are frozen once the
+ * quota is reserved, and so is the name of a Kubernetes PriorityClass. The name of a
+ * WorkloadPriorityClass stays mutable, which is what Kueue relies on to raise the priority of a
+ * Workload waiting for its admission checks.
+ */
+ private static boolean isPriorityClassChangeAllowed(
+ final Workload workload, final PriorityClassRef desired) {
+ if (workload.getStatus() == null || !workload.getStatus().isQuotaReserved()) {
+ return true;
+ }
+ PriorityClassRef current = workload.getSpec().getPriorityClassRef();
+ return current != null
+ && desired != null
+ && Objects.equals(current.getGroup(), desired.getGroup())
+ && Objects.equals(current.getKind(), desired.getKind())
+ && !SCHEDULING_API_GROUP.equals(desired.getGroup());
+ }
+
+ /**
+ * Sets the priority of the desired Workload in the same way as Kueue built-in integrations. The
+ * WorkloadPriorityClass of the `kueue.x-k8s.io/priority-class` label takes precedence over the
+ * PriorityClass of the first pod set which has one, and the global default PriorityClass is used
+ * without both. Without any of them, the priority is 0 without a priority class. The priority is
+ * left unset if the operator is not allowed to read the cluster-scoped priority classes.
+ *
+ * @param client The KubernetesClient.
+ * @param workload The Workload whose priority is set in place.
+ * @throws IllegalStateException if the priority class does not exist.
+ */
+ static void setPriority(final KubernetesClient client, final Workload workload) {
+ WorkloadSpec spec = workload.getSpec();
+ try {
+ Map<String, String> labels = workload.getMetadata().getLabels();
+ String workloadPriorityClassName =
+ labels == null ? null : labels.get(Constants.LABEL_WORKLOAD_PRIORITY_CLASS);
+ if (StringUtils.isNotEmpty(workloadPriorityClassName)) {
+ WorkloadPriorityClass workloadPriorityClass =
+ client.resources(WorkloadPriorityClass.class).withName(workloadPriorityClassName).get();
+ if (workloadPriorityClass == null) {
+ throw new IllegalStateException(
+ "Kueue WorkloadPriorityClass " + workloadPriorityClassName + " is not found.");
+ }
+ spec.setPriorityClassRef(
+ new PriorityClassRef(
+ Constants.KUEUE_API_GROUP, "WorkloadPriorityClass", workloadPriorityClassName));
+ spec.setPriority(workloadPriorityClass.getValue());
+ return;
+ }
+ PriorityClass priorityClass = getPodPriorityClass(client, spec);
+ if (priorityClass == null) {
+ spec.setPriorityClassRef(null);
+ spec.setPriority(DEFAULT_PRIORITY);
+ } else {
+ spec.setPriorityClassRef(
+ new PriorityClassRef(
+ SCHEDULING_API_GROUP, "PriorityClass", priorityClass.getMetadata().getName()));
+ spec.setPriority(priorityClass.getValue());
+ }
+ } catch (KubernetesClientException e) {
+ if (e.getCode() != HTTP_FORBIDDEN) {
+ throw e;
+ }
+ log.warn(
+ "Requesting the Kueue Workload {} without priority because the operator is not allowed "
+ + "to read the priority classes.",
+ workload.getMetadata().getName());
+ }
+ }
+
+ /**
+ * Returns the PriorityClass of the first pod set which has one, or the global default
+ * PriorityClass. Like Kueue, the lowest one wins if there are more than one global default.
+ */
+ private static PriorityClass getPodPriorityClass(
+ final KubernetesClient client, final WorkloadSpec spec) {
+ for (PodSet podSet : spec.getPodSets()) {
+ PodTemplateSpec template = podSet.getTemplate();
+ String name =
+ template == null || template.getSpec() == null
+ ? null
+ : template.getSpec().getPriorityClassName();
+ if (StringUtils.isNotEmpty(name)) {
+ PriorityClass priorityClass =
+ client.scheduling().v1().priorityClasses().withName(name).get();
+ if (priorityClass == null) {
+ throw new IllegalStateException("PriorityClass " + name + " is not found.");
+ }
+ return priorityClass;
+ }
+ }
+ return client.scheduling().v1().priorityClasses().list().getItems().stream()
Review Comment:
**Finding 7.** This lookup runs on every reconcile, so a `Workload` with no label and no pod priority class gets re-priced whenever an admin changes the cluster's default. Kueue never does that.
Observed on the mock server, with nothing about the Spark resource changed between the two reconciles:
```
ref after create = PriorityClassRef(group=scheduling.k8s.io, kind=PriorityClass, name=default-a) priority=50
ref after the globalDefault change = PriorityClassRef(group=scheduling.k8s.io, kind=PriorityClass, name=default-b) priority=70
```
`classifyWorkloadsForPriorityUpdate` drops such a Workload from the update set outright:
```go
if !workload.HasNoPriority(wl) && !workload.IsWorkloadPriorityClass(wl) {
continue
}
```
`HasNoPriority` is `PriorityClassRef == nil` and `IsWorkloadPriorityClass` requires the `kueue.x-k8s.io` group, so a `scheduling.k8s.io`-backed Workload never has its priority rewritten after creation. The comment above the function says it outright: "a Pod PriorityClass-backed workload does not follow the label at all". The effect here is that Spark `Workload`s would re-rank inside a `ClusterQueue` while every other workload in it holds still, one write each, which is the opposite of what this PR is trying to match.
This also corrects the fix I sketched for finding 2. Kueue's gate has two parts and I only described the second:
1. skip the Workload entirely when its ref is a Pod `PriorityClass`;
2. otherwise compare the label to the ref's name, and resolve only when they differ.
Part 1 is what fixes this finding, and it is also what makes part 2 well-defined — without it a Pod-`PriorityClass`-backed Workload has no label to compare against, so a name comparison would resolve on every reconcile anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59655] Fill `priorityClassRef` and `priority` of Kueue `Workload` [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #851:
URL: https://github.com/apache/spark-kubernetes-operator/pull/851#issuecomment-5738689296
All tests passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59655] Fill `priorityClassRef` and `priority` of Kueue `Workload` [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #851:
URL: https://github.com/apache/spark-kubernetes-operator/pull/851#issuecomment-5751418533
Thanks again, @peter-toth! Rebased on `main` (which now carries #850) and addressed 6, 7 and 8.
- **6.** Added tests for each condition of `isPriorityClassChangeAllowed`, and I re-ran the mutations: removing the follow rule fails `pendingWorkloadWithPodPriorityClassDoesNotFollowTheLabel` and `pendingWorkloadKeepsItsPodPriorityClassName`, removing `current != null` fails `quotaReservedWorkloadWithoutPriorityClassDoesNotGainOne`, removing the `desired` group check fails `quotaReservedWorkloadDoesNotSwitchPriorityClassGroup`, and removing `desired != null` fails `quotaReservedWorkloadKeepsFrozenPriorityClass`. `priorityClass(...)` and `forbiddenClient()` are extracted as you suggested.
- **7.** Good catch, fixed. `isPriorityClassChangeAllowed` now starts with part 1 of Kueue's gate: a Workload whose ref is a `scheduling.k8s.io` `PriorityClass` is never rewritten, so a `globalDefault` change no longer re-prices a pending Workload. That also simplifies the quota-reserved check, which now only has to accept a `kueue.x-k8s.io` ref on both sides.
- **8.** Reworded to "before the `Workload` reserves quota", and spelled out both the freeze and the fact that a `PriorityClass`-backed `Workload` does not follow the label.
- **2.** Thank you for the correction on the sequencing, I had the wrong PR. I will do the get/create split as a follow-up after #847 lands, with the two-part gate from finding 7.
One note on the rebase: with `resolvePodSetFlavors` from #850 in the same class, PMD now reports `GodClass` for `KueueWorkloadUtils`, so I added `@SuppressWarnings("PMD.GodClass")` like `KueueWorkloadFactory` has. If you would rather see the priority resolution split into its own class, I am happy to do that instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59655] Fill `priorityClassRef` and `priority` of Kueue `Workload` [spark-kubernetes-operator]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #851:
URL: https://github.com/apache/spark-kubernetes-operator/pull/851#discussion_r4052922179
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/kueue/KueueWorkloadUtils.java:
##########
@@ -124,9 +140,100 @@ public static AdmissionResult requestAdmission(
deleteWorkload(client, workload);
return AdmissionResult.STALE;
}
+ WorkloadSpec spec = workload.getSpec();
+ if (desired.getSpec().getPriority() != null
Review Comment:
**Finding 1.** This clause is what stops a permission blip from wiping a pending Workload's priority, and nothing pins it.
I deleted just `desired.getSpec().getPriority() != null` and ran the whole `spark-operator` suite: **it stays green**. Then, with a Workload already carrying `WorkloadPriorityClass high` (priority 1000) and the priority-class reads failing 403 on the next reconcile, the stored Workload came back `priority=null, priorityClassRef=null`. So the operator would hand it back to Kueue at the default priority 0 because it briefly could not read the class. The CEL rules allow that write, since presence is only frozen once quota is reserved, so nothing downstream catches it either.
This test fails on that mutant and passes as written:
```java
@Test
void pendingWorkloadKeepsItsPriorityWhenThePermissionIsLost() {
createWorkloadPriorityClass("high", 1000);
KueueWorkloadUtils.requestAdmission(kubernetesClient, workloadWithPriorityClass("high"));
Assertions.assertEquals(1000, getWorkload().getSpec().getPriority());
// The operator loses the RBAC rules while the Workload waits for quota
KubernetesClient forbiddenClient =
mock(KubernetesClient.class, withSettings().defaultAnswer(delegatesTo(kubernetesClient)));
KubernetesClientException forbidden = new KubernetesClientException("forbidden", 403, null);
doThrow(forbidden).when(forbiddenClient).resources(WorkloadPriorityClass.class);
doThrow(forbidden).when(forbiddenClient).scheduling();
Assertions.assertEquals(
AdmissionResult.PENDING,
KueueWorkloadUtils.requestAdmission(forbiddenClient, workloadWithPriorityClass("high")));
Assertions.assertEquals(1000, getWorkload().getSpec().getPriority());
Assertions.assertEquals("high", getWorkload().getSpec().getPriorityClassRef().getName());
}
```
It needs `org.mockito.AdditionalAnswers.delegatesTo`, `Mockito.doThrow` and `Mockito.withSettings`. `priorityIsNotSetWithoutPermission` covers the same 403 but only through `setPriority`, so it never reaches this branch.
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/kueue/KueueWorkloadUtils.java:
##########
@@ -76,16 +90,18 @@ public enum AdmissionResult {
* Creates the given Workload if it does not exist yet and reports whether Kueue admitted it.
*
* @param client The KubernetesClient.
- * @param desired The Workload built for the resource to be admitted. The pod sets hash annotation
- * is added to it in place.
+ * @param desired The Workload built for the resource to be admitted. The priority and the pod
+ * sets hash annotation are added to it in place.
* @return The AdmissionResult for the Workload.
- * @throws IllegalStateException if the Workload can neither be read nor created.
+ * @throws IllegalStateException if the Workload can neither be read nor created, or if its
+ * priority class does not exist.
* @throws KubernetesClientException if a stale Workload cannot be deleted. Unlike {@link
* #releaseWorkload}, this is not swallowed so that the resource is not created until the
* stale Workload is gone.
*/
public static AdmissionResult requestAdmission(
final KubernetesClient client, final Workload desired) {
+ setPriority(client, desired);
Review Comment:
**Finding 2.** Resolving here means every reconcile of a queued resource pays for it, including the ones where nothing changed.
I counted the mock-server requests for a steady-state `PENDING` reconcile with no priority class anywhere: **2 with this line, 1 without it**. The extra one is a cluster-wide `priorityclasses` LIST, because `getPodPriorityClass` falls through to the `globalDefault` lookup, and that is the default shape for a resource with no priority class at all. On the `STALE` path it repeats every 5 s.
Kueue's own integration gates exactly this, in `jobframework.updateWorkloadPriorities`:
```go
// Resolving only when at least one workload needs a transition keeps
// steady-state reconciles from re-resolving and overwriting the
// otherwise-mutable priority value.
if len(needsClassChange) == 0 {
return nil
}
```
`classifyWorkloadsForPriorityUpdate` decides that by comparing class *names*, which costs nothing. The same gate is available here: `KueueWorkloadFactory` copies the owner's labels onto the Workload (`spark-operator/src/main/java/org/apache/spark/k8s/operator/kueue/KueueWorkloadFactory.java:167`), so the stored Workload carries the same `kueue.x-k8s.io/priority-class` label, and its `priorityClassRef.name` is already in hand. So resolve only when the Workload has to be created, or when the existing `priorityClassRef` name differs from the label. That is also exactly the documented behaviour that "a changed value of the same class does not affect the existing `Workload`".
It does need the read to happen before the create, i.e. splitting `getOrCreateSecondaryResource` into a get and a create in this method.
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/kueue/KueueWorkloadUtils.java:
##########
@@ -56,6 +65,11 @@ public final class KueueWorkloadUtils {
*/
public static final Duration STALE_WORKLOAD_REQUEUE_INTERVAL = Duration.ofSeconds(5);
+ /** Like Kueue, the priority of a Workload without any priority class. */
+ private static final int DEFAULT_PRIORITY = 0;
+
+ private static final int HTTP_FORBIDDEN = 403;
Review Comment:
**Finding 5.** The JDK has this one, and the module static-imports it elsewhere: `ReconcilerUtils.java:27` for `HTTP_NOT_FOUND`, `SparkExceptionUtils.java:22` and `StatusRecorder.java:22` for `HTTP_CONFLICT`, and `ReadinessProbe.java:22` already pulls in `HTTP_FORBIDDEN` that way. `Constants.HTTP_TOO_MANY_REQUESTS` exists only because 429 has no JDK constant.
```java
import static java.net.HttpURLConnection.HTTP_FORBIDDEN;
```
and drop the field.
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/kueue/KueueWorkloadUtils.java:
##########
@@ -124,9 +140,100 @@ public static AdmissionResult requestAdmission(
deleteWorkload(client, workload);
return AdmissionResult.STALE;
}
+ WorkloadSpec spec = workload.getSpec();
+ if (desired.getSpec().getPriority() != null
+ && !Objects.equals(spec.getPriorityClassRef(), desired.getSpec().getPriorityClassRef())
+ && (workload.getStatus() == null || !workload.getStatus().isQuotaReserved())) {
+ // Like Kueue, a changed priority class is applied in place so that the Workload keeps its
+ // position in the queue. Kueue does not allow the change once the quota is reserved.
+ log.info(
+ "Updating the priority class of the pending Kueue Workload {}.",
+ workload.getMetadata().getName());
+ spec.setPriorityClassRef(desired.getSpec().getPriorityClassRef());
+ spec.setPriority(desired.getSpec().getPriority());
+ client.resource(workload).update();
+ }
return AdmissionResult.PENDING;
}
+ /**
+ * Sets the priority of the desired Workload in the same way as Kueue built-in integrations. The
+ * WorkloadPriorityClass of the `kueue.x-k8s.io/priority-class` label takes precedence over the
+ * PriorityClass of the first pod set which has one, and the global default PriorityClass is used
+ * without both. Without any of them, the priority is 0 without a priority class. The priority is
+ * left unset if the operator is not allowed to read the cluster-scoped priority classes.
+ *
+ * @param client The KubernetesClient.
+ * @param workload The Workload whose priority is set in place.
+ * @throws IllegalStateException if the priority class does not exist.
+ */
+ static void setPriority(final KubernetesClient client, final Workload workload) {
+ WorkloadSpec spec = workload.getSpec();
+ try {
+ Map<String, String> labels = workload.getMetadata().getLabels();
+ String workloadPriorityClassName =
+ labels == null ? null : labels.get(Constants.LABEL_WORKLOAD_PRIORITY_CLASS);
+ if (StringUtils.isNotEmpty(workloadPriorityClassName)) {
+ WorkloadPriorityClass workloadPriorityClass =
+ client.resources(WorkloadPriorityClass.class).withName(workloadPriorityClassName).get();
+ if (workloadPriorityClass == null) {
+ throw new IllegalStateException(
+ "Kueue WorkloadPriorityClass " + workloadPriorityClassName + " is not found.");
Review Comment:
**Finding 4.** A typo in the label leaves the resource stuck with nothing for the user to look at.
`AppInitStep.holdForKueueAdmission` catches `IllegalStateException`, logs a warning and requeues after 5 s. The first attempt's `Submitted` status is never persisted, so `kubectl get` shows an empty `Current State` and `kubectl describe` shows nothing. The result is a resource that retries forever, every 5 s, with the reason only in the operator log, for a plain user error that will not fix itself.
`docs/spark_custom_resources.md` is honest about it ("The resource keeps waiting without a `Workload` while the named priority class does not exist"), but that does not help someone working out why their application never starts.
The pod `PriorityClass` case at line 226 is the worse half, because it is unambiguously permanent: without Kueue the same spec reaches the driver pod and the Priority admission plugin rejects it, so the user gets a `SchedulingFailure` naming the class. Throwing `UnsupportedOperationException` there would restore that. `KueueWorkloadFactory.buildWorkload` already uses it for an unsupported spec and the init steps turn it into `SchedulingFailure`.
For the `WorkloadPriorityClass` case retrying is the right outcome, since an admin may still create the class, so that one needs a signal rather than a different outcome.
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/kueue/KueueWorkloadUtils.java:
##########
@@ -124,9 +140,100 @@ public static AdmissionResult requestAdmission(
deleteWorkload(client, workload);
return AdmissionResult.STALE;
}
+ WorkloadSpec spec = workload.getSpec();
+ if (desired.getSpec().getPriority() != null
+ && !Objects.equals(spec.getPriorityClassRef(), desired.getSpec().getPriorityClassRef())
+ && (workload.getStatus() == null || !workload.getStatus().isQuotaReserved())) {
+ // Like Kueue, a changed priority class is applied in place so that the Workload keeps its
+ // position in the queue. Kueue does not allow the change once the quota is reserved.
Review Comment:
**Finding 3.** "Kueue does not allow the change once the quota is reserved" is narrower than that, and the guard above is correspondingly stricter than Kueue.
The Workload CRD freezes four things while `QuotaReserved` is true (`apis/kueue/v1beta2/workload_types.go:1191-1194`): the *presence* of `priorityClassRef`, its `group`, its `kind`, and its `name` **only** for `scheduling.k8s.io/PriorityClass`. The `name` of a `WorkloadPriorityClass` and the `priority` value both stay mutable, and Kueue relies on that: `hasSameOrEmptyPriorityClass` returns true for two `WorkloadPriorityClass` refs with different names, so `updateWorkloadPriorities` goes ahead and writes.
The part of the guard that matters is real. A Workload that reserved quota with no ref can never be given one, which is the first skip in `classifyWorkloadsForPriorityUpdate`. What it also blocks is raising the `WorkloadPriorityClass` on a Workload waiting behind admission checks, which can take minutes with a `ProvisioningRequest`, and that is the case the label exists for.
Mirroring `hasSameOrEmptyPriorityClass` keeps the safe part:
```java
if (desired.getSpec().getPriority() != null
&& !Objects.equals(spec.getPriorityClassRef(), desired.getSpec().getPriorityClassRef())
&& (workload.getStatus() == null
|| !workload.getStatus().isQuotaReserved()
|| isSameGroupAndKind(
spec.getPriorityClassRef(), desired.getSpec().getPriorityClassRef()))) {
```
where the helper requires both refs present with equal `group` and `kind`, and equal `name` too when the group is `scheduling.k8s.io`. `quotaReservedWorkloadKeepsPriorityClass` then needs its two classes to differ in group or kind (or to go from a ref to none) so that it keeps testing the case that really is frozen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59655] Fill `priorityClassRef` and `priority` of Kueue `Workload` [spark-kubernetes-operator]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #851:
URL: https://github.com/apache/spark-kubernetes-operator/pull/851#discussion_r4057760272
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/kueue/KueueWorkloadUtils.java:
##########
@@ -228,6 +260,107 @@ private static void checkNoNodeSelectorConflict(
}
}
+ /**
+ * Checks whether the priority class of the given Workload is followed and whether Kueue accepts
+ * the change. Like Kueue's `classifyWorkloadsForPriorityUpdate`, only a Workload without a
+ * priority class or with a WorkloadPriorityClass follows the owner, so the priority of a Workload
+ * backed by a Kubernetes PriorityClass is never rewritten, e.g. when the global default changes.
+ * Once the quota is reserved, the Workload CEL rules freeze the presence, the group and the kind
+ * of the priorityClassRef. The name of a WorkloadPriorityClass stays mutable, which is what Kueue
+ * relies on to raise the priority of a Workload waiting for its admission checks.
+ */
+ private static boolean isPriorityClassChangeAllowed(
+ final Workload workload, final PriorityClassRef desired) {
+ PriorityClassRef current = workload.getSpec().getPriorityClassRef();
+ if (current != null && !Constants.KUEUE_API_GROUP.equals(current.getGroup())) {
+ return false;
+ }
+ if (workload.getStatus() == null || !workload.getStatus().isQuotaReserved()) {
Review Comment:
**Finding 9.** I re-ran the per-clause sweep on the rewritten guard. Five of the six clauses are killed — the Pod-`PriorityClass` skip, `current != null`, `desired != null`, the desired-group check, and the caller's `getPriority() != null`. This one survives.
Removing it is not a no-op. Without the short-circuit the guard falls straight to `current != null && …`, so a `Workload` that currently has **no** priority class can never gain one — including while it is merely pending. That is the case a user hits most: a queued resource created without the label, then labelled while it waits for quota. Every existing test starts from a `Workload` that already carries a `WorkloadPriorityClass` ref (`low` → `high`), and `quotaReservedWorkloadWithoutPriorityClassDoesNotGainOne` covers only the negative, quota-reserved side of the same transition.
This passes as written and fails with the clause removed — I ran both:
```java
@Test
void pendingWorkloadWithoutPriorityClassFollowsAnAddedLabel() {
createWorkloadPriorityClass("high", 1000);
// No label and no pod priority class, so the Workload is created without a ref
KueueWorkloadUtils.requestAdmission(kubernetesClient, workload("owner-uid-1", 1));
Assertions.assertNull(getWorkload().getSpec().getPriorityClassRef());
// The label is added while the Workload waits for quota
Assertions.assertEquals(
AdmissionResult.PENDING,
KueueWorkloadUtils.requestAdmission(kubernetesClient, workloadWithPriorityClass("high")));
Assertions.assertEquals("high", getWorkload().getSpec().getPriorityClassRef().getName());
Assertions.assertEquals(1000, getWorkload().getSpec().getPriority());
}
```
It also documents the nil-to-ref direction, which is currently only asserted in its blocked form.
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/kueue/KueueWorkloadUtils.java:
##########
@@ -40,21 +42,29 @@
import com.fasterxml.jackson.databind.SerializationFeature;
import io.fabric8.kubernetes.api.model.HasMetadata;
import io.fabric8.kubernetes.api.model.OwnerReference;
+import io.fabric8.kubernetes.api.model.PodTemplateSpec;
import io.fabric8.kubernetes.api.model.Toleration;
+import io.fabric8.kubernetes.api.model.scheduling.v1.PriorityClass;
import io.fabric8.kubernetes.client.KubernetesClient;
import io.fabric8.kubernetes.client.KubernetesClientException;
import lombok.extern.slf4j.Slf4j;
+import org.apache.spark.k8s.operator.Constants;
import org.apache.spark.k8s.operator.kueue.v1beta2.PodSet;
import org.apache.spark.k8s.operator.kueue.v1beta2.PodSetAssignment;
+import org.apache.spark.k8s.operator.kueue.v1beta2.PriorityClassRef;
import org.apache.spark.k8s.operator.kueue.v1beta2.ResourceFlavor;
import org.apache.spark.k8s.operator.kueue.v1beta2.Workload;
+import org.apache.spark.k8s.operator.kueue.v1beta2.WorkloadPriorityClass;
+import org.apache.spark.k8s.operator.kueue.v1beta2.WorkloadSpec;
import org.apache.spark.k8s.operator.kueue.v1beta2.WorkloadStatus;
import org.apache.spark.k8s.operator.utils.ModelUtils;
import org.apache.spark.k8s.operator.utils.ReconcilerUtils;
+import org.apache.spark.k8s.operator.utils.StringUtils;
/** Utilities to create, check and release Kueue Workloads. */
@Slf4j
+@SuppressWarnings("PMD.GodClass")
Review Comment:
**Finding 10.** Answering the question at the end of [your comment](https://github.com/apache/spark-kubernetes-operator/pull/851#issuecomment-5751418533): yes, I would take the split.
The numbers argue for it rather than against. `pmdMain` is green on `main` for this class, and with this PR it reports:
```
KueueWorkloadUtils.java:67: GodClass: Possible God Class (WMC=61, ATFD=97, TCC=3.297%)
```
That is 14 over the threshold of 47, not one or two, and `TCC=3.297%` says the methods share almost no state — the class is now three unrelated jobs: the `Workload` lifecycle (`requestAdmission`, `releaseWorkload`, `isAdmitted`, `hashPodSets`), flavor resolution from #850, and priority resolution from this PR. #847 adds `holdForAdmission` on top of that.
I tried the cut. Moving `setPriority`, `getPodPriorityClass` and `isPriorityClassChangeAllowed` into a `KueueWorkloadPriority` class in the same package, with `requestAdmission` calling `KueueWorkloadPriority.setPriority(...)` and `KueueWorkloadPriority.isPriorityClassChangeAllowed(...)`, makes `pmdMain` pass with **no suppression on either class** — the only fallout was the imports left behind in `KueueWorkloadUtils`, which have to move too. It also gives the priority tests an obvious home, mirroring how #850 put `KueuePodSetFlavor` beside the resolver.
The `KueueWorkloadFactory` precedent is real, so suppressing is not unreasonable. The difference is that this class trips the rule because of the change in front of us, and the cut that fixes it is small and available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59655] Fill `priorityClassRef` and `priority` of Kueue `Workload` [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #851:
URL: https://github.com/apache/spark-kubernetes-operator/pull/851#issuecomment-5752251419
Thank you so much, @peter-toth !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59655] Fill `priorityClassRef` and `priority` of Kueue `Workload` [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #851: [SPARK-59655] Fill `priorityClassRef` and `priority` of Kueue `Workload`
URL: https://github.com/apache/spark-kubernetes-operator/pull/851
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org