You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/09/27 21:03:34 UTC
(spark-kubernetes-operator) branch main updated: [SPARK-59803] Handle an evicted or deactivated Kueue `Workload` of a running `SparkCluster`
This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 5cfbd9ce [SPARK-59803] Handle an evicted or deactivated Kueue `Workload` of a running `SparkCluster`
5cfbd9ce is described below
commit 5cfbd9cef98425e4ee62dd55e67e74236eddd5d5
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Sun Sep 27 14:03:27 2026 -0700
[SPARK-59803] Handle an evicted or deactivated Kueue `Workload` of a running `SparkCluster`
### What changes were proposed in this pull request?
This PR aims to handle an evicted or deactivated Kueue `Workload` of a running `SparkCluster`, like SPARK-59800 does before the master is requested.
- An evicted cluster enters `Suspended` and releases its master and workers like `spec.suspend`. Once the pods are gone and Kueue's requeue backoff elapses, its `Workload` is deleted and the cluster is queued again with a new `Workload`.
- A deactivated `Workload` is kept, so the cluster stays `Suspended` until the `Workload` is reactivated.
- A `PodsReadyTimeout` eviction is only reported by a new `KueueEvictionIgnored` warning event, since the operator does not report `PodsReady`.
### Why are the changes needed?
Currently, a running `SparkCluster` ignores a Kueue eviction. Kueue keeps the quota of an evicted `Workload` until its integration releases it, so a preemption never completes. In contrast, a deactivated `Workload` is no longer counted, so the cluster runs outside of the quota.
### Does this PR introduce _any_ user-facing change?
No for the released versions, because the Kueue integration is not released yet in v1.0.0.
### How was this patch tested?
Pass the CIs with the newly added test cases.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #893 from dongjoon-hyun/SPARK-59803.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
docs/architecture.md | 13 +-
docs/config_properties.md | 2 +-
docs/configuration.md | 1 +
docs/spark_custom_resources.md | 27 ++-
.../org/apache/spark/k8s/operator/Constants.java | 13 +
.../k8s/operator/status/ClusterStateSummary.java | 6 +-
.../k8s/operator/config/SparkOperatorConf.java | 5 +-
.../k8s/operator/kueue/KueueWorkloadUtils.java | 21 ++
.../reconcilesteps/ClusterSuspendStep.java | 131 +++++++++-
.../spark/k8s/operator/utils/EventUtils.java | 6 +
.../k8s/operator/kueue/KueueWorkloadUtilsTest.java | 29 +++
.../reconcilesteps/ClusterSuspendStepTest.java | 267 +++++++++++++++++++++
tests/e2e/kueue/chainsaw-test.yaml | 141 +++++++++++
.../e2e/kueue/spark-cluster-example-preempted.yaml | 43 ++++
14 files changed, 683 insertions(+), 22 deletions(-)
diff --git a/docs/architecture.md b/docs/architecture.md
index 1378c73c..1594b741 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -201,8 +201,8 @@ stateDiagram-v2
SchedulingFailure --> Failed
- RunningHealthy --> Suspended : spec.suspend=true
- Suspended --> Submitted : spec.suspend=false
+ RunningHealthy --> Suspended : spec.suspend=true or Kueue eviction
+ Suspended --> Submitted : spec.suspend=false or after Kueue eviction
Submitted --> Suspended : spec.suspend=true after resume
RunningHealthy --> [*]
@@ -211,9 +211,12 @@ stateDiagram-v2
```
* Spark clusters are expected to be always running after submitted, until they are suspended by
- [`spec.suspend`](spark_custom_resources.md#suspend).
-* Apart from `spec.suspend`, a cluster leaves `RunningHealthy`, `Suspended` or `Failed` only when
- its custom resource is deleted. At that point, the K8s resources created for the cluster are
+ [`spec.suspend`](spark_custom_resources.md#suspend). A cluster queued by
+ [Kueue](spark_custom_resources.md#kueue) is suspended as well when Kueue evicts its `Workload`,
+ and moves to `Submitted` once its master and workers are released, or once its `Workload` is
+ reactivated if it was deactivated.
+* Apart from `spec.suspend` and a Kueue eviction, a cluster leaves `RunningHealthy`, `Suspended` or
+ `Failed` only when its custom resource is deleted. At that point, the K8s resources created for the cluster are
garbage collected through their `ownerReference` to the `SparkCluster` custom resource.
* A `Failed` cluster is not reconciled any further.
* `ResourceReleased` exists in the API enum but is currently not used for clusters.
diff --git a/docs/config_properties.md b/docs/config_properties.md
index e2c0b471..1739e4c6 100644
--- a/docs/config_properties.md
+++ b/docs/config_properties.md
@@ -16,7 +16,7 @@
| spark.kubernetes.operator.dynamicConfig.reloadIntervalSeconds | Long | 60 | false | Interval (in seconds) at which the dynamic config file is re-read. Used by the 'file' source. |
| spark.kubernetes.operator.dynamicConfig.selector | String | app.kubernetes.io/name=spark-kubernetes-operator,app.kubernetes.io/component=operator-dynamic-config-overrides | false | The selector str applied to dynamic config map. Used by the 'configMap' source. |
| spark.kubernetes.operator.dynamicConfig.source | String | configMap | false | Source of dynamic config overrides when spark.kubernetes.operator.dynamicConfig.enabled is true. Supported values: 'configMap' (default) watches a ConfigMap via a Kubernetes informer and requires RBAC to read ConfigMaps; 'file' periodically reloads a properties file mounted from a ConfigMap and requires no extra RBAC. |
- | spark.kubernetes.operator.events.enabled | Boolean | false | true | When enabled, operator would publish Kubernetes Event objects into the namespace of the Spark resource they describe, making them visible to users of that namespace via 'kubectl describe' and 'kubectl get events'. Events are published when a resource transitions into a new state, as Warning for failure states and Normal for most other states, when the operator cannot reconcile or update the status of a resource, when [...]
+ | spark.kubernetes.operator.events.enabled | Boolean | false | true | When enabled, operator would publish Kubernetes Event objects into the namespace of the Spark resource they describe, making them visible to users of that namespace via 'kubectl describe' and 'kubectl get events'. Events are published when a resource transitions into a new state, as Warning for failure states and Normal for most other states, when the operator cannot reconcile or update the status of a resource, when [...]
| spark.kubernetes.operator.events.excludedReasons | String | | true | Comma-separated list of Java regular expressions of Kubernetes Event reasons that operator would not publish even when 'spark.kubernetes.operator.events.enabled' is true, e.g. 'RunningHealthy,RunningWithPartialCapacity' or 'Running.*'. Each expression must match the whole reason and is case-sensitive. An invalid expression matches only the reason identical to it. If empty, operator would publish events of all reasons. |
| spark.kubernetes.operator.events.minIntervalSeconds | Long | 300 | true | Minimum interval (in seconds) between two Kubernetes Events with the same reason and the same message on the same Spark resource. The first event of a reason on a resource is always published right away, only a repeat that says exactly what the last one said is dropped. Such a repeat only bumps the 'count' of the one Event object, so it carries no information while it still costs a read and a write on the API se [...]
| spark.kubernetes.operator.health.probePort | Integer | 19091 | false | The port used for health/readiness check probe status. |
diff --git a/docs/configuration.md b/docs/configuration.md
index d93ee9a8..7f534761 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -81,6 +81,7 @@ In addition, the operator publishes the following `Warning` events.
| `SuspendReleaseFailed` | Deleting the master and worker StatefulSets, the HorizontalPodAutoscaler or PodDisruptionBudget of the workers, or the Kueue `Workload` of a `SparkCluster` suspended while running fails, or its pods cannot be listed. A failure which may clear on its own, such as a timeout, is skipped. Every failure is retried with the default interval. |
| `KueueResourceFlavorReadFailed` | Reading the `ResourceFlavor`s which Kueue assigned to an admitted `Workload` fails, so the node selector and the tolerations of the flavors cannot be applied. Transport-level errors are skipped and retried every 5 seconds, while a persistent failure is retried with the default interval. |
| `KueueDisabled` | A `SparkApplication` or `SparkCluster` labeled with `kueue.x-k8s.io/queue-name` is not queued, since `spark.kubernetes.operator.kueue.enabled` is disabled, so the driver (or master and worker) is requested without the Kueue admission. |
+| `KueueEvictionIgnored` | Kueue evicted the `Workload` of a running `SparkCluster` by a `PodsReadyTimeout`, which the operator does not act on, so the master and workers keep running and holding its quota. It is republished on every reconciliation while the eviction lasts, subject to [`minIntervalSeconds`](#event-frequency). See [Kueue](spark_custom_resources.md#kueue). |
For a resource held by [`spec.suspend`](spark_custom_resources.md#suspend) or queued by
[Kueue](spark_custom_resources.md#kueue), the operator also publishes the following `Normal`
diff --git a/docs/spark_custom_resources.md b/docs/spark_custom_resources.md
index 5127b37e..ccfcc2b1 100644
--- a/docs/spark_custom_resources.md
+++ b/docs/spark_custom_resources.md
@@ -663,7 +663,8 @@ spec:
* A `SparkCluster` requests the resources set on the `master` and `worker` containers of its pod
templates. A missing request defaults to the limit, or else to 1 CPU and `SPARK_DAEMON_MEMORY`
plus overhead. A worker uses `SPARK_WORKER_CORES` for the CPU and adds `SPARK_WORKER_MEMORY` to
- the memory when they are set. A `SparkCluster` keeps the quota until it is deleted or suspended.
+ the memory when they are set. A `SparkCluster` keeps the quota until it is deleted, suspended or
+ evicted.
Set a cpu and memory request or limit on the `worker` container, or `SPARK_WORKER_CORES` and
`SPARK_WORKER_MEMORY`: a worker with none of them advertises the whole node to its executors,
well above the default the `Workload` requests.
@@ -739,10 +740,26 @@ spec:
with a new `Workload`. Unlike Kueue built-in integrations, which keep the evicted
`Workload`, the new `Workload` is ordered in its queue like a newly created one, and it starts
over the retry counts of its admission checks.
-* Preemption is not honored yet after the driver (or master) is requested. The operator checks the
- admission only before it creates the resources, so after a later eviction the driver and
- executors, or the master and workers, keep running. An evicted `Workload` keeps its Kueue quota
- meanwhile, while a deactivated one no longer counts against it.
+* A running `SparkCluster` whose `Workload` Kueue evicts, e.g. to preempt it for a workload of a
+ higher priority or since its `ClusterQueue` is stopped, is released like on `spec.suspend`
+ without setting it. The cluster enters `Suspended` with a message giving the reason of the
+ eviction, and the operator deletes its master and worker StatefulSets. Kueue keeps the quota of
+ an evicted `Workload` until then, so the operator deletes the `Workload` only after the master
+ and worker pods are gone, like on `spec.suspend`, and after its requeue backoff elapsed, like
+ above. The cluster then moves to `Submitted` and is queued again with a new `Workload`. The
+ Workload informer reconciles the cluster as soon as its `Workload` is evicted.
+* A running `SparkCluster` whose `Workload` is deactivated is released as well, since Kueue stops
+ counting its quota right away. Unlike an evicted one, the operator keeps the `Workload`, so the
+ cluster stays `Suspended` until the `Workload` is reactivated, and then it is queued again.
+* A `PodsReadyTimeout` of the `waitForPodsReady` configuration is not acted on for a running
+ `SparkCluster`, and the `KueueEvictionIgnored` [event](configuration.md#kubernetes-events) is
+ published instead, since the operator does not report the `PodsReady` condition, so it would
+ recur on every attempt. Kueue keeps counting the quota of such a `Workload`, so suspend or delete
+ the cluster to release it.
+* Preemption is not honored yet after the driver of a `SparkApplication` is requested. The operator
+ checks the admission only before it creates the resources, so after a later eviction the driver
+ and executors keep running. An evicted `Workload` keeps its Kueue quota meanwhile, while a
+ deactivated one no longer counts against it.
## Spark Cluster
diff --git a/spark-operator-api/src/main/java/org/apache/spark/k8s/operator/Constants.java b/spark-operator-api/src/main/java/org/apache/spark/k8s/operator/Constants.java
index 5db67ef1..2c6c32df 100644
--- a/spark-operator-api/src/main/java/org/apache/spark/k8s/operator/Constants.java
+++ b/spark-operator-api/src/main/java/org/apache/spark/k8s/operator/Constants.java
@@ -260,6 +260,19 @@ public final class Constants {
public static final String CLUSTER_RESUMED_MESSAGE =
"Cluster is resumed as spec.suspend is set to false.";
+ /**
+ * Message indicating that the running cluster is evicted by Kueue, which is followed by the
+ * reason and the message of the eviction.
+ */
+ public static final String CLUSTER_EVICTED_MESSAGE =
+ "Cluster is evicted by Kueue, so its master and workers are being released before its "
+ + "Workload. It is queued again then, or once the Workload is reactivated if it was "
+ + "deactivated.";
+
+ /** Message indicating that the cluster evicted by Kueue is submitted again. */
+ public static final String CLUSTER_REQUEUED_MESSAGE =
+ "Cluster is submitted again after its eviction by Kueue.";
+
/** Message indicating that the cluster status cannot be processed. */
public static final String UNKNOWN_CLUSTER_STATE_MESSAGE = "Cannot process cluster status.";
diff --git a/spark-operator-api/src/main/java/org/apache/spark/k8s/operator/status/ClusterStateSummary.java b/spark-operator-api/src/main/java/org/apache/spark/k8s/operator/status/ClusterStateSummary.java
index b77ee1f6..d37b5451 100644
--- a/spark-operator-api/src/main/java/org/apache/spark/k8s/operator/status/ClusterStateSummary.java
+++ b/spark-operator-api/src/main/java/org/apache/spark/k8s/operator/status/ClusterStateSummary.java
@@ -41,9 +41,9 @@ public enum ClusterStateSummary implements BaseStateSummary {
ResourceReleased,
/**
- * Cluster is suspended by spec.suspend after it started running, so its master and workers are
- * released until it is resumed from Submitted. It is declared last, since {@link #isStarting()}
- * compares the ordinals.
+ * Cluster is suspended by spec.suspend, or by the eviction of its Kueue Workload, after it
+ * started running, so its master and workers are released until it is resumed from Submitted.
+ * It is declared last, since {@link #isStarting()} compares the ordinals.
*/
Suspended;
diff --git a/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java b/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java
index 1391a7b4..c46f5093 100644
--- a/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java
+++ b/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java
@@ -335,8 +335,9 @@ public final class SparkOperatorConf {
+ "failure states and Normal for most other states, when the operator "
+ "cannot reconcile or update the status of a resource, when a resource is "
+ "held by spec.suspend, when a resource queued by Kueue waits for, gets or "
- + "fails to request the admission, and when the Kueue queue name label of a "
- + "resource is ignored. These are Event resources in the core "
+ + "fails to request the admission, when the Kueue queue name label of a "
+ + "resource is ignored, and when a Kueue eviction of a running resource is not "
+ + "acted on. These are Event resources in the core "
+ "API group, unrelated to the internal events that trigger reconciliation.")
.typeParameterClass(Boolean.class)
.defaultValue(false)
diff --git a/spark-operator/src/main/java/org/apache/spark/k8s/operator/kueue/KueueWorkloadUtils.java b/spark-operator/src/main/java/org/apache/spark/k8s/operator/kueue/KueueWorkloadUtils.java
index 972e3b39..54b2b618 100644
--- a/spark-operator/src/main/java/org/apache/spark/k8s/operator/kueue/KueueWorkloadUtils.java
+++ b/spark-operator/src/main/java/org/apache/spark/k8s/operator/kueue/KueueWorkloadUtils.java
@@ -83,6 +83,9 @@ public final class KueueWorkloadUtils {
/** Type of the Kueue condition which marks an evicted Workload. */
private static final String CONDITION_EVICTED = "Evicted";
+ /** Reason of the eviction of a Workload whose pods are not ready within the Kueue timeout. */
+ private static final String EVICTED_BY_PODS_READY_TIMEOUT = "PodsReadyTimeout";
+
/**
* Requeue interval after {@link AdmissionResult#STALE} and after a transient API failure. It is
* short because both go away shortly, while an unchanged admission and a persistent failure are
@@ -686,6 +689,24 @@ public final class KueueWorkloadUtils {
.findFirst();
}
+ /**
+ * Checks whether the running resources of the given evicted Workload are kept rather than
+ * released. That is the case for a `PodsReadyTimeout` of an active Workload only, since the
+ * operator does not report the `PodsReady` condition, so every attempt would be evicted again,
+ * while Kueue keeps counting the quota of the Workload. Any other eviction releases the
+ * resources, e.g. to preempt them or since the ClusterQueue is stopped. That includes a
+ * deactivation, since Kueue stops counting the quota of a deactivated Workload right away.
+ *
+ * @param workload The Workload to check.
+ * @return True if the Workload is evicted and its resources keep running.
+ */
+ public static boolean isKeptOnEviction(final Workload workload) {
+ Optional<Condition> eviction = findEviction(workload);
+ return eviction.isPresent()
+ && !isDeactivated(workload)
+ && EVICTED_BY_PODS_READY_TIMEOUT.equals(eviction.get().getReason());
+ }
+
/**
* Checks whether the given Workload is deactivated by `spec.active` set to false, e.g. by a user
* or by Kueue after too many retries. Kueue neither counts nor admits it until it is reactivated.
diff --git a/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStep.java b/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStep.java
index 33658581..49863190 100644
--- a/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStep.java
+++ b/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStep.java
@@ -19,6 +19,8 @@
package org.apache.spark.k8s.operator.reconciler.reconcilesteps;
+import static org.apache.spark.k8s.operator.Constants.CLUSTER_EVICTED_MESSAGE;
+import static org.apache.spark.k8s.operator.Constants.CLUSTER_REQUEUED_MESSAGE;
import static org.apache.spark.k8s.operator.Constants.CLUSTER_RESUMED_MESSAGE;
import static org.apache.spark.k8s.operator.Constants.CLUSTER_SUSPENDED_MESSAGE;
import static org.apache.spark.k8s.operator.Constants.CLUSTER_SUSPENDED_WITH_STUCK_PODS_MESSAGE;
@@ -39,9 +41,11 @@ import java.time.Duration;
import java.time.Instant;
import java.util.Comparator;
import java.util.List;
+import java.util.NavigableMap;
import java.util.Optional;
import java.util.stream.Collectors;
+import io.fabric8.kubernetes.api.model.Condition;
import io.fabric8.kubernetes.api.model.Pod;
import io.fabric8.kubernetes.client.KubernetesClient;
import io.fabric8.kubernetes.client.KubernetesClientException;
@@ -50,6 +54,7 @@ import lombok.extern.slf4j.Slf4j;
import org.apache.spark.k8s.operator.SparkCluster;
import org.apache.spark.k8s.operator.context.SparkClusterContext;
import org.apache.spark.k8s.operator.kueue.KueueWorkloadUtils;
+import org.apache.spark.k8s.operator.kueue.v1beta2.Workload;
import org.apache.spark.k8s.operator.reconciler.ReconcileProgress;
import org.apache.spark.k8s.operator.status.ClusterState;
import org.apache.spark.k8s.operator.status.ClusterStateSummary;
@@ -68,6 +73,9 @@ import org.apache.spark.k8s.operator.utils.SparkClusterStatusRecorder;
* Submitted, where {@link ClusterInitStep} applies all of its resources again and requests a new
* Kueue admission. Resuming earlier would let it keep the StatefulSets or the admitted Workload of
* the previous spec, so that a spec changed in the meantime would run on the old admission.
+ *
+ * <p>A running cluster whose Kueue Workload is evicted, e.g. to preempt it, is suspended the same
+ * way without spec.suspend, so that it is queued again once everything is released.
*/
@Slf4j
public final class ClusterSuspendStep extends ClusterReconcileStep {
@@ -80,7 +88,8 @@ public final class ClusterSuspendStep extends ClusterReconcileStep {
private static final Duration POD_RELEASE_TIMEOUT = Duration.ofMinutes(5);
/**
- * Reconciles a running or suspended cluster against spec.suspend.
+ * Reconciles a running or suspended cluster against spec.suspend and the eviction of its Kueue
+ * Workload.
*
* @param context The SparkClusterContext for the cluster.
* @param statusRecorder The SparkClusterStatusRecorder for recording status updates.
@@ -98,12 +107,30 @@ public final class ClusterSuspendStep extends ClusterReconcileStep {
statusRecorder,
new ClusterState(ClusterStateSummary.Suspended, CLUSTER_SUSPENDED_MESSAGE));
}
+ if (summary == ClusterStateSummary.RunningHealthy) {
+ return suspendOnKueueEviction(context, statusRecorder);
+ }
if (summary == ClusterStateSummary.Suspended) {
- Optional<ReconcileProgress> waiting = releaseResources(context, statusRecorder);
+ boolean evicted = isSuspendedByEviction(cluster);
+ if (evicted && suspend) {
+ // The cluster is held by spec.suspend from now on, so that its status no longer says that
+ // it is queued again, and a later resume is reported as such.
+ return appendStateAndImmediateRequeue(
+ context,
+ statusRecorder,
+ new ClusterState(ClusterStateSummary.Suspended, CLUSTER_SUSPENDED_MESSAGE));
+ }
+ Optional<Duration> keepWorkload = evicted ? keepKueueWorkload(context) : Optional.empty();
+ Optional<ReconcileProgress> waiting =
+ releaseResources(context, statusRecorder, keepWorkload.isEmpty());
if (waiting.isPresent()) {
return waiting.get();
}
+ if (keepWorkload.isPresent()) {
+ return completeAndRequeueAfter(keepWorkload.get());
+ }
if (!suspend) {
+ String message = evicted ? CLUSTER_REQUEUED_MESSAGE : CLUSTER_RESUMED_MESSAGE;
// Like a restarted application attempt, the resumed cluster drops the history of the run
// which is over, so that suspending and resuming it again and again keeps it bounded.
return updateStatusAndRequeueAfter(
@@ -112,7 +139,7 @@ public final class ClusterSuspendStep extends ClusterReconcileStep {
cluster
.getStatus()
.appendNewState(
- new ClusterState(ClusterStateSummary.Submitted, CLUSTER_RESUMED_MESSAGE),
+ new ClusterState(ClusterStateSummary.Submitted, message),
TRIM_ATTEMPT_STATE_TRANSITION_HISTORY.getValue()),
Duration.ZERO);
}
@@ -122,6 +149,93 @@ public final class ClusterSuspendStep extends ClusterReconcileStep {
return proceed();
}
+ /**
+ * Suspends a running cluster whose Kueue Workload is evicted, so that its master and workers and
+ * then its Workload are released like on spec.suspend. Kueue keeps the quota of an evicted
+ * Workload until then. As spec.suspend is not set, the cluster is queued again once everything is
+ * released, see {@link #keepKueueWorkload} for when that waits. An eviction which {@link
+ * KueueWorkloadUtils#isKeptOnEviction} is only reported, and the cluster keeps running.
+ *
+ * @param context The SparkClusterContext for the cluster.
+ * @param statusRecorder The SparkClusterStatusRecorder for recording status updates.
+ * @return The ReconcileProgress indicating the next step.
+ */
+ private ReconcileProgress suspendOnKueueEviction(
+ SparkClusterContext context, SparkClusterStatusRecorder statusRecorder) {
+ Optional<Workload> cached = context.getCachedKueueWorkload();
+ Optional<Condition> eviction = cached.flatMap(KueueWorkloadUtils::findEviction);
+ if (eviction.isEmpty()) {
+ return proceed();
+ }
+ Workload workload = cached.get();
+ String cause = eviction.get().getReason() + ": " + eviction.get().getMessage();
+ if (KueueWorkloadUtils.isKeptOnEviction(workload)) {
+ EventUtils.warn(
+ context.getEventRecorder(),
+ EventUtils.REASON_KUEUE_EVICTION_IGNORED,
+ "Kueue evicted Workload "
+ + workload.getMetadata().getName()
+ + " ("
+ + cause
+ + "), which the operator does not act on, so the master and workers keep running "
+ + "and holding its quota. Set spec.suspend to true or delete the cluster to release "
+ + "them.");
+ return proceed();
+ }
+ log.info("Kueue evicted the Workload of the cluster ({}), suspending it.", cause);
+ return appendStateAndImmediateRequeue(
+ context,
+ statusRecorder,
+ new ClusterState(ClusterStateSummary.Suspended, CLUSTER_EVICTED_MESSAGE + " " + cause));
+ }
+
+ /**
+ * Checks whether the cluster is suspended by the eviction of its Kueue Workload rather than by
+ * spec.suspend. The Suspended state which names its stuck pods, if any, follows the one which
+ * says why it is suspended.
+ *
+ * @param cluster The suspended SparkCluster.
+ * @return True if the cluster was suspended by an eviction and not held by spec.suspend since.
+ */
+ private static boolean isSuspendedByEviction(SparkCluster cluster) {
+ NavigableMap<Long, ClusterState> history =
+ (NavigableMap<Long, ClusterState>) cluster.getStatus().getStateTransitionHistory();
+ for (ClusterState state : history.descendingMap().values()) {
+ String message = state.getMessage();
+ if (state.getCurrentStateSummary() != ClusterStateSummary.Suspended
+ || CLUSTER_SUSPENDED_MESSAGE.equals(message)) {
+ return false;
+ }
+ if (message != null && message.startsWith(CLUSTER_EVICTED_MESSAGE)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Returns how long the Kueue Workload of a cluster which is suspended without spec.suspend is
+ * kept after its master and workers are released, or empty to release it right away and queue
+ * the cluster again. A deactivated Workload is kept until it is reactivated, which the Workload
+ * informer reconciles right away: Kueue no longer counts its quota, and a new Workload of the
+ * cluster would come back active. An evicted Workload is kept until the requeue backoff which
+ * Kueue records on it elapses, see {@link KueueWorkloadUtils#remainingRequeueBackoff}.
+ *
+ * @param context The SparkClusterContext for the cluster.
+ * @return The requeue interval while the Workload is kept, or empty to release it.
+ */
+ private static Optional<Duration> keepKueueWorkload(SparkClusterContext context) {
+ Optional<Workload> workload = context.getCachedKueueWorkload();
+ if (workload.isEmpty()) {
+ return Optional.empty();
+ }
+ if (KueueWorkloadUtils.isDeactivated(workload.get())) {
+ return Optional.of(Duration.ofSeconds(SUSPEND_HOLD_REQUEUE_INTERVAL_SECONDS.getValue()));
+ }
+ Duration backoff = KueueWorkloadUtils.remainingRequeueBackoff(workload.get());
+ return backoff.isZero() ? Optional.empty() : Optional.of(backoff);
+ }
+
/**
* Releases the master and workers of a suspended cluster, and then its Kueue Workload. The
* Services and the NetworkPolicy are kept, since they hold no quota and are applied again on
@@ -149,11 +263,14 @@ public final class ClusterSuspendStep extends ClusterReconcileStep {
*
* @param context The SparkClusterContext for the cluster.
* @param statusRecorder The SparkClusterStatusRecorder for recording status updates.
+ * @param releaseWorkload Whether the Kueue Workload is released as well.
* @return Empty once everything is released, or the ReconcileProgress to retry with while pods
* remain or a release failed.
*/
private Optional<ReconcileProgress> releaseResources(
- SparkClusterContext context, SparkClusterStatusRecorder statusRecorder) {
+ SparkClusterContext context,
+ SparkClusterStatusRecorder statusRecorder,
+ boolean releaseWorkload) {
SparkCluster cluster = context.getResource();
String namespace = cluster.getMetadata().getNamespace();
String name = cluster.getMetadata().getName();
@@ -212,8 +329,10 @@ public final class ClusterSuspendStep extends ClusterReconcileStep {
? completeAndRequeueAfter(remaining)
: defaultRequeue);
}
- // A Workload admitted before the queue label was removed is released as well.
- KueueWorkloadUtils.deleteWorkloadOf(client, cluster);
+ if (releaseWorkload) {
+ // A Workload admitted before the queue label was removed is released as well.
+ KueueWorkloadUtils.deleteWorkloadOf(client, cluster);
+ }
if (!pods.isEmpty()) {
log.debug("Waiting for the stuck pods of the suspended cluster to be deleted.");
// This may last until a lost node is deleted, so the state says why, once per set of pods
diff --git a/spark-operator/src/main/java/org/apache/spark/k8s/operator/utils/EventUtils.java b/spark-operator/src/main/java/org/apache/spark/k8s/operator/utils/EventUtils.java
index 00371da9..2a43625e 100644
--- a/spark-operator/src/main/java/org/apache/spark/k8s/operator/utils/EventUtils.java
+++ b/spark-operator/src/main/java/org/apache/spark/k8s/operator/utils/EventUtils.java
@@ -83,6 +83,12 @@ public final class EventUtils {
*/
public static final String REASON_KUEUE_DISABLED = "KueueDisabled";
+ /**
+ * Reason for an event describing that Kueue evicted the Workload of a running resource in a way
+ * which the operator does not act on, so the resource keeps running.
+ */
+ public static final String REASON_KUEUE_EVICTION_IGNORED = "KueueEvictionIgnored";
+
/** Maximum number of characters an event message may have, including the ellipsis. */
static final int MAX_MESSAGE_LENGTH = 1024;
diff --git a/spark-operator/src/test/java/org/apache/spark/k8s/operator/kueue/KueueWorkloadUtilsTest.java b/spark-operator/src/test/java/org/apache/spark/k8s/operator/kueue/KueueWorkloadUtilsTest.java
index 1d829e7b..5da226f0 100644
--- a/spark-operator/src/test/java/org/apache/spark/k8s/operator/kueue/KueueWorkloadUtilsTest.java
+++ b/spark-operator/src/test/java/org/apache/spark/k8s/operator/kueue/KueueWorkloadUtilsTest.java
@@ -375,6 +375,35 @@ class KueueWorkloadUtilsTest {
Assertions.assertTrue(KueueWorkloadUtils.isDeactivated(workload));
}
+ @Test
+ void evictionKeepingTheResourcesIsDetected() {
+ Workload admitted = workload("owner-uid-1", 1);
+ admitted.setStatus(status("Admitted", "True"));
+ Assertions.assertFalse(KueueWorkloadUtils.isKeptOnEviction(admitted));
+
+ for (String reason :
+ List.of(
+ "Preempted",
+ "NodeFailures",
+ "ClusterQueueStopped",
+ "LocalQueueStopped",
+ "AdmissionCheck",
+ "FlavorMigration",
+ "Deactivated")) {
+ Workload evicted = workload("owner-uid-1", 1);
+ evicted.setStatus(evictedStatus(reason));
+ Assertions.assertFalse(KueueWorkloadUtils.isKeptOnEviction(evicted), reason);
+ }
+
+ // The operator does not report PodsReady, so every admitted Workload would time out again
+ Workload podsReadyTimeout = workload("owner-uid-1", 1);
+ podsReadyTimeout.setStatus(evictedStatus("PodsReadyTimeout"));
+ Assertions.assertTrue(KueueWorkloadUtils.isKeptOnEviction(podsReadyTimeout));
+ // Unless it is deactivated, e.g. after too many retries, since Kueue stops counting its quota
+ podsReadyTimeout.getSpec().setActive(false);
+ Assertions.assertFalse(KueueWorkloadUtils.isKeptOnEviction(podsReadyTimeout));
+ }
+
@ParameterizedTest
@ValueSource(strings = {"Preempted", "PodsReadyTimeout"})
void evictedWorkloadIsRecreated(String reason) {
diff --git a/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStepTest.java b/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStepTest.java
index 82cf989a..f33ea965 100644
--- a/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStepTest.java
+++ b/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStepTest.java
@@ -31,9 +31,12 @@ import static org.mockito.Mockito.when;
import java.time.Duration;
import java.time.Instant;
+import java.util.List;
import java.util.Map;
+import java.util.Optional;
import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import io.fabric8.kubernetes.api.model.ConditionBuilder;
import io.fabric8.kubernetes.api.model.KubernetesResourceList;
import io.fabric8.kubernetes.api.model.ObjectMetaBuilder;
import io.fabric8.kubernetes.api.model.Pod;
@@ -64,6 +67,7 @@ import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.CsvSource;
import org.junit.jupiter.params.provider.ValueSource;
import org.mockito.ArgumentCaptor;
@@ -72,7 +76,9 @@ import org.apache.spark.k8s.operator.SparkCluster;
import org.apache.spark.k8s.operator.config.SparkOperatorConf;
import org.apache.spark.k8s.operator.context.SparkClusterContext;
import org.apache.spark.k8s.operator.kueue.KueueWorkloadFactory;
+import org.apache.spark.k8s.operator.kueue.v1beta2.RequeueState;
import org.apache.spark.k8s.operator.kueue.v1beta2.Workload;
+import org.apache.spark.k8s.operator.kueue.v1beta2.WorkloadStatus;
import org.apache.spark.k8s.operator.reconciler.ReconcileProgress;
import org.apache.spark.k8s.operator.spec.ClusterSpec;
import org.apache.spark.k8s.operator.spec.ClusterTolerations;
@@ -490,6 +496,219 @@ class ClusterSuspendStepTest {
Assertions.assertNull(getWorkload());
}
+ @ParameterizedTest
+ @CsvSource({"Preempted, true", "Deactivated, false", "Deactivated, true"})
+ void evictedRunningClusterEntersSuspendedBeforeReleasingResources(String reason, boolean active) {
+ // A deactivated Workload no longer counts against the quota, and a reactivated one is queued
+ // again, so both release the master and workers like a preemption
+ SparkCluster cluster = buildKueueCluster(ClusterStateSummary.RunningHealthy, false);
+ stubContext(cluster);
+ createRunningCluster();
+ createWorkload(cluster, evictedStatus(reason), active);
+
+ Assertions.assertEquals(
+ ReconcileProgress.completeAndImmediateRequeue(),
+ new ClusterSuspendStep().reconcile(mockContext, recorder));
+
+ ClusterState state = captureAppendedState();
+ Assertions.assertEquals(ClusterStateSummary.Suspended, state.getCurrentStateSummary());
+ Assertions.assertEquals(
+ Constants.CLUSTER_EVICTED_MESSAGE + " " + reason + ": " + reason + " by the test",
+ state.getMessage());
+ // Like spec.suspend, nothing is released until Suspended is persisted, and the Workload is
+ // released only after the master and worker pods are gone
+ Assertions.assertNotNull(get(masterStatefulSetSpec));
+ Assertions.assertNotNull(get(workerStatefulSetSpec));
+ Assertions.assertNotNull(getWorkload());
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"Preempted", "Deactivated"})
+ void evictedClusterIsQueuedAgainAfterEverythingIsReleased(String reason) {
+ // A deactivated Workload which is reactivated is queued again like a preempted one
+ SparkCluster cluster = buildEvictedCluster(false);
+ stubContext(cluster);
+ createWorkload(cluster, evictedStatus(reason), true);
+
+ Assertions.assertEquals(
+ ReconcileProgress.completeAndImmediateRequeue(),
+ new ClusterSuspendStep().reconcile(mockContext, recorder));
+
+ Assertions.assertEquals(
+ Constants.CLUSTER_REQUEUED_MESSAGE, capturePersistedState().getMessage());
+ // ClusterInitStep creates a new Workload of the same name and waits for its admission
+ Assertions.assertNull(getWorkload());
+ }
+
+ @Test
+ void evictedClusterWhosePodsWereStuckIsQueuedAgain() {
+ // The state which names the stuck pods follows the one which says why it is suspended
+ SparkCluster cluster = buildEvictedCluster(false);
+ cluster.setStatus(
+ cluster
+ .getStatus()
+ .appendNewState(
+ new ClusterState(
+ ClusterStateSummary.Suspended,
+ String.format(
+ Constants.CLUSTER_SUSPENDED_WITH_STUCK_PODS_MESSAGE,
+ "cluster1-master-0"))));
+ stubContext(cluster);
+
+ Assertions.assertEquals(
+ ReconcileProgress.completeAndImmediateRequeue(),
+ new ClusterSuspendStep().reconcile(mockContext, recorder));
+
+ Assertions.assertEquals(
+ Constants.CLUSTER_REQUEUED_MESSAGE, capturePersistedState().getMessage());
+ }
+
+ @Test
+ void deactivatedWorkloadIsKeptUntilItIsReactivated() {
+ SparkCluster cluster = buildEvictedCluster(false);
+ stubContext(cluster);
+ createWorkload(cluster, evictedStatus("Deactivated"), false);
+
+ // Kueue no longer counts it, and a new Workload of the cluster would come back active
+ Assertions.assertEquals(
+ SUSPEND_HOLD_PROGRESS, new ClusterSuspendStep().reconcile(mockContext, recorder));
+
+ Assertions.assertNotNull(getWorkload());
+ verifyNoInteractions(recorder);
+ }
+
+ @Test
+ void resumedClusterReleasesDeactivatedWorkload() {
+ SparkCluster cluster = buildKueueCluster(ClusterStateSummary.Suspended, false);
+ stubContext(cluster);
+ // The Workload outlived the release on spec.suspend, e.g. after a failed delete
+ createWorkload(cluster, evictedStatus("Deactivated"), false);
+
+ // Only an eviction keeps the Workload, so the resume does not wait for its reactivation
+ Assertions.assertEquals(
+ ReconcileProgress.completeAndImmediateRequeue(),
+ new ClusterSuspendStep().reconcile(mockContext, recorder));
+
+ Assertions.assertEquals(
+ Constants.CLUSTER_RESUMED_MESSAGE, capturePersistedState().getMessage());
+ Assertions.assertNull(getWorkload());
+ }
+
+ @Test
+ void evictedWorkloadIsKeptUntilTheRequeueBackoffElapses() {
+ SparkCluster cluster = buildEvictedCluster(false);
+ stubContext(cluster);
+ WorkloadStatus status = evictedStatus("AdmissionCheck");
+ status.setRequeueState(
+ RequeueState.builder()
+ .count(1)
+ .requeueAt(Instant.now().plusSeconds(60).toString())
+ .build());
+ createWorkload(cluster, status);
+
+ ReconcileProgress progress = new ClusterSuspendStep().reconcile(mockContext, recorder);
+
+ // Its deletion would drop the backoff, so it is released once the backoff elapsed
+ Assertions.assertTrue(progress.isRequeue());
+ Assertions.assertTrue(
+ progress.getRequeueAfterDuration().compareTo(Duration.ofSeconds(50)) > 0,
+ progress.getRequeueAfterDuration().toString());
+ Assertions.assertNotNull(getWorkload());
+ verifyNoInteractions(recorder);
+ }
+
+ @Test
+ void suspendingEvictedClusterIsRecorded() {
+ SparkCluster cluster = buildEvictedCluster(true);
+ stubContext(cluster);
+ createWorkload(cluster, evictedStatus("Preempted"));
+
+ // The cluster is no longer queued again once released, and resuming it later says so
+ Assertions.assertEquals(
+ ReconcileProgress.completeAndImmediateRequeue(),
+ new ClusterSuspendStep().reconcile(mockContext, recorder));
+
+ ClusterState state = captureAppendedState();
+ Assertions.assertEquals(ClusterStateSummary.Suspended, state.getCurrentStateSummary());
+ Assertions.assertEquals(Constants.CLUSTER_SUSPENDED_MESSAGE, state.getMessage());
+
+ // Once that state is persisted, the cluster is held rather than suspended again
+ cluster.setStatus(cluster.getStatus().appendNewState(state));
+ Assertions.assertEquals(
+ SUSPEND_HOLD_PROGRESS, new ClusterSuspendStep().reconcile(mockContext, recorder));
+ verify(recorder).appendNewStateAndPersist(any(), any());
+ }
+
+ @Test
+ void evictedThenSuspendedClusterIsResumedRatherThanQueuedAgain() {
+ SparkCluster cluster = buildEvictedCluster(true);
+ stubContext(cluster);
+ createWorkload(cluster, evictedStatus("Preempted"));
+ new ClusterSuspendStep().reconcile(mockContext, recorder);
+ cluster.setStatus(cluster.getStatus().appendNewState(captureAppendedState()));
+
+ // Held by spec.suspend since the eviction, so clearing it is a resume, not a requeue
+ cluster.getSpec().setSuspend(false);
+ new ClusterSuspendStep().reconcile(mockContext, recorder);
+
+ Assertions.assertEquals(
+ Constants.CLUSTER_RESUMED_MESSAGE, capturePersistedState().getMessage());
+ }
+
+ @Test
+ void runningClusterWithAdmittedWorkloadProceeds() {
+ SparkCluster cluster = buildKueueCluster(ClusterStateSummary.RunningHealthy, false);
+ stubContext(cluster);
+ createRunningCluster();
+ createWorkload(
+ cluster,
+ WorkloadStatus.builder()
+ .conditions(
+ List.of(new ConditionBuilder().withType("Admitted").withStatus("True").build()))
+ .build());
+
+ Assertions.assertEquals(
+ ReconcileProgress.proceed(), new ClusterSuspendStep().reconcile(mockContext, recorder));
+
+ verifyNoInteractions(recorder);
+ Assertions.assertNotNull(getWorkload());
+ }
+
+ @Test
+ void evictedClusterWhoseQueueLabelIsRemovedEntersSuspended() {
+ SparkCluster cluster = buildKueueCluster(ClusterStateSummary.RunningHealthy, false);
+ stubContext(cluster);
+ createWorkload(cluster, evictedStatus("Preempted"));
+ // Like the release on spec.suspend, the Workload admitted before the label was removed counts
+ cluster.getMetadata().setLabels(Map.of());
+
+ Assertions.assertEquals(
+ ReconcileProgress.completeAndImmediateRequeue(),
+ new ClusterSuspendStep().reconcile(mockContext, recorder));
+
+ Assertions.assertEquals(
+ ClusterStateSummary.Suspended, captureAppendedState().getCurrentStateSummary());
+ }
+
+ @Test
+ void podsReadyTimeoutIsReportedAndIgnored() {
+ SparkCluster cluster = buildKueueCluster(ClusterStateSummary.RunningHealthy, false);
+ stubContext(cluster);
+ createWorkload(cluster, evictedStatus("PodsReadyTimeout"));
+ when(mockContext.getEventRecorder()).thenReturn(eventRecorder);
+
+ // Kueue keeps counting its quota, and every attempt would time out again
+ Assertions.assertEquals(
+ ReconcileProgress.proceed(), new ClusterSuspendStep().reconcile(mockContext, recorder));
+
+ Assertions.assertNotNull(getWorkload());
+ ArgumentCaptor<EventRecord> event = ArgumentCaptor.forClass(EventRecord.class);
+ verify(eventRecorder).record(event.capture());
+ Assertions.assertEquals(EventType.WARNING, event.getValue().type());
+ Assertions.assertEquals(EventUtils.REASON_KUEUE_EVICTION_IGNORED, event.getValue().reason());
+ verifyNoInteractions(recorder);
+ }
+
@Test
void otherStatesProceed() {
for (ClusterStateSummary summary :
@@ -517,6 +736,9 @@ class ClusterSuspendStepTest {
when(recorder.persistStatus(any(), any())).thenReturn(true);
when(mockContext.getResource()).thenReturn(cluster);
when(mockContext.getClient()).thenReturn(client);
+
+ // Like the Workload informer, the cache follows the Workload in the API server
+ when(mockContext.getCachedKueueWorkload()).thenAnswer(i -> Optional.ofNullable(getWorkload()));
}
/**
@@ -606,6 +828,51 @@ class ClusterSuspendStepTest {
return cluster;
}
+ /** Builds a cluster which entered Suspended on a Kueue eviction. */
+ private static SparkCluster buildEvictedCluster(boolean suspend) {
+ SparkCluster cluster = buildKueueCluster(ClusterStateSummary.Suspended, suspend);
+ cluster
+ .getStatus()
+ .getCurrentState()
+ .setMessage(Constants.CLUSTER_EVICTED_MESSAGE + " Preempted: Preempted by the test");
+ return cluster;
+ }
+
+ private void createWorkload(SparkCluster cluster, WorkloadStatus status) {
+ createWorkload(cluster, status, true);
+ }
+
+ private void createWorkload(SparkCluster cluster, WorkloadStatus status, boolean active) {
+ Workload desired = KueueWorkloadFactory.buildWorkload(cluster);
+ desired.getSpec().setActive(active);
+ kubernetesClient.resource(desired).create();
+ Workload workload = getWorkload();
+ workload.setStatus(status);
+ kubernetesClient.resource(workload).update();
+ }
+
+ private ClusterState capturePersistedState() {
+ ArgumentCaptor<ClusterStatus> status = ArgumentCaptor.forClass(ClusterStatus.class);
+ verify(recorder).persistStatus(eq(mockContext), status.capture());
+ ClusterState state = status.getValue().getCurrentState();
+ Assertions.assertEquals(ClusterStateSummary.Submitted, state.getCurrentStateSummary());
+ return state;
+ }
+
+ private static WorkloadStatus evictedStatus(String reason) {
+ return WorkloadStatus.builder()
+ .conditions(
+ List.of(
+ new ConditionBuilder().withType("Admitted").withStatus("True").build(),
+ new ConditionBuilder()
+ .withType("Evicted")
+ .withStatus("True")
+ .withReason(reason)
+ .withMessage(reason + " by the test")
+ .build()))
+ .build();
+ }
+
private static StatefulSet statefulSet(String name) {
return new StatefulSetBuilder()
.withNewMetadata()
diff --git a/tests/e2e/kueue/chainsaw-test.yaml b/tests/e2e/kueue/chainsaw-test.yaml
index 7acf8f81..ec8d10b2 100644
--- a/tests/e2e/kueue/chainsaw-test.yaml
+++ b/tests/e2e/kueue/chainsaw-test.yaml
@@ -680,3 +680,144 @@ spec:
kubectl delete localqueue "$LOCAL_QUEUE" -n default --ignore-not-found=true --wait=false
kubectl delete clusterqueue "$CLUSTER_QUEUE" --ignore-not-found=true --wait=false
kubectl delete resourceflavor "$FLAVOR" --ignore-not-found=true --wait=false
+ - name: spark-cluster-is-preempted-by-kueue
+ # Each step uses its own queue objects, so that a step never waits for Kueue to finish
+ # deleting the objects of another step.
+ bindings:
+ - name: FLAVOR
+ value: spark-e2e-preemption-flavor
+ - name: CLUSTER_QUEUE
+ value: spark-e2e-preemption-cluster-queue
+ - name: LOCAL_QUEUE
+ value: spark-e2e-preemption-queue
+ - name: LOW_PRIORITY_CLASS
+ value: spark-e2e-preemption-low-priority
+ - name: PRIORITY_CLASS
+ value: spark-e2e-preemption-high-priority
+ try:
+ - apply:
+ file: kueue-queues.yaml
+ # The quota fits either the cluster (1 CPU for the master and 1 for the worker) or the
+ # application (1 CPU for the driver and 1 for the executor), but not both, and a Workload of a
+ # higher priority may preempt one of a lower priority.
+ - script:
+ timeout: 30s
+ env:
+ - name: CLUSTER_QUEUE
+ value: ($CLUSTER_QUEUE)
+ content: |
+ kubectl patch clusterqueue "$CLUSTER_QUEUE" --type json -p \
+ '[{"op": "replace", "path": "/spec/resourceGroups/0/flavors/0/resources/0/nominalQuota", "value": "2"},
+ {"op": "replace", "path": "/spec/resourceGroups/0/flavors/0/resources/1/nominalQuota", "value": "8Gi"},
+ {"op": "add", "path": "/spec/preemption", "value": {"withinClusterQueue": "LowerPriority"}}]'
+ - apply:
+ file: spark-cluster-example-preempted.yaml
+ - assert:
+ timeout: 5m
+ resource:
+ apiVersion: spark.apache.org/v1
+ kind: SparkCluster
+ metadata:
+ name: spark-cluster-kueue-preempted-test
+ namespace: default
+ status:
+ currentState:
+ currentStateSummary: RunningHealthy
+ - apply:
+ file: spark-example-prioritized.yaml
+ # Kueue evicts the Workload of the cluster, and keeps its quota until the operator releases the
+ # master and workers first and then the Workload.
+ - assert:
+ timeout: 5m
+ resource:
+ apiVersion: v1
+ kind: Event
+ metadata:
+ namespace: default
+ involvedObject:
+ kind: SparkCluster
+ name: spark-cluster-kueue-preempted-test
+ reason: Suspended
+ (contains(message || '', 'evicted by Kueue')): true
+ (contains(message || '', 'Preempted')): true
+ - assert:
+ timeout: 5m
+ resource:
+ apiVersion: v1
+ kind: Event
+ metadata:
+ namespace: default
+ involvedObject:
+ kind: SparkCluster
+ name: spark-cluster-kueue-preempted-test
+ reason: Submitted
+ (contains(message || '', 'after its eviction')): true
+ # The application is admitted into the quota which the cluster gave back
+ - assert:
+ bindings:
+ - name: SPARK_APP_NAMESPACE
+ value: default
+ - name: SPARK_APPLICATION_NAME
+ value: spark-job-kueue-prioritized-test
+ timeout: 10m
+ file: "../assertions/spark-application/spark-state-transition.yaml"
+ # The cluster is admitted again with a new Workload once the application releases the quota
+ - assert:
+ timeout: 5m
+ resource:
+ apiVersion: spark.apache.org/v1
+ kind: SparkCluster
+ metadata:
+ name: spark-cluster-kueue-preempted-test
+ namespace: default
+ status:
+ currentState:
+ currentStateSummary: RunningHealthy
+ - assert:
+ bindings:
+ - name: SPARK_APP_NAMESPACE
+ value: default
+ - name: SPARK_CLUSTER_NAME
+ value: spark-cluster-kueue-preempted-test
+ timeout: 5m
+ file: "../assertions/spark-cluster/spark-cluster-worker-statefulset.yaml"
+ catch:
+ - describe:
+ apiVersion: spark.apache.org/v1
+ kind: SparkCluster
+ namespace: default
+ - describe:
+ apiVersion: spark.apache.org/v1
+ kind: SparkApplication
+ namespace: default
+ - describe:
+ apiVersion: kueue.x-k8s.io/v1beta2
+ kind: Workload
+ namespace: default
+ - describe:
+ apiVersion: kueue.x-k8s.io/v1beta2
+ kind: ClusterQueue
+ - events:
+ namespace: default
+ finally:
+ - script:
+ timeout: 120s
+ env:
+ - name: FLAVOR
+ value: ($FLAVOR)
+ - name: CLUSTER_QUEUE
+ value: ($CLUSTER_QUEUE)
+ - name: LOCAL_QUEUE
+ value: ($LOCAL_QUEUE)
+ - name: LOW_PRIORITY_CLASS
+ value: ($LOW_PRIORITY_CLASS)
+ - name: PRIORITY_CLASS
+ value: ($PRIORITY_CLASS)
+ content: |
+ kubectl delete sparkapplication spark-job-kueue-prioritized-test -n default --ignore-not-found=true
+ kubectl delete sparkcluster spark-cluster-kueue-preempted-test -n default --ignore-not-found=true
+ # Kueue finalizers may delay the deletion, which later steps do not depend on.
+ kubectl delete localqueue "$LOCAL_QUEUE" -n default --ignore-not-found=true --wait=false
+ kubectl delete clusterqueue "$CLUSTER_QUEUE" --ignore-not-found=true --wait=false
+ kubectl delete resourceflavor "$FLAVOR" --ignore-not-found=true --wait=false
+ kubectl delete workloadpriorityclass "$LOW_PRIORITY_CLASS" "$PRIORITY_CLASS" --ignore-not-found=true --wait=false
diff --git a/tests/e2e/kueue/spark-cluster-example-preempted.yaml b/tests/e2e/kueue/spark-cluster-example-preempted.yaml
new file mode 100644
index 00000000..4ba10f00
--- /dev/null
+++ b/tests/e2e/kueue/spark-cluster-example-preempted.yaml
@@ -0,0 +1,43 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+apiVersion: kueue.x-k8s.io/v1beta2
+kind: WorkloadPriorityClass
+metadata:
+ name: ($LOW_PRIORITY_CLASS)
+value: 100
+---
+apiVersion: spark.apache.org/v1
+kind: SparkCluster
+metadata:
+ name: spark-cluster-kueue-preempted-test
+ namespace: default
+ labels:
+ kueue.x-k8s.io/queue-name: ($LOCAL_QUEUE)
+ kueue.x-k8s.io/priority-class: ($LOW_PRIORITY_CLASS)
+spec:
+ runtimeVersions:
+ sparkVersion: "4.2.0"
+ clusterTolerations:
+ instanceConfig:
+ initWorkers: 1
+ minWorkers: 1
+ maxWorkers: 1
+ sparkConf:
+ spark.kubernetes.container.image: "apache/spark:{{SPARK_VERSION}}-scala"
+ spark.master.ui.title: "Spark Cluster Kueue Preemption E2E Test"
+ spark.master.rest.host: "0.0.0.0"
+ spark.ui.reverseProxy: "true"
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org