You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/09/23 17:58:03 UTC
(spark-kubernetes-operator) branch main updated: [SPARK-59750] Support `spec.suspend` for a running `SparkCluster`
This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new fcc9b026 [SPARK-59750] Support `spec.suspend` for a running `SparkCluster`
fcc9b026 is described below
commit fcc9b026d45962c1c04ae27afee84be4af3c1ab4
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Wed Sep 23 10:57:55 2026 -0700
[SPARK-59750] Support `spec.suspend` for a running `SparkCluster`
### What changes were proposed in this pull request?
This PR aims to support `spec.suspend` for a running `SparkCluster` via a new `Suspended` state.
- `RunningHealthy` with `spec.suspend=true` moves to `Suspended`, where the operator deletes the master/worker `StatefulSet`s, the worker HPA/PDB, and then the Kueue `Workload` after the pods are gone.
- Setting `spec.suspend=false` moves the cluster back to `Submitted` once everything is released, and the cluster is requested again from scratch.
- `ClusterInitStep` retries transient API failures (e.g. timeout, `429`, `500`) instead of failing the cluster with the terminal `SchedulingFailure`.
### Why are the changes needed?
Currently, `spec.suspend` has no effect on a running `SparkCluster`, so deleting the cluster is the only way to release its resources and its Kueue quota.
### Does this PR introduce _any_ user-facing change?
Yes. A running `SparkCluster` can be suspended and resumed, with a new `Suspended` state and `SuspendReleaseFailed` event.
**BEFORE**
```mermaid
stateDiagram-v2
[*] --> Submitted
Submitted --> RunningHealthy
Submitted --> SchedulingFailure
SchedulingFailure --> Failed
RunningHealthy --> [*]
Failed --> [*]
```
**AFTER**
```mermaid
stateDiagram-v2
[*] --> Submitted
Submitted --> RunningHealthy
Submitted --> SchedulingFailure
SchedulingFailure --> Failed
RunningHealthy --> Suspended : spec.suspend=true
Suspended --> Submitted : spec.suspend=false
Submitted --> Suspended : spec.suspend=true after resume
RunningHealthy --> [*]
Suspended --> [*]
Failed --> [*]
```
### How was this patch tested?
Pass the CIs with the newly added unit tests and E2E tests (`tests/e2e/suspend`, `tests/e2e/kueue`).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #875 from dongjoon-hyun/SPARK-59750.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.../crds/sparkclusters.spark.apache.org-v1.yaml | 4 +
docs/architecture.md | 14 +-
docs/config_properties.md | 4 +-
docs/configuration.md | 4 +-
docs/spark_custom_resources.md | 42 +-
.../org/apache/spark/k8s/operator/Constants.java | 9 +
.../k8s/operator/status/ClusterStateSummary.java | 9 +-
.../spark/k8s/operator/status/ClusterStatus.java | 21 +
.../operator/status/ClusterStateSummaryTest.java | 49 ++
.../k8s/operator/status/ClusterStatusTest.java | 19 +
.../k8s/operator/config/SparkOperatorConf.java | 15 +-
.../k8s/operator/kueue/KueueWorkloadUtils.java | 5 +-
.../reconciler/SparkClusterReconciler.java | 5 +-
.../reconciler/reconcilesteps/ClusterInitStep.java | 63 ++-
.../reconcilesteps/ClusterReconcileStep.java | 22 +-
.../reconcilesteps/ClusterSuspendStep.java | 203 ++++++++
.../spark/k8s/operator/utils/EventUtils.java | 12 +
.../spark/k8s/operator/utils/ReconcilerUtils.java | 18 +-
.../operator/utils/SparkClusterStatusRecorder.java | 5 +-
.../reconciler/SparkClusterReconcilerTest.java | 23 +-
.../reconcilesteps/ClusterInitStepTest.java | 102 ++++
.../reconcilesteps/ClusterSuspendStepTest.java | 541 +++++++++++++++++++++
.../k8s/operator/SparkClusterResourceSpec.java | 52 +-
tests/e2e/kueue/chainsaw-test.yaml | 76 +++
tests/e2e/suspend/chainsaw-test.yaml | 71 +++
25 files changed, 1331 insertions(+), 57 deletions(-)
diff --git a/build-tools/helm/spark-kubernetes-operator/crds/sparkclusters.spark.apache.org-v1.yaml b/build-tools/helm/spark-kubernetes-operator/crds/sparkclusters.spark.apache.org-v1.yaml
index 72c56680..a3f2a23e 100644
--- a/build-tools/helm/spark-kubernetes-operator/crds/sparkclusters.spark.apache.org-v1.yaml
+++ b/build-tools/helm/spark-kubernetes-operator/crds/sparkclusters.spark.apache.org-v1.yaml
@@ -14753,6 +14753,7 @@ spec:
- RunningHealthy
- SchedulingFailure
- Submitted
+ - Suspended
type: string
lastTransitionTime:
type: string
@@ -14770,6 +14771,7 @@ spec:
- RunningHealthy
- SchedulingFailure
- Submitted
+ - Suspended
type: string
lastTransitionTime:
type: string
@@ -14795,6 +14797,7 @@ spec:
- RunningHealthy
- SchedulingFailure
- Submitted
+ - Suspended
type: string
lastTransitionTime:
type: string
@@ -14813,6 +14816,7 @@ spec:
- RunningHealthy
- SchedulingFailure
- Submitted
+ - Suspended
type: string
lastTransitionTime:
type: string
diff --git a/docs/architecture.md b/docs/architecture.md
index 9d0ce4e7..1378c73c 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -201,13 +201,19 @@ stateDiagram-v2
SchedulingFailure --> Failed
+ RunningHealthy --> Suspended : spec.suspend=true
+ Suspended --> Submitted : spec.suspend=false
+ Submitted --> Suspended : spec.suspend=true after resume
+
RunningHealthy --> [*]
+ Suspended --> [*]
Failed --> [*]
```
-* Spark clusters are expected to be always running after submitted.
-* A cluster leaves `RunningHealthy` or `Failed` only when its custom resource is deleted. At that
- point, the K8s resources created for the cluster are garbage collected through their
- `ownerReference` to the `SparkCluster` custom resource.
+* Spark clusters are expected to be always running after submitted, until they are suspended by
+ [`spec.suspend`](spark_custom_resources.md#suspend).
+* Apart from `spec.suspend`, a cluster leaves `RunningHealthy`, `Suspended` or `Failed` only when
+ its custom resource is deleted. At that point, the K8s resources created for the cluster are
+ garbage collected through their `ownerReference` to the `SparkCluster` custom resource.
* A `Failed` cluster is not reconciled any further.
* `ResourceReleased` exists in the API enum but is currently not used for clusters.
diff --git a/docs/config_properties.md b/docs/config_properties.md
index a7dcbe8d..ca9cfaf7 100644
--- a/docs/config_properties.md
+++ b/docs/config_properties.md
@@ -53,9 +53,9 @@
| spark.kubernetes.operator.reconciler.retry.initialIntervalSeconds | Integer | 5 | false | Initial interval(in seconds) of retries on unhandled controller errors. |
| spark.kubernetes.operator.reconciler.retry.intervalMultiplier | Double | 1.5 | false | Interval multiplier of retries on unhandled controller errors. Setting this to 1 for linear retry. |
| spark.kubernetes.operator.reconciler.retry.maxIntervalSeconds | Integer | -1 | false | Max interval(in seconds) of retries on unhandled controller errors. Set to non-positive for unlimited. |
- | spark.kubernetes.operator.reconciler.suspendHoldRequeueIntervalSeconds | Long | 1800 | true | Requeue interval (in seconds) at which a resource held by spec.suspend is reconciled, so that its SuspendHeld event is republished. The hold ends only when a user clears spec.suspend, which arrives as a watch event and reconciles right away, so nothing waits for this interval. It is deliberately much coarser than spark.kubernetes.operator.reconciler.intervalSeconds, since a suspended resource [...]
+ | spark.kubernetes.operator.reconciler.suspendHoldRequeueIntervalSeconds | Long | 1800 | true | Requeue interval (in seconds) at which a resource held by spec.suspend is reconciled, so that its SuspendHeld event is republished. A SparkCluster suspended while running publishes no such event, since its Suspended status says so, and uses this interval only to look at its released resources again. The hold ends only when a user clears spec.suspend, which arrives as a watch event and reconci [...]
| spark.kubernetes.operator.reconciler.terminationTimeoutSeconds | Integer | 30 | false | Grace period for operator shutdown before reconciliation threads are killed. |
- | spark.kubernetes.operator.reconciler.trimStateTransitionHistoryEnabled | Boolean | true | true | When enabled, operator would trim state transition history when a new attempt starts, keeping previous attempt summary only. |
+ | spark.kubernetes.operator.reconciler.trimStateTransitionHistoryEnabled | Boolean | true | true | When enabled, operator would trim state transition history when a new attempt starts, keeping previous attempt summary only. It also drops the history of a suspended SparkCluster when it is resumed. |
| spark.kubernetes.operator.watchedNamespaces | String | default | true | Comma-separated list of namespaces that the operator would be watching for Spark resources. If set to '*', operator would watch all namespaces. |
| spark.logConf | Boolean | false | false | When enabled, operator will print configurations |
| spark.redaction.regex | String | (?i)secret\|password\|token\|access[.]?key | false | Regex to decide which parts of configuration properties contain sensitive information, whose values would be redacted in operator logs. This reuses Spark's redaction configuration key and default pattern. |
diff --git a/docs/configuration.md b/docs/configuration.md
index 8edbd87b..8ca776ba 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -67,7 +67,7 @@ replacing its `message`. A repeat carrying the same `message` may be paced by
|---|---|
| `Warning` | `SparkApplication`: `SchedulingFailure`, `Failed`, `DriverEvicted`, `DriverStartTimedOut`, `DriverReadyTimedOut`, `ExecutorsStartTimedOut`, `RunningWithBelowThresholdExecutors`, `TerminatedWithoutReleaseResources` |
| `Warning` | `SparkCluster`: `SchedulingFailure`, `Failed` |
-| `Normal` | All other states, e.g. `Submitted`, `ScheduledToRestart`, `DriverRequested`, `DriverStarted`, `DriverReady`, `InitializedBelowThresholdExecutors`, `RunningHealthy`, `RunningWithPartialCapacity`, `Succeeded` and `ResourceReleased` |
+| `Normal` | All other states, e.g. `Submitted`, `ScheduledToRestart`, `DriverRequested`, `DriverStarted`, `DriverReady`, `InitializedBelowThresholdExecutors`, `RunningHealthy`, `RunningWithPartialCapacity`, `Succeeded`, `ResourceReleased` and `Suspended` |
In addition, the operator publishes the following `Warning` events.
@@ -77,6 +77,8 @@ In addition, the operator publishes the following `Warning` events.
| `CleanupError` | A cleanup throws, so the resource cannot finish deleting. |
| `StatusUpdateFailed` | A status patch is rejected. Transport-level errors are skipped. |
| `KueueAdmissionRequestFailed` | Creating, reading or deleting a stale Kueue `Workload` fails, or the driver or master cannot be read before the admission is requested. Transport-level errors are skipped and retried every 5 seconds, while a persistent failure is retried with the default interval. |
+| `ClusterRequestFailed` | Applying the Services, StatefulSets, NetworkPolicy, HorizontalPodAutoscaler or PodDisruptionBudget of a `SparkCluster` fails in a way which may yet succeed, such as a throttled request or an internal server error from an admission webhook, so it is retried with the default interval instead of failing the cluster. Transport-level errors are skipped. |
+| `SuspendReleaseFailed` | Deleting the master and worker StatefulSets, the HorizontalPodAutoscaler or PodDisruptionBudget of the workers, or the Kueue `Workload` of a `SparkCluster` suspended while running fails, or its pods cannot be listed. A failure which may clear on its own, such as a timeout, is skipped. Every failure is retried with the default interval. |
| `KueueResourceFlavorReadFailed` | Reading the `ResourceFlavor`s which Kueue assigned to an admitted `Workload` fails, so the node selector and the tolerations of the flavors cannot be applied. Transport-level errors are skipped and retried every 5 seconds, while a persistent failure is retried with the default interval. |
For a resource held by [`spec.suspend`](spark_custom_resources.md#suspend) or queued by
diff --git a/docs/spark_custom_resources.md b/docs/spark_custom_resources.md
index 9824b812..8a1494ab 100644
--- a/docs/spark_custom_resources.md
+++ b/docs/spark_custom_resources.md
@@ -530,10 +530,11 @@ to this app after `ttlAfterStopMillis`.
## Suspend
-Both `SparkApplication` and `SparkCluster` support `.spec.suspend`. When it is set to `true`, the
-operator keeps the resource in its initializing state (`Submitted`, or `ScheduledToRestart` for an
-application that is scheduled to restart) and does not request the driver pod or the master / worker
-StatefulSets. Setting it back to `false` resumes the regular lifecycle.
+Both `SparkApplication` and `SparkCluster` support `.spec.suspend`. When it is set to `true` before
+the driver pod or the master / worker StatefulSets are requested, the operator keeps the resource in
+its initializing state (`Submitted`, or `ScheduledToRestart` for an application that is scheduled to
+restart) and does not request them. Setting it back to `false` resumes the regular lifecycle. A
+running `SparkCluster` is stopped instead, as described below.
`Submitted` here is the operator's in-memory view. For a valid resource created with
`suspend: true`, the initial `Submitted` status is not persisted to the API server, so
@@ -558,10 +559,30 @@ spec:
sparkVersion: "4.2.0"
```
-* `suspend` only takes effect before the driver (or master / worker) resources are requested.
+* For a `SparkApplication`, `suspend` only takes effect before the driver resources are requested.
Setting it to `true` on a running application does not stop the current attempt. If the
application is configured to restart, the next attempt is held until `suspend` is set back to
- `false`. Setting it to `true` on a running cluster has no effect in the current version.
+ `false`.
+* Setting it to `true` on a running `SparkCluster` (`RunningHealthy`) stops it, since a cluster
+ runs until it is deleted and has no other way to give its resources back. The cluster enters
+ `Suspended` first, and then the operator deletes its master and worker StatefulSets, and the
+ HorizontalPodAutoscaler and PodDisruptionBudget of its workers if any. Its Services and
+ NetworkPolicy are kept. Any application running on the cluster is terminated with it, since there
+ is no graceful decommission. A failure to delete them, other than one which may clear on its own,
+ publishes the `SuspendReleaseFailed` [event](configuration.md#kubernetes-events) when enabled.
+* Setting it back to `false` moves the cluster to `Submitted` once its master and worker pods are
+ gone and its Kueue `Workload`, if any, is released, and the master and workers are requested
+ again from scratch, like a newly created cluster. Only pods labeled with the `master` or `worker`
+ `spark-role` count, so other pods which carry the `spark.operator/spark-cluster-name` label do
+ not hold the cluster back. A request which may yet succeed, such as a timeout or an unavailable
+ API server or admission webhook, is retried, while a rejected one fails the cluster with
+ `SchedulingFailure`, as for a newly created cluster. With
+ `spark.kubernetes.operator.reconciler.trimStateTransitionHistoryEnabled`, the resumed cluster drops
+ the state transition history of its previous run, so that suspending it again and again keeps the
+ status bounded. Setting it to `true` again before the master is requested moves the cluster back
+ to `Suspended`.
+* An operator version without the `Suspended` state cannot read a `SparkCluster` which is
+ `Suspended`, so resume or delete such clusters before downgrading the operator.
* Deleting a suspended resource works as usual.
* This is the building block for external job queueing systems. See [Kueue](#kueue) for the
built-in integration.
@@ -621,8 +642,8 @@ spec:
* A `SparkCluster` requests the resources set on the `master` and `worker` containers of its pod
templates. A missing request defaults to the limit, or else to 1 CPU and `SPARK_DAEMON_MEMORY`
plus overhead. A worker uses `SPARK_WORKER_CORES` for the CPU and adds `SPARK_WORKER_MEMORY` to
- the memory when they are set. A `SparkCluster` keeps the quota until it is deleted. Set a cpu and
- memory request or limit on the `worker` container, or `SPARK_WORKER_CORES` and
+ the memory when they are set. A `SparkCluster` keeps the quota until it is deleted or suspended.
+ Set a cpu and memory request or limit on the `worker` container, or `SPARK_WORKER_CORES` and
`SPARK_WORKER_MEMORY`: a worker with none of them advertises the whole node to its executors,
well above the default the `Workload` requests.
* Like Kueue built-in integrations, the `nodeLabels` and `tolerations` of the `ResourceFlavor`s
@@ -638,8 +659,9 @@ spec:
through the ClusterRole, hence `operatorRbac.clusterRole.create` as well. Until they can be
read, the resource is held and the read is retried.
* `spec.suspend` takes precedence. A suspended resource does not get a `Workload`, and suspending
- a queued resource deletes its `Workload` to release the quota. The resource is queued again when
- it is resumed.
+ a queued resource deletes its `Workload` to release the quota. Suspending a running
+ `SparkCluster` deletes its `Workload` only after its master and worker pods are gone, since
+ terminating pods still occupy the quota. The resource is queued again when it is resumed.
* Dynamic allocation, a `SparkCluster` with `minWorkers < maxWorkers`, and pod template files set
through `spark.kubernetes.{driver,executor}.podTemplateFile` are not supported yet. Such a
resource fails with `SchedulingFailure` instead of being queued.
diff --git a/spark-operator-api/src/main/java/org/apache/spark/k8s/operator/Constants.java b/spark-operator-api/src/main/java/org/apache/spark/k8s/operator/Constants.java
index 59c2761a..b952b8c4 100644
--- a/spark-operator-api/src/main/java/org/apache/spark/k8s/operator/Constants.java
+++ b/spark-operator-api/src/main/java/org/apache/spark/k8s/operator/Constants.java
@@ -242,6 +242,15 @@ public final class Constants {
/** Message indicating that the cluster has reached a ready state. */
public static final String CLUSTER_READY_MESSAGE = "Cluster has reached ready state.";
+ /** Message indicating that the cluster is suspended by spec.suspend while it was running. */
+ public static final String CLUSTER_SUSPENDED_MESSAGE =
+ "Cluster is suspended by spec.suspend, so its master and workers are being released. "
+ + "Setting spec.suspend to false resumes it once they are released.";
+
+ /** Message indicating that the suspended cluster is resumed by clearing spec.suspend. */
+ public static final String CLUSTER_RESUMED_MESSAGE =
+ "Cluster is resumed as spec.suspend is set to false.";
+
/** Message indicating that the cluster status cannot be processed. */
public static final String UNKNOWN_CLUSTER_STATE_MESSAGE = "Cannot process cluster status.";
diff --git a/spark-operator-api/src/main/java/org/apache/spark/k8s/operator/status/ClusterStateSummary.java b/spark-operator-api/src/main/java/org/apache/spark/k8s/operator/status/ClusterStateSummary.java
index ac05e245..b77ee1f6 100644
--- a/spark-operator-api/src/main/java/org/apache/spark/k8s/operator/status/ClusterStateSummary.java
+++ b/spark-operator-api/src/main/java/org/apache/spark/k8s/operator/status/ClusterStateSummary.java
@@ -38,7 +38,14 @@ public enum ClusterStateSummary implements BaseStateSummary {
Failed,
/** all resources (pods, services .etc have been cleaned up) */
- ResourceReleased;
+ ResourceReleased,
+
+ /**
+ * Cluster is suspended by spec.suspend after it started running, so its master and workers are
+ * released until it is resumed from Submitted. It is declared last, since {@link #isStarting()}
+ * compares the ordinals.
+ */
+ Suspended;
/**
* Checks if the cluster is in an initializing state.
diff --git a/spark-operator-api/src/main/java/org/apache/spark/k8s/operator/status/ClusterStatus.java b/spark-operator-api/src/main/java/org/apache/spark/k8s/operator/status/ClusterStatus.java
index 3ed75b11..3ca2fb92 100644
--- a/spark-operator-api/src/main/java/org/apache/spark/k8s/operator/status/ClusterStatus.java
+++ b/spark-operator-api/src/main/java/org/apache/spark/k8s/operator/status/ClusterStatus.java
@@ -83,6 +83,27 @@ public class ClusterStatus
currentAttemptSummary);
}
+ /**
+ * Appends a new state, and drops the states before it when trimStateTransitionHistory is set, so
+ * that a cluster which is suspended and resumed again and again keeps a bounded history. The new
+ * state keeps the next id, so that it is still told apart from the states before it.
+ *
+ * @param state The state to append.
+ * @param trimStateTransitionHistory Whether to drop the states before the new one.
+ * @return The updated status.
+ * @since 1.1.0
+ */
+ public ClusterStatus appendNewState(ClusterState state, boolean trimStateTransitionHistory) {
+ if (!trimStateTransitionHistory) {
+ return appendNewState(state);
+ }
+ return new ClusterStatus(
+ state,
+ Map.of(getCurrentStateId() + 1, state),
+ previousAttemptSummary,
+ currentAttemptSummary);
+ }
+
/**
* Creates an updated state transition history with a new state appended.
*
diff --git a/spark-operator-api/src/test/java/org/apache/spark/k8s/operator/status/ClusterStateSummaryTest.java b/spark-operator-api/src/test/java/org/apache/spark/k8s/operator/status/ClusterStateSummaryTest.java
new file mode 100644
index 00000000..e4ce4efd
--- /dev/null
+++ b/spark-operator-api/src/test/java/org/apache/spark/k8s/operator/status/ClusterStateSummaryTest.java
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.status;
+
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import org.junit.jupiter.api.Test;
+
+class ClusterStateSummaryTest {
+
+ @Test
+ void testSuspendedIsNeitherStartingNorTerminated() {
+ // A suspended cluster is reconciled again to be resumed, so it must not be terminated, and it
+ // must not be mistaken for a starting cluster by the ordinal comparison of isStarting
+ assertFalse(ClusterStateSummary.Suspended.isInitializing());
+ assertFalse(ClusterStateSummary.Suspended.isStarting());
+ assertFalse(ClusterStateSummary.Suspended.isTerminated());
+ assertFalse(ClusterStateSummary.Suspended.isFailure());
+ assertFalse(ClusterStateSummary.Suspended.isInfrastructureFailure());
+ }
+
+ @Test
+ void testIsStarting() {
+ assertTrue(ClusterStateSummary.Submitted.isStarting());
+ assertTrue(ClusterStateSummary.SchedulingFailure.isStarting());
+
+ assertFalse(ClusterStateSummary.RunningHealthy.isStarting());
+ assertFalse(ClusterStateSummary.Failed.isStarting());
+ assertFalse(ClusterStateSummary.ResourceReleased.isStarting());
+ }
+}
diff --git a/spark-operator-api/src/test/java/org/apache/spark/k8s/operator/status/ClusterStatusTest.java b/spark-operator-api/src/test/java/org/apache/spark/k8s/operator/status/ClusterStatusTest.java
index 6e18b3e4..96270f93 100644
--- a/spark-operator-api/src/test/java/org/apache/spark/k8s/operator/status/ClusterStatusTest.java
+++ b/spark-operator-api/src/test/java/org/apache/spark/k8s/operator/status/ClusterStatusTest.java
@@ -41,4 +41,23 @@ class ClusterStatusTest {
assertEquals(2, newStatus.getStateTransitionHistory().size());
assertEquals(newState, newStatus.getStateTransitionHistory().get(1L));
}
+
+ @Test
+ void testAppendNewStateWithTrim() {
+ ClusterStatus status =
+ new ClusterStatus()
+ .appendNewState(new ClusterState(ClusterStateSummary.RunningHealthy, ""))
+ .appendNewState(new ClusterState(ClusterStateSummary.Suspended, ""));
+ ClusterState resumed = new ClusterState(Submitted, "resumed");
+
+ // Only the new state is kept, with the next id, so that it is still a new state
+ ClusterStatus trimmed = status.appendNewState(resumed, true);
+ assertEquals(resumed, trimmed.getCurrentState());
+ assertEquals(1, trimmed.getStateTransitionHistory().size());
+ assertEquals(resumed, trimmed.getStateTransitionHistory().get(3L));
+
+ ClusterStatus kept = status.appendNewState(resumed, false);
+ assertEquals(4, kept.getStateTransitionHistory().size());
+ assertEquals(resumed, kept.getStateTransitionHistory().get(3L));
+ }
}
diff --git a/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java b/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java
index b075d4f4..49ca5056 100644
--- a/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java
+++ b/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java
@@ -264,7 +264,9 @@ public final class SparkOperatorConf {
/**
* Requeue interval (in seconds) at which a resource held by {@code spec.suspend} is reconciled,
- * so that its {@code SuspendHeld} event is republished. The hold ends only when a user clears
+ * so that its {@code SuspendHeld} event is republished. A SparkCluster suspended while running
+ * publishes no such event, since its Suspended status says so, and uses this interval only to
+ * look at its released resources again. The hold ends only when a user clears
* {@code spec.suspend}, which arrives as a watch event and reconciles right away, so nothing
* waits for this interval. It is deliberately much coarser than {@link
* #RECONCILER_INTERVAL_SECONDS}, since a suspended resource has nothing to observe while each
@@ -280,7 +282,10 @@ public final class SparkOperatorConf {
.enableDynamicOverride(true)
.description(
"Requeue interval (in seconds) at which a resource held by spec.suspend is "
- + "reconciled, so that its SuspendHeld event is republished. The hold ends only "
+ + "reconciled, so that its SuspendHeld event is republished. A SparkCluster "
+ + "suspended while running publishes no such event, since its Suspended status "
+ + "says so, and uses this interval only to look at its released resources "
+ + "again. The hold ends only "
+ "when a user clears spec.suspend, which arrives as a watch event and "
+ "reconciles right away, so nothing waits for this interval. It is "
+ "deliberately much coarser than "
@@ -298,7 +303,8 @@ public final class SparkOperatorConf {
/**
* When enabled, operator would trim state transition history when a new attempt starts, keeping
- * previous attempt summary only.
+ * previous attempt summary only. It also drops the history of a suspended SparkCluster when it is
+ * resumed.
*/
public static final ConfigOption<Boolean> TRIM_ATTEMPT_STATE_TRANSITION_HISTORY =
ConfigOption.<Boolean>builder()
@@ -306,7 +312,8 @@ public final class SparkOperatorConf {
.enableDynamicOverride(true)
.description(
"When enabled, operator would trim state transition history when a "
- + "new attempt starts, keeping previous attempt summary only.")
+ + "new attempt starts, keeping previous attempt summary only. It also drops "
+ + "the history of a suspended SparkCluster when it is resumed.")
.typeParameterClass(Boolean.class)
.defaultValue(true)
.build();
diff --git a/spark-operator/src/main/java/org/apache/spark/k8s/operator/kueue/KueueWorkloadUtils.java b/spark-operator/src/main/java/org/apache/spark/k8s/operator/kueue/KueueWorkloadUtils.java
index 74feb74f..aae7c15e 100644
--- a/spark-operator/src/main/java/org/apache/spark/k8s/operator/kueue/KueueWorkloadUtils.java
+++ b/spark-operator/src/main/java/org/apache/spark/k8s/operator/kueue/KueueWorkloadUtils.java
@@ -519,9 +519,12 @@ public final class KueueWorkloadUtils {
* Deletes the Workload of the given resource and reports whether one was there to delete, unlike
* {@link #releaseWorkload} without swallowing the failure.
*
+ * @param client The KubernetesClient.
+ * @param owner The SparkApplication or SparkCluster owning the Workload.
+ * @return True if a Workload was deleted, false if there was none.
* @throws KubernetesClientException if the Workload cannot be deleted.
*/
- private static boolean deleteWorkloadOf(final KubernetesClient client, final HasMetadata owner) {
+ public static boolean deleteWorkloadOf(final KubernetesClient client, final HasMetadata owner) {
return !client
.resources(Workload.class)
.inNamespace(owner.getMetadata().getNamespace())
diff --git a/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/SparkClusterReconciler.java b/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/SparkClusterReconciler.java
index 26970380..b1c2833c 100644
--- a/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/SparkClusterReconciler.java
+++ b/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/SparkClusterReconciler.java
@@ -190,9 +190,8 @@ public class SparkClusterReconciler implements Reconciler<SparkCluster>, Cleaner
steps.add(new ClusterTerminatedStep());
switch (cluster.getStatus().getCurrentState().getCurrentStateSummary()) {
case Submitted -> steps.add(new ClusterInitStep());
- case RunningHealthy -> {
- // There is nothing to do because Spark Cluster is supposed to run infinitely.
- }
+ // Spark Cluster is supposed to run infinitely, until it is suspended by spec.suspend.
+ case RunningHealthy, Suspended -> steps.add(new ClusterSuspendStep());
case SchedulingFailure, Failed, ResourceReleased ->
steps.add(new ClusterUnknownStateStep());
}
diff --git a/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterInitStep.java b/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterInitStep.java
index 4f3d6af6..a91ca578 100644
--- a/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterInitStep.java
+++ b/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterInitStep.java
@@ -21,9 +21,11 @@ package org.apache.spark.k8s.operator.reconciler.reconcilesteps;
import static org.apache.spark.k8s.operator.Constants.CLUSTER_READY_MESSAGE;
import static org.apache.spark.k8s.operator.Constants.CLUSTER_SCHEDULE_FAILURE_MESSAGE;
+import static org.apache.spark.k8s.operator.Constants.CLUSTER_SUSPENDED_MESSAGE;
import static org.apache.spark.k8s.operator.reconciler.ReconcileProgress.*;
import static org.apache.spark.k8s.operator.status.ClusterStateSummary.RunningHealthy;
import static org.apache.spark.k8s.operator.status.ClusterStateSummary.SchedulingFailure;
+import static org.apache.spark.k8s.operator.status.ClusterStateSummary.Suspended;
import static org.apache.spark.k8s.operator.utils.SparkExceptionUtils.buildGeneralErrorMessage;
import java.time.Duration;
@@ -43,6 +45,8 @@ import org.apache.spark.k8s.operator.kueue.KueueWorkloadUtils;
import org.apache.spark.k8s.operator.reconciler.ReconcileProgress;
import org.apache.spark.k8s.operator.status.ClusterState;
import org.apache.spark.k8s.operator.status.ClusterStatus;
+import org.apache.spark.k8s.operator.utils.EventUtils;
+import org.apache.spark.k8s.operator.utils.ReconcilerUtils;
import org.apache.spark.k8s.operator.utils.SparkClusterStatusRecorder;
/** Request cluster master and its resources when starting an attempt. */
@@ -77,6 +81,13 @@ public final class ClusterInitStep extends ClusterReconcileStep {
return completeAndDefaultRequeue();
}
if (!masterRequested) {
+ // Unlike a first attempt, a cluster resumed from Suspended has persisted this Submitted
+ // state, which would keep saying that it is resumed. It goes back to Suspended instead,
+ // where ClusterSuspendStep releases whatever it holds only after its pods are gone.
+ if (cluster.getStatus().getStateTransitionHistory().lastKey() > 0) {
+ return appendStateAndImmediateRequeue(
+ context, statusRecorder, new ClusterState(Suspended, CLUSTER_SUSPENDED_MESSAGE));
+ }
return SuspendUtils.holdForSuspend(context, "master and workers");
}
}
@@ -151,19 +162,49 @@ public final class ClusterInitStep extends ClusterReconcileStep {
.appendNewState(new ClusterState(RunningHealthy, CLUSTER_READY_MESSAGE));
statusRecorder.persistStatus(context, updatedStatus);
return completeAndDefaultRequeue();
- } catch (Exception e) {
- if (log.isErrorEnabled()) {
- log.error("Failed to request master resource.", e);
+ } catch (KubernetesClientException e) {
+ if (ReconcilerUtils.isRetryableError(e)) {
+ // SchedulingFailure is terminal for a cluster, so a request which may yet succeed is sent
+ // again, e.g. for a cluster resumed from Suspended, while a rejected one fails the cluster.
+ log.warn("Failed to request master resource, will retry.", e);
+ if (!ReconcilerUtils.isTransientError(e)) {
+ // Unlike SchedulingFailure, the retry leaves nothing in the status, so a failure which
+ // keeps coming back, e.g. from an admission webhook which is down, is reported.
+ EventUtils.warn(
+ context.getEventRecorder(),
+ EventUtils.REASON_CLUSTER_REQUEST_FAILED,
+ "Failed to request the master and workers of the cluster, will retry. "
+ + EventUtils.describe(e));
+ }
+ return completeAndDefaultRequeue();
}
- String msg = CLUSTER_SCHEDULE_FAILURE_MESSAGE + " StackTrace: " + buildGeneralErrorMessage(e);
- statusRecorder.persistStatus(
- context,
- context
- .getResource()
- .getStatus()
- .appendNewState(new ClusterState(SchedulingFailure, msg)));
- return completeAndImmediateRequeue();
+ return failScheduling(context, statusRecorder, e);
+ } catch (Exception e) {
+ return failScheduling(context, statusRecorder, e);
+ }
+ }
+
+ /**
+ * Fails the cluster with SchedulingFailure for the given failure of requesting its resources.
+ *
+ * @param context The SparkClusterContext for the cluster.
+ * @param statusRecorder The SparkClusterStatusRecorder for recording status updates.
+ * @param e The failure.
+ * @return The ReconcileProgress indicating an immediate re-queue.
+ */
+ private ReconcileProgress failScheduling(
+ SparkClusterContext context, SparkClusterStatusRecorder statusRecorder, Exception e) {
+ if (log.isErrorEnabled()) {
+ log.error("Failed to request master resource.", e);
}
+ String msg = CLUSTER_SCHEDULE_FAILURE_MESSAGE + " StackTrace: " + buildGeneralErrorMessage(e);
+ statusRecorder.persistStatus(
+ context,
+ context
+ .getResource()
+ .getStatus()
+ .appendNewState(new ClusterState(SchedulingFailure, msg)));
+ return completeAndImmediateRequeue();
}
/**
diff --git a/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterReconcileStep.java b/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterReconcileStep.java
index c033a510..52846b96 100644
--- a/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterReconcileStep.java
+++ b/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterReconcileStep.java
@@ -29,7 +29,7 @@ import org.apache.spark.k8s.operator.utils.SparkClusterStatusRecorder;
/** Basic reconcile step for cluster. */
public abstract sealed class ClusterReconcileStep
- permits ClusterInitStep, ClusterTerminatedStep, ClusterUnknownStateStep,
+ permits ClusterInitStep, ClusterSuspendStep, ClusterTerminatedStep, ClusterUnknownStateStep,
ClusterValidateStep {
/**
* Reconciles a specific step for a Spark cluster.
@@ -48,15 +48,19 @@ public abstract sealed class ClusterReconcileStep
* @param statusRecorder The SparkClusterStatusRecorder for recording status updates.
* @param updatedStatus The updated ClusterStatus.
* @param requeueAfter The duration after which to re-queue.
- * @return The ReconcileProgress indicating the re-queue.
+ * @return The ReconcileProgress indicating the re-queue, or the default re-queue if the status
+ * is not persisted.
*/
protected ReconcileProgress updateStatusAndRequeueAfter(
SparkClusterContext context,
SparkClusterStatusRecorder statusRecorder,
ClusterStatus updatedStatus,
Duration requeueAfter) {
- statusRecorder.persistStatus(context, updatedStatus);
- return ReconcileProgress.completeAndRequeueAfter(requeueAfter);
+ // A rejected update, e.g. of a state unknown to an older CRD, would be rejected again right
+ // away, so it is retried with the default interval, not immediately.
+ return statusRecorder.persistStatus(context, updatedStatus)
+ ? ReconcileProgress.completeAndRequeueAfter(requeueAfter)
+ : ReconcileProgress.completeAndDefaultRequeue();
}
/**
@@ -67,15 +71,19 @@ public abstract sealed class ClusterReconcileStep
* @param statusRecorder The SparkClusterStatusRecorder for recording status updates.
* @param newState The new ClusterState to append.
* @param requeueAfter The duration after which to re-queue.
- * @return The ReconcileProgress indicating the re-queue.
+ * @return The ReconcileProgress indicating the re-queue, or the default re-queue if the state
+ * is not persisted.
*/
protected ReconcileProgress appendStateAndRequeueAfter(
SparkClusterContext context,
SparkClusterStatusRecorder statusRecorder,
ClusterState newState,
Duration requeueAfter) {
- statusRecorder.appendNewStateAndPersist(context, newState);
- return ReconcileProgress.completeAndRequeueAfter(requeueAfter);
+ // A rejected update, e.g. of a state unknown to an older CRD, would be rejected again right
+ // away, so it is retried with the default interval, not immediately.
+ return statusRecorder.appendNewStateAndPersist(context, newState)
+ ? ReconcileProgress.completeAndRequeueAfter(requeueAfter)
+ : ReconcileProgress.completeAndDefaultRequeue();
}
/**
diff --git a/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStep.java b/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStep.java
new file mode 100644
index 00000000..511e6b3b
--- /dev/null
+++ b/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStep.java
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.reconciler.reconcilesteps;
+
+import static org.apache.spark.k8s.operator.Constants.CLUSTER_RESUMED_MESSAGE;
+import static org.apache.spark.k8s.operator.Constants.CLUSTER_SUSPENDED_MESSAGE;
+import static org.apache.spark.k8s.operator.Constants.LABEL_SPARK_CLUSTER_NAME;
+import static org.apache.spark.k8s.operator.Constants.LABEL_SPARK_ROLE_MASTER_VALUE;
+import static org.apache.spark.k8s.operator.Constants.LABEL_SPARK_ROLE_NAME;
+import static org.apache.spark.k8s.operator.Constants.LABEL_SPARK_ROLE_WORKER_VALUE;
+import static org.apache.spark.k8s.operator.SparkClusterResourceSpec.getMasterStatefulSetName;
+import static org.apache.spark.k8s.operator.SparkClusterResourceSpec.getWorkerHorizontalPodAutoscalerName;
+import static org.apache.spark.k8s.operator.SparkClusterResourceSpec.getWorkerPodDisruptionBudgetName;
+import static org.apache.spark.k8s.operator.SparkClusterResourceSpec.getWorkerStatefulSetName;
+import static org.apache.spark.k8s.operator.config.SparkOperatorConf.KUEUE_WORKLOAD_INFORMER_ENABLED;
+import static org.apache.spark.k8s.operator.config.SparkOperatorConf.SUSPEND_HOLD_REQUEUE_INTERVAL_SECONDS;
+import static org.apache.spark.k8s.operator.config.SparkOperatorConf.TRIM_ATTEMPT_STATE_TRANSITION_HISTORY;
+import static org.apache.spark.k8s.operator.reconciler.ReconcileProgress.completeAndDefaultRequeue;
+import static org.apache.spark.k8s.operator.reconciler.ReconcileProgress.completeAndRequeueAfter;
+import static org.apache.spark.k8s.operator.reconciler.ReconcileProgress.proceed;
+
+import java.time.Duration;
+
+import io.fabric8.kubernetes.api.model.ListOptionsBuilder;
+import io.fabric8.kubernetes.api.model.PodList;
+import io.fabric8.kubernetes.client.KubernetesClient;
+import io.fabric8.kubernetes.client.KubernetesClientException;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.spark.k8s.operator.SparkCluster;
+import org.apache.spark.k8s.operator.context.SparkClusterContext;
+import org.apache.spark.k8s.operator.kueue.KueueWorkloadFactory;
+import org.apache.spark.k8s.operator.kueue.KueueWorkloadUtils;
+import org.apache.spark.k8s.operator.reconciler.ReconcileProgress;
+import org.apache.spark.k8s.operator.status.ClusterState;
+import org.apache.spark.k8s.operator.status.ClusterStateSummary;
+import org.apache.spark.k8s.operator.utils.EventUtils;
+import org.apache.spark.k8s.operator.utils.ReconcilerUtils;
+import org.apache.spark.k8s.operator.utils.SparkClusterStatusRecorder;
+import org.apache.spark.k8s.operator.utils.StringUtils;
+
+/**
+ * Suspends a running cluster by spec.suspend, and resumes it. A cluster runs until it is deleted,
+ * so suspending it is the only way to give its quota back without deleting it.
+ *
+ * <p>A running cluster enters Suspended before anything is released, and the master and workers
+ * are released only in Suspended. Releasing them first would leave a cluster without them in
+ * RunningHealthy when spec.suspend is cleared in the meantime, and nothing recreates them there.
+ * A resumed cluster leaves Suspended only after everything is released, and then starts over from
+ * Submitted, where {@link ClusterInitStep} applies all of its resources again and requests a new
+ * Kueue admission. Resuming earlier would let it keep the StatefulSets or the admitted Workload of
+ * the previous spec, so that a spec changed in the meantime would run on the old admission.
+ */
+@Slf4j
+public final class ClusterSuspendStep extends ClusterReconcileStep {
+ /**
+ * Reconciles a running or suspended cluster against spec.suspend.
+ *
+ * @param context The SparkClusterContext for the cluster.
+ * @param statusRecorder The SparkClusterStatusRecorder for recording status updates.
+ * @return The ReconcileProgress indicating the next step.
+ */
+ @Override
+ public ReconcileProgress reconcile(
+ SparkClusterContext context, SparkClusterStatusRecorder statusRecorder) {
+ SparkCluster cluster = context.getResource();
+ boolean suspend = cluster.getSpec().isSuspend();
+ ClusterStateSummary summary = cluster.getStatus().getCurrentState().getCurrentStateSummary();
+ if (summary == ClusterStateSummary.RunningHealthy && suspend) {
+ return appendStateAndImmediateRequeue(
+ context,
+ statusRecorder,
+ new ClusterState(ClusterStateSummary.Suspended, CLUSTER_SUSPENDED_MESSAGE));
+ }
+ if (summary == ClusterStateSummary.Suspended) {
+ if (!releaseResources(context)) {
+ return completeAndDefaultRequeue();
+ }
+ if (!suspend) {
+ // Like a restarted application attempt, the resumed cluster drops the history of the run
+ // which is over, so that suspending and resuming it again and again keeps it bounded.
+ return updateStatusAndRequeueAfter(
+ context,
+ statusRecorder,
+ cluster
+ .getStatus()
+ .appendNewState(
+ new ClusterState(ClusterStateSummary.Submitted, CLUSTER_RESUMED_MESSAGE),
+ TRIM_ATTEMPT_STATE_TRANSITION_HISTORY.getValue()),
+ Duration.ZERO);
+ }
+ return completeAndRequeueAfter(
+ Duration.ofSeconds(SUSPEND_HOLD_REQUEUE_INTERVAL_SECONDS.getValue()));
+ }
+ return proceed();
+ }
+
+ /**
+ * Releases the master and workers of a suspended cluster, and then its Kueue Workload. The
+ * Services and the NetworkPolicy are kept, since they hold no quota and are applied again on
+ * resume. Everything is deleted by name rather than from the current spec. The spec may no longer
+ * ask for the HorizontalPodAutoscaler or the PodDisruptionBudget of the workers, and it may not
+ * even build after being edited while suspended, which would otherwise hold the pods and the
+ * quota until the spec is fixed.
+ *
+ * <p>The Workload is released only after the master and worker pods are gone, since Kueue would
+ * admit another workload into the quota which the terminating pods still occupy. Other pods which
+ * carry the cluster label, e.g. to reach the workers through their NetworkPolicy, do not count.
+ * Everything here is idempotent, so pods that are not gone yet, or a release that failed, are
+ * simply looked at again. A failure which is not expected to clear on its own is reported, since
+ * Suspended is a steady state which would not tell it apart from waiting for the pods to go.
+ *
+ * @param context The SparkClusterContext for the cluster.
+ * @return True once everything is released, false while pods remain or a release failed.
+ */
+ private boolean releaseResources(SparkClusterContext context) {
+ SparkCluster cluster = context.getResource();
+ String namespace = cluster.getMetadata().getNamespace();
+ String name = cluster.getMetadata().getName();
+ KubernetesClient client = context.getClient();
+ try {
+ client
+ .apps()
+ .statefulSets()
+ .inNamespace(namespace)
+ .withName(getWorkerStatefulSetName(name))
+ .delete();
+ client
+ .apps()
+ .statefulSets()
+ .inNamespace(namespace)
+ .withName(getMasterStatefulSetName(name))
+ .delete();
+ client
+ .autoscaling()
+ .v2()
+ .horizontalPodAutoscalers()
+ .inNamespace(namespace)
+ .withName(getWorkerHorizontalPodAutoscalerName(name))
+ .delete();
+ client
+ .policy()
+ .v1()
+ .podDisruptionBudget()
+ .inNamespace(namespace)
+ .withName(getWorkerPodDisruptionBudgetName(name))
+ .delete();
+ // Only whether any pod remains matters, so one is enough. A page may come back empty with a
+ // continue token, which still means that more pods remain.
+ PodList pods =
+ client
+ .pods()
+ .inNamespace(namespace)
+ .withLabel(LABEL_SPARK_CLUSTER_NAME, name)
+ .withLabelIn(
+ LABEL_SPARK_ROLE_NAME,
+ LABEL_SPARK_ROLE_MASTER_VALUE,
+ LABEL_SPARK_ROLE_WORKER_VALUE)
+ .list(new ListOptionsBuilder().withLimit(1L).build());
+ boolean morePods = StringUtils.isNotEmpty(pods.getMetadata().getContinue());
+ if (!pods.getItems().isEmpty() || morePods) {
+ // The deletion of each pod is observed by the pod informer, which reconciles again.
+ log.debug("Waiting for the pods of the suspended cluster to be deleted.");
+ return false;
+ }
+ // A Workload admitted before its queue label was removed is released as well, whenever the
+ // operator may access Workloads, which the Workload informer requires too.
+ if (KueueWorkloadFactory.hasQueueName(cluster)
+ || KUEUE_WORKLOAD_INFORMER_ENABLED.getValue()) {
+ KueueWorkloadUtils.deleteWorkloadOf(client, cluster);
+ }
+ } catch (KubernetesClientException e) {
+ log.warn("Failed to release the resources of the suspended cluster, will retry.", e);
+ if (!ReconcilerUtils.isTransientError(e)) {
+ EventUtils.warn(
+ context.getEventRecorder(),
+ EventUtils.REASON_SUSPEND_RELEASE_FAILED,
+ "Failed to release the master and workers or the Kueue Workload of the suspended "
+ + "cluster, will retry. "
+ + EventUtils.describe(e));
+ }
+ return false;
+ }
+ return true;
+ }
+}
diff --git a/spark-operator/src/main/java/org/apache/spark/k8s/operator/utils/EventUtils.java b/spark-operator/src/main/java/org/apache/spark/k8s/operator/utils/EventUtils.java
index 110c0b79..5600c505 100644
--- a/spark-operator/src/main/java/org/apache/spark/k8s/operator/utils/EventUtils.java
+++ b/spark-operator/src/main/java/org/apache/spark/k8s/operator/utils/EventUtils.java
@@ -49,9 +49,21 @@ public final class EventUtils {
/** Reason for an event describing an unhandled error thrown out of a cleanup. */
public static final String REASON_CLEANUP_ERROR = "CleanupError";
+ /**
+ * Reason for an event describing a failure to request the resources of a SparkCluster, which is
+ * retried.
+ */
+ public static final String REASON_CLUSTER_REQUEST_FAILED = "ClusterRequestFailed";
+
/** Reason for an event describing that a resource is held by {@code spec.suspend}. */
public static final String REASON_SUSPEND_HELD = "SuspendHeld";
+ /**
+ * Reason for an event describing a failure to release the resources of a SparkCluster which is
+ * suspended while running.
+ */
+ public static final String REASON_SUSPEND_RELEASE_FAILED = "SuspendReleaseFailed";
+
/** Reason for an event describing that a resource waits for Kueue to admit its Workload. */
public static final String REASON_KUEUE_ADMISSION_PENDING = "KueueAdmissionPending";
diff --git a/spark-operator/src/main/java/org/apache/spark/k8s/operator/utils/ReconcilerUtils.java b/spark-operator/src/main/java/org/apache/spark/k8s/operator/utils/ReconcilerUtils.java
index b5e194df..0779450a 100644
--- a/spark-operator/src/main/java/org/apache/spark/k8s/operator/utils/ReconcilerUtils.java
+++ b/spark-operator/src/main/java/org/apache/spark/k8s/operator/utils/ReconcilerUtils.java
@@ -251,9 +251,7 @@ public final class ReconcilerUtils {
if (e.getCode() == HTTP_NOT_FOUND) {
return Optional.empty();
}
- if (isTransientError(e)
- || e.getCode() == HTTP_INTERNAL_ERROR
- || e.getCode() == Constants.HTTP_TOO_MANY_REQUESTS) {
+ if (isRetryableError(e)) {
log.warn(
"Failed to read the resource with responseCode={}, considering it absent.",
e.getCode(),
@@ -332,6 +330,20 @@ public final class ReconcilerUtils {
};
}
+ /**
+ * Whether the same request may yet succeed when it is sent again. Besides a transient failure,
+ * a throttled request and an internal server error, such as an admission webhook which is down,
+ * qualify, while a request the API server rejected, e.g. as forbidden or invalid, does not.
+ *
+ * @param e The failure to classify.
+ * @return True if the request is worth sending again, false otherwise.
+ */
+ public static boolean isRetryableError(KubernetesClientException e) {
+ return isTransientError(e)
+ || e.getCode() == HTTP_INTERNAL_ERROR
+ || e.getCode() == Constants.HTTP_TOO_MANY_REQUESTS;
+ }
+
/**
* Whether the given failure left the request unanswered, so that asking again may yet work. It
* is defined as the complement of the status-less failures that repeating cannot change: a
diff --git a/spark-operator/src/main/java/org/apache/spark/k8s/operator/utils/SparkClusterStatusRecorder.java b/spark-operator/src/main/java/org/apache/spark/k8s/operator/utils/SparkClusterStatusRecorder.java
index fa497935..d033781b 100644
--- a/spark-operator/src/main/java/org/apache/spark/k8s/operator/utils/SparkClusterStatusRecorder.java
+++ b/spark-operator/src/main/java/org/apache/spark/k8s/operator/utils/SparkClusterStatusRecorder.java
@@ -44,9 +44,10 @@ public class SparkClusterStatusRecorder
*
* @param context The SparkClusterContext for the cluster.
* @param newState The new ClusterState to append.
+ * @return true if the status is successfully patched.
*/
- public void appendNewStateAndPersist(SparkClusterContext context, ClusterState newState) {
+ public boolean appendNewStateAndPersist(SparkClusterContext context, ClusterState newState) {
ClusterStatus updatedStatus = context.getResource().getStatus().appendNewState(newState);
- persistStatus(context, updatedStatus);
+ return persistStatus(context, updatedStatus);
}
}
diff --git a/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/SparkClusterReconcilerTest.java b/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/SparkClusterReconcilerTest.java
index 98cfd181..bfce2ddb 100644
--- a/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/SparkClusterReconcilerTest.java
+++ b/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/SparkClusterReconcilerTest.java
@@ -104,7 +104,7 @@ class SparkClusterReconcilerTest {
ClusterStatus updatedStatus =
cluster.getStatus().appendNewState(invocation.getArgument(1));
cluster.setStatus(updatedStatus);
- return null;
+ return true;
})
.when(mockRecorder)
.appendNewStateAndPersist(any(SparkClusterContext.class), any(ClusterState.class));
@@ -183,15 +183,32 @@ class SparkClusterReconcilerTest {
@Test
void testGetReconcileStepsForRunningHealthyCluster() {
- // RunningHealthy state should not have additional steps beyond validation and termination check
+ // RunningHealthy state should include ClusterSuspendStep to observe spec.suspend
cluster.setStatus(cluster.getStatus().appendNewState(
new ClusterState(ClusterStateSummary.RunningHealthy, "")));
List<ClusterReconcileStep> steps = reconciler.getReconcileSteps(cluster);
- assertEquals(2, steps.size());
+ assertEquals(3, steps.size());
+ assertEquals(
+ "ClusterValidateStep", steps.get(0).getClass().getSimpleName());
+ assertEquals(
+ "ClusterTerminatedStep", steps.get(1).getClass().getSimpleName());
+ assertEquals(
+ "ClusterSuspendStep", steps.get(2).getClass().getSimpleName());
+ }
+
+ @Test
+ void testGetReconcileStepsForSuspendedCluster() {
+ // Suspended state should include ClusterSuspendStep to release resources or resume
+ cluster.setStatus(cluster.getStatus().appendNewState(
+ new ClusterState(ClusterStateSummary.Suspended, "")));
+ List<ClusterReconcileStep> steps = reconciler.getReconcileSteps(cluster);
+ assertEquals(3, steps.size());
assertEquals(
"ClusterValidateStep", steps.get(0).getClass().getSimpleName());
assertEquals(
"ClusterTerminatedStep", steps.get(1).getClass().getSimpleName());
+ assertEquals(
+ "ClusterSuspendStep", steps.get(2).getClass().getSimpleName());
}
@Test
diff --git a/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterInitStepTest.java b/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterInitStepTest.java
index 1dbdc0b5..c4c5651d 100644
--- a/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterInitStepTest.java
+++ b/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterInitStepTest.java
@@ -65,6 +65,8 @@ import io.javaoperatorsdk.operator.api.event.ResourceEventRecorder;
import io.javaoperatorsdk.operator.api.reconciler.Context;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.CsvSource;
import org.mockito.ArgumentCaptor;
import org.mockito.InOrder;
@@ -233,6 +235,106 @@ class ClusterInitStepTest {
Assertions.assertEquals(EventUtils.REASON_SUSPEND_HELD, captureEvents(1).get(0).reason());
}
+ @ParameterizedTest
+ @CsvSource({"429, 1", "500, 1", "503, 0"})
+ void retryableFailureOfRequestingResourcesIsRetried(int code, int events) {
+ ClusterInitStep clusterInitStep = new ClusterInitStep();
+ SparkClusterContext mockContext = mock(SparkClusterContext.class);
+ SparkClusterStatusRecorder recorder = mock(SparkClusterStatusRecorder.class);
+ SparkCluster cluster = buildCluster();
+ stubFailingServiceApply(
+ mockContext, cluster, new KubernetesClientException("failed", code, null));
+
+ // A cluster resumed from Suspended goes through this path again, so a request which may yet
+ // succeed, e.g. while an admission webhook is down, must not fail it with SchedulingFailure
+ Assertions.assertEquals(
+ ReconcileProgress.completeAndDefaultRequeue(),
+ clusterInitStep.reconcile(mockContext, recorder));
+ verifyNoInteractions(recorder);
+ Assertions.assertEquals(
+ ClusterStateSummary.Submitted,
+ cluster.getStatus().getCurrentState().getCurrentStateSummary());
+ // The retry leaves nothing in the status, so it is reported unless it may clear on its own
+ for (EventRecord event : captureEvents(events)) {
+ Assertions.assertEquals(EventType.WARNING, event.type());
+ Assertions.assertEquals(EventUtils.REASON_CLUSTER_REQUEST_FAILED, event.reason());
+ }
+ }
+
+ @Test
+ void rejectedRequestOfResourcesFailsScheduling() {
+ ClusterInitStep clusterInitStep = new ClusterInitStep();
+ SparkClusterContext mockContext = mock(SparkClusterContext.class);
+ SparkClusterStatusRecorder recorder = mock(SparkClusterStatusRecorder.class);
+ SparkCluster cluster = buildCluster();
+ stubFailingServiceApply(
+ mockContext, cluster, new KubernetesClientException("Invalid", 422, null));
+
+ // A rejected request would be rejected again, so it is reported as SchedulingFailure rather
+ // than retried with the status staying empty
+ Assertions.assertEquals(
+ ReconcileProgress.completeAndImmediateRequeue(),
+ clusterInitStep.reconcile(mockContext, recorder));
+ ArgumentCaptor<ClusterStatus> statusCaptor = ArgumentCaptor.forClass(ClusterStatus.class);
+ verify(recorder).persistStatus(any(), statusCaptor.capture());
+ Assertions.assertEquals(
+ ClusterStateSummary.SchedulingFailure,
+ statusCaptor.getValue().getCurrentState().getCurrentStateSummary());
+ // The status already says why, so no event is needed
+ verifyNoInteractions(eventRecorder);
+ }
+
+ @Test
+ void resumedClusterSuspendedAgainGoesBackToSuspended() {
+ ClusterInitStep clusterInitStep = new ClusterInitStep();
+ SparkClusterContext mockContext = mock(SparkClusterContext.class);
+ SparkClusterStatusRecorder recorder = mock(SparkClusterStatusRecorder.class);
+ when(recorder.appendNewStateAndPersist(any(), any())).thenReturn(true);
+ SparkCluster cluster = buildCluster();
+ // Resumed from Suspended: the persisted Submitted state follows the states of the earlier run
+ cluster.setStatus(
+ cluster
+ .getStatus()
+ .appendNewState(new ClusterState(ClusterStateSummary.Suspended, ""))
+ .appendNewState(
+ new ClusterState(ClusterStateSummary.Submitted, Constants.CLUSTER_RESUMED_MESSAGE),
+ true));
+ cluster.getSpec().setSuspend(true);
+ KubernetesClient mockClient = mock(KubernetesClient.class, RETURNS_DEEP_STUBS);
+ when(mockClient.resource(masterStatefulSetSpec).get()).thenReturn(null);
+ when(mockContext.getResource()).thenReturn(cluster);
+ when(mockContext.getClient()).thenReturn(mockClient);
+ when(mockContext.getMasterStatefulSetSpec()).thenReturn(masterStatefulSetSpec);
+
+ // Its status would otherwise keep saying that it is resumed, while ClusterSuspendStep releases
+ // its Workload only once its pods are gone
+ Assertions.assertEquals(
+ ReconcileProgress.completeAndImmediateRequeue(),
+ clusterInitStep.reconcile(mockContext, recorder));
+ ArgumentCaptor<ClusterState> stateCaptor = ArgumentCaptor.forClass(ClusterState.class);
+ verify(recorder).appendNewStateAndPersist(any(), stateCaptor.capture());
+ Assertions.assertEquals(
+ ClusterStateSummary.Suspended, stateCaptor.getValue().getCurrentStateSummary());
+ Assertions.assertEquals(
+ Constants.CLUSTER_SUSPENDED_MESSAGE, stateCaptor.getValue().getMessage());
+ verify(mockContext, never()).getEventRecorder();
+ }
+
+ @SuppressWarnings("unchecked")
+ private void stubFailingServiceApply(
+ SparkClusterContext mockContext, SparkCluster cluster, KubernetesClientException failure) {
+ KubernetesClient mockClient = mock(KubernetesClient.class, RETURNS_DEEP_STUBS);
+ ServerSideApplicable<Service> serviceApplicable = mock(ServerSideApplicable.class);
+ ServiceResource<Service> serviceResource = mock(ServiceResource.class);
+ when(serviceResource.forceConflicts()).thenReturn(serviceApplicable);
+ when(serviceApplicable.serverSideApply()).thenThrow(failure);
+ when(mockClient.services().resource(any(Service.class))).thenReturn(serviceResource);
+ when(mockContext.getResource()).thenReturn(cluster);
+ when(mockContext.getClient()).thenReturn(mockClient);
+ when(mockContext.getMasterServiceSpec()).thenReturn(service("cluster1-master-svc"));
+ when(mockContext.getEventRecorder()).thenReturn(eventRecorder);
+ }
+
@Test
@SuppressWarnings("unchecked")
void suspendAfterMasterRequestedCompletesInitialization() {
diff --git a/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStepTest.java b/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStepTest.java
new file mode 100644
index 00000000..645c5638
--- /dev/null
+++ b/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStepTest.java
@@ -0,0 +1,541 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.reconciler.reconcilesteps;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoInteractions;
+import static org.mockito.Mockito.when;
+
+import java.time.Duration;
+import java.util.Map;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import io.fabric8.kubernetes.api.model.KubernetesResourceList;
+import io.fabric8.kubernetes.api.model.ListOptions;
+import io.fabric8.kubernetes.api.model.ObjectMetaBuilder;
+import io.fabric8.kubernetes.api.model.Pod;
+import io.fabric8.kubernetes.api.model.PodBuilder;
+import io.fabric8.kubernetes.api.model.PodList;
+import io.fabric8.kubernetes.api.model.PodListBuilder;
+import io.fabric8.kubernetes.api.model.Service;
+import io.fabric8.kubernetes.api.model.ServiceBuilder;
+import io.fabric8.kubernetes.api.model.apps.StatefulSet;
+import io.fabric8.kubernetes.api.model.apps.StatefulSetBuilder;
+import io.fabric8.kubernetes.api.model.autoscaling.v2.HorizontalPodAutoscaler;
+import io.fabric8.kubernetes.api.model.autoscaling.v2.HorizontalPodAutoscalerBuilder;
+import io.fabric8.kubernetes.api.model.policy.v1.PodDisruptionBudget;
+import io.fabric8.kubernetes.api.model.policy.v1.PodDisruptionBudgetBuilder;
+import io.fabric8.kubernetes.client.KubernetesClient;
+import io.fabric8.kubernetes.client.KubernetesClientException;
+import io.fabric8.kubernetes.client.dsl.FilterWatchListDeletable;
+import io.fabric8.kubernetes.client.dsl.MixedOperation;
+import io.fabric8.kubernetes.client.dsl.NonNamespaceOperation;
+import io.fabric8.kubernetes.client.dsl.PodResource;
+import io.fabric8.kubernetes.client.dsl.Resource;
+import io.fabric8.kubernetes.client.server.mock.EnableKubernetesMockClient;
+import io.javaoperatorsdk.operator.api.event.EventRecord;
+import io.javaoperatorsdk.operator.api.event.EventType;
+import io.javaoperatorsdk.operator.api.event.ResourceEventRecorder;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+import org.mockito.ArgumentCaptor;
+
+import org.apache.spark.k8s.operator.Constants;
+import org.apache.spark.k8s.operator.SparkCluster;
+import org.apache.spark.k8s.operator.config.SparkOperatorConf;
+import org.apache.spark.k8s.operator.context.SparkClusterContext;
+import org.apache.spark.k8s.operator.kueue.KueueWorkloadFactory;
+import org.apache.spark.k8s.operator.kueue.v1beta2.Workload;
+import org.apache.spark.k8s.operator.reconciler.ReconcileProgress;
+import org.apache.spark.k8s.operator.spec.ClusterSpec;
+import org.apache.spark.k8s.operator.spec.ClusterTolerations;
+import org.apache.spark.k8s.operator.spec.RuntimeVersions;
+import org.apache.spark.k8s.operator.spec.WorkerInstanceConfig;
+import org.apache.spark.k8s.operator.status.ClusterState;
+import org.apache.spark.k8s.operator.status.ClusterStateSummary;
+import org.apache.spark.k8s.operator.status.ClusterStatus;
+import org.apache.spark.k8s.operator.utils.EventUtils;
+import org.apache.spark.k8s.operator.utils.SparkClusterStatusRecorder;
+import org.apache.spark.k8s.operator.utils.TestUtils;
+
+@EnableKubernetesMockClient(crud = true)
+@SuppressFBWarnings(
+ value = {"UWF_UNWRITTEN_FIELD", "NP_UNWRITTEN_FIELD"},
+ justification = "Unwritten fields are covered by Kubernetes mock client")
+class ClusterSuspendStepTest {
+ private KubernetesClient kubernetesClient;
+
+ // The default of spark.kubernetes.operator.reconciler.suspendHoldRequeueIntervalSeconds
+ private static final ReconcileProgress SUSPEND_HOLD_PROGRESS =
+ ReconcileProgress.completeAndRequeueAfter(Duration.ofMinutes(30));
+
+ private final StatefulSet masterStatefulSetSpec = statefulSet("cluster1-master");
+ private final StatefulSet workerStatefulSetSpec = statefulSet("cluster1-worker");
+ private final SparkClusterContext mockContext = mock(SparkClusterContext.class);
+ private final SparkClusterStatusRecorder recorder = mock(SparkClusterStatusRecorder.class);
+ private final ResourceEventRecorder eventRecorder = mock(ResourceEventRecorder.class);
+
+ @Test
+ void runningClusterWithoutSuspendProceeds() {
+ SparkCluster cluster = buildCluster(ClusterStateSummary.RunningHealthy, false);
+ stubContext(cluster);
+ createRunningCluster();
+
+ Assertions.assertEquals(
+ ReconcileProgress.proceed(), new ClusterSuspendStep().reconcile(mockContext, recorder));
+
+ verifyNoInteractions(recorder);
+ Assertions.assertNotNull(get(masterStatefulSetSpec));
+ Assertions.assertNotNull(get(workerStatefulSetSpec));
+ }
+
+ @Test
+ void suspendingRunningClusterEntersSuspendedBeforeReleasingResources() {
+ SparkCluster cluster = buildCluster(ClusterStateSummary.RunningHealthy, true);
+ stubContext(cluster);
+ createRunningCluster();
+
+ Assertions.assertEquals(
+ ReconcileProgress.completeAndImmediateRequeue(),
+ new ClusterSuspendStep().reconcile(mockContext, recorder));
+
+ ClusterState state = captureAppendedState();
+ Assertions.assertEquals(ClusterStateSummary.Suspended, state.getCurrentStateSummary());
+ Assertions.assertEquals(Constants.CLUSTER_SUSPENDED_MESSAGE, state.getMessage());
+ // Nothing is released until Suspended is persisted, so that a cluster which is resumed in the
+ // meantime is never left in RunningHealthy without its master and workers.
+ Assertions.assertNotNull(get(masterStatefulSetSpec));
+ Assertions.assertNotNull(get(workerStatefulSetSpec));
+ }
+
+ @Test
+ void suspendedStateWhichIsNotPersistedIsRetriedLater() {
+ SparkCluster cluster = buildCluster(ClusterStateSummary.RunningHealthy, true);
+ stubContext(cluster);
+ createRunningCluster();
+ when(recorder.appendNewStateAndPersist(any(), any())).thenReturn(false);
+
+ // A status which is rejected, e.g. by a CRD without the Suspended state, is not retried at once
+ Assertions.assertEquals(
+ ReconcileProgress.completeAndDefaultRequeue(),
+ new ClusterSuspendStep().reconcile(mockContext, recorder));
+
+ Assertions.assertNotNull(get(masterStatefulSetSpec));
+ Assertions.assertNotNull(get(workerStatefulSetSpec));
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void suspendedClusterKeepsKueueWorkloadWhilePodsRemain(boolean suspend) {
+ // Resuming waits as well, so that the cluster starts over only once everything is released
+ SparkCluster cluster = buildKueueCluster(ClusterStateSummary.Suspended, suspend);
+ stubContext(cluster);
+ createRunningCluster();
+ kubernetesClient.resource(KueueWorkloadFactory.buildWorkload(cluster)).create();
+ Service masterService = service("cluster1-master-svc");
+ kubernetesClient.resource(masterService).create();
+
+ Assertions.assertEquals(
+ ReconcileProgress.completeAndDefaultRequeue(),
+ new ClusterSuspendStep().reconcile(mockContext, recorder));
+
+ Assertions.assertNull(get(masterStatefulSetSpec));
+ Assertions.assertNull(get(workerStatefulSetSpec));
+ // The pods are still terminating, so they keep holding the quota of the Workload
+ Assertions.assertNotNull(getWorkload());
+ // The Services hold no quota and are applied again on resume
+ Assertions.assertNotNull(kubernetesClient.resource(masterService).get());
+ verifyNoInteractions(recorder);
+ }
+
+ @Test
+ void suspendedClusterReleasesKueueWorkloadAfterPodsAreGone() {
+ SparkCluster cluster = buildKueueCluster(ClusterStateSummary.Suspended, true);
+ stubContext(cluster);
+ kubernetesClient.resource(KueueWorkloadFactory.buildWorkload(cluster)).create();
+
+ Assertions.assertEquals(
+ SUSPEND_HOLD_PROGRESS, new ClusterSuspendStep().reconcile(mockContext, recorder));
+
+ Assertions.assertNull(getWorkload());
+ verifyNoInteractions(recorder);
+ }
+
+ @Test
+ void suspendedClusterReleasesAutoscalerAndDisruptionBudgetByName() {
+ SparkCluster cluster = buildCluster(ClusterStateSummary.Suspended, true);
+ stubContext(cluster);
+ HorizontalPodAutoscaler hpa =
+ new HorizontalPodAutoscalerBuilder()
+ .withNewMetadata()
+ .withName("cluster1-worker-hpa")
+ .withNamespace("default")
+ .endMetadata()
+ .build();
+ PodDisruptionBudget pdb =
+ new PodDisruptionBudgetBuilder()
+ .withNewMetadata()
+ .withName("cluster1-worker-pdb")
+ .withNamespace("default")
+ .endMetadata()
+ .build();
+ kubernetesClient.resource(hpa).create();
+ kubernetesClient.resource(pdb).create();
+
+ // The current spec (minWorkers == maxWorkers) no longer asks for them, but they are released
+ Assertions.assertEquals(
+ SUSPEND_HOLD_PROGRESS, new ClusterSuspendStep().reconcile(mockContext, recorder));
+
+ Assertions.assertNull(kubernetesClient.resource(hpa).get());
+ Assertions.assertNull(kubernetesClient.resource(pdb).get());
+ verify(mockContext, never()).getHorizontalPodAutoscalerSpec();
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ void onlyMasterAndWorkerPodsHoldRelease() {
+ SparkCluster cluster = buildKueueCluster(ClusterStateSummary.Suspended, true);
+ KubernetesClient client = spy(kubernetesClient);
+ FilterWatchListDeletable<Pod, PodList, PodResource> rolePods = stubPodList(client);
+ // Other pods carry the cluster label too, e.g. a client pod which reaches the workers through
+ // their NetworkPolicy, but only the master and workers are listed, and none of them is left
+ when(rolePods.list(any(ListOptions.class)))
+ .thenReturn(new PodListBuilder().withNewMetadata().endMetadata().build());
+ stubContext(cluster, client);
+ kubernetesClient.resource(KueueWorkloadFactory.buildWorkload(cluster)).create();
+
+ Assertions.assertEquals(
+ SUSPEND_HOLD_PROGRESS, new ClusterSuspendStep().reconcile(mockContext, recorder));
+
+ Assertions.assertNull(getWorkload());
+ }
+
+ @Test
+ void workloadIsReleasedAfterQueueLabelIsRemoved() {
+ SparkCluster cluster = buildKueueCluster(ClusterStateSummary.Suspended, true);
+ stubContext(cluster);
+ kubernetesClient.resource(KueueWorkloadFactory.buildWorkload(cluster)).create();
+ cluster.getMetadata().setLabels(Map.of());
+ TestUtils.setConfigKey(SparkOperatorConf.KUEUE_WORKLOAD_INFORMER_ENABLED, true);
+ try {
+ Assertions.assertEquals(
+ SUSPEND_HOLD_PROGRESS, new ClusterSuspendStep().reconcile(mockContext, recorder));
+ } finally {
+ TestUtils.setConfigKey(SparkOperatorConf.KUEUE_WORKLOAD_INFORMER_ENABLED, false);
+ }
+
+ // The Workload admitted before the label was removed does not keep the quota
+ Assertions.assertNull(getWorkload());
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ void continueTokenMeansThatPodsRemain() {
+ SparkCluster cluster = buildKueueCluster(ClusterStateSummary.Suspended, true);
+ KubernetesClient client = spy(kubernetesClient);
+ FilterWatchListDeletable<Pod, PodList, PodResource> rolePods = stubPodList(client);
+ ArgumentCaptor<ListOptions> options = ArgumentCaptor.forClass(ListOptions.class);
+ // An empty page with a continue token, as a limited LIST with a label selector may return
+ when(rolePods.list(options.capture()))
+ .thenReturn(
+ new PodListBuilder().withNewMetadata().withContinue("next").endMetadata().build());
+ stubContext(cluster, client);
+ kubernetesClient.resource(KueueWorkloadFactory.buildWorkload(cluster)).create();
+
+ Assertions.assertEquals(
+ ReconcileProgress.completeAndDefaultRequeue(),
+ new ClusterSuspendStep().reconcile(mockContext, recorder));
+
+ Assertions.assertEquals(1L, options.getValue().getLimit());
+ Assertions.assertNotNull(getWorkload());
+ }
+
+ @Test
+ void suspendedClusterWhoseSpecCannotBeBuiltIsReleased() {
+ SparkCluster cluster = buildKueueCluster(ClusterStateSummary.Suspended, true);
+ stubContext(cluster);
+ // The spec may be edited into one which cannot be built while the cluster is suspended
+ IllegalArgumentException failure = new IllegalArgumentException("invalid spec");
+ when(mockContext.getMasterStatefulSetSpec()).thenThrow(failure);
+ when(mockContext.getWorkerStatefulSetSpec()).thenThrow(failure);
+ kubernetesClient.resource(masterStatefulSetSpec).create();
+ kubernetesClient.resource(workerStatefulSetSpec).create();
+ kubernetesClient.resource(KueueWorkloadFactory.buildWorkload(cluster)).create();
+
+ Assertions.assertEquals(
+ SUSPEND_HOLD_PROGRESS, new ClusterSuspendStep().reconcile(mockContext, recorder));
+
+ Assertions.assertNull(get(masterStatefulSetSpec));
+ Assertions.assertNull(get(workerStatefulSetSpec));
+ Assertions.assertNull(getWorkload());
+ }
+
+ @Test
+ void failedReleaseIsRetriedWithoutReleasingKueueWorkload() {
+ SparkCluster cluster = buildKueueCluster(ClusterStateSummary.Suspended, true);
+ KubernetesClient mockClient = mock(KubernetesClient.class);
+ when(mockClient.apps())
+ .thenThrow(new KubernetesClientException("Service Unavailable", 503, null));
+ stubContext(cluster, mockClient);
+
+ Assertions.assertEquals(
+ ReconcileProgress.completeAndDefaultRequeue(),
+ new ClusterSuspendStep().reconcile(mockContext, recorder));
+
+ verify(mockClient, never()).resources(Workload.class);
+ verifyNoInteractions(recorder);
+ // A failure which may clear on its own is not reported
+ verify(mockContext, never()).getEventRecorder();
+ }
+
+ @ParameterizedTest
+ @ValueSource(ints = {403, 429, 500})
+ void persistentReleaseFailureIsReportedAndRetried(int code) {
+ SparkCluster cluster = buildKueueCluster(ClusterStateSummary.Suspended, false);
+ KubernetesClient mockClient = mock(KubernetesClient.class);
+ // Unlike a timeout or a 503, these do not clear on their own, e.g. an admission webhook
+ // which is down keeps failing with 500, so they are reported
+ when(mockClient.apps()).thenThrow(new KubernetesClientException("failed", code, null));
+ stubContext(cluster, mockClient);
+ when(mockContext.getEventRecorder()).thenReturn(eventRecorder);
+
+ Assertions.assertEquals(
+ ReconcileProgress.completeAndDefaultRequeue(),
+ new ClusterSuspendStep().reconcile(mockContext, recorder));
+
+ ArgumentCaptor<EventRecord> event = ArgumentCaptor.forClass(EventRecord.class);
+ verify(eventRecorder).record(event.capture());
+ Assertions.assertEquals(EventType.WARNING, event.getValue().type());
+ Assertions.assertEquals(EventUtils.REASON_SUSPEND_RELEASE_FAILED, event.getValue().reason());
+ // It stays Suspended rather than resuming on what it still holds
+ verifyNoInteractions(recorder);
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ void failedKueueWorkloadReleaseIsRetried() {
+ SparkCluster cluster = buildKueueCluster(ClusterStateSummary.Suspended, false);
+ KubernetesClient client = spy(kubernetesClient);
+ MixedOperation<Workload, KubernetesResourceList<Workload>, Resource<Workload>> workloads =
+ mock(MixedOperation.class);
+ NonNamespaceOperation<Workload, KubernetesResourceList<Workload>, Resource<Workload>>
+ namespaced = mock(NonNamespaceOperation.class);
+ Resource<Workload> workload = mock(Resource.class);
+ doReturn(workloads).when(client).resources(Workload.class);
+ when(workloads.inNamespace("default")).thenReturn(namespaced);
+ when(namespaced.withName("sparkcluster-cluster1")).thenReturn(workload);
+ when(workload.delete())
+ .thenThrow(new KubernetesClientException("Service Unavailable", 503, null));
+ stubContext(cluster, client);
+
+ // The quota is not given back yet, so the cluster is neither held for the hold interval nor
+ // resumed on the Workload of the previous spec
+ Assertions.assertEquals(
+ ReconcileProgress.completeAndDefaultRequeue(),
+ new ClusterSuspendStep().reconcile(mockContext, recorder));
+
+ verifyNoInteractions(recorder);
+ }
+
+ @Test
+ void resumingSuspendedClusterReleasesEverythingBeforeSubmitted() {
+ SparkCluster cluster = buildKueueCluster(ClusterStateSummary.Suspended, false);
+ stubContext(cluster);
+ // Resumed before anything was released: the StatefulSets and the Workload are still there
+ kubernetesClient.resource(masterStatefulSetSpec).create();
+ kubernetesClient.resource(workerStatefulSetSpec).create();
+ kubernetesClient.resource(KueueWorkloadFactory.buildWorkload(cluster)).create();
+ long lastStateId = cluster.getStatus().getStateTransitionHistory().lastKey();
+
+ Assertions.assertEquals(
+ ReconcileProgress.completeAndImmediateRequeue(),
+ new ClusterSuspendStep().reconcile(mockContext, recorder));
+
+ ArgumentCaptor<ClusterStatus> status = ArgumentCaptor.forClass(ClusterStatus.class);
+ verify(recorder).persistStatus(eq(mockContext), status.capture());
+ ClusterState state = status.getValue().getCurrentState();
+ Assertions.assertEquals(ClusterStateSummary.Submitted, state.getCurrentStateSummary());
+ Assertions.assertEquals(Constants.CLUSTER_RESUMED_MESSAGE, state.getMessage());
+ // The history of the run which is over is dropped, so that it stays bounded
+ Assertions.assertEquals(
+ Map.of(lastStateId + 1, state), status.getValue().getStateTransitionHistory());
+ // Whatever the Workload was admitted for, ClusterInitStep requests a new admission from scratch
+ Assertions.assertNull(get(masterStatefulSetSpec));
+ Assertions.assertNull(get(workerStatefulSetSpec));
+ Assertions.assertNull(getWorkload());
+ }
+
+ @Test
+ void otherStatesProceed() {
+ for (ClusterStateSummary summary :
+ new ClusterStateSummary[] {
+ ClusterStateSummary.Submitted,
+ ClusterStateSummary.SchedulingFailure,
+ ClusterStateSummary.Failed,
+ ClusterStateSummary.ResourceReleased
+ }) {
+ SparkCluster cluster = buildCluster(summary, true);
+ stubContext(cluster);
+
+ Assertions.assertEquals(
+ ReconcileProgress.proceed(), new ClusterSuspendStep().reconcile(mockContext, recorder));
+ }
+ verifyNoInteractions(recorder);
+ }
+
+ private void stubContext(SparkCluster cluster) {
+ stubContext(cluster, kubernetesClient);
+ }
+
+ private void stubContext(SparkCluster cluster, KubernetesClient client) {
+ when(recorder.appendNewStateAndPersist(any(), any())).thenReturn(true);
+ when(recorder.persistStatus(any(), any())).thenReturn(true);
+ when(mockContext.getResource()).thenReturn(cluster);
+ when(mockContext.getClient()).thenReturn(client);
+ }
+
+ /**
+ * Stubs the LIST of the master and worker pods of cluster1 on the given spy, since the mock
+ * server does not evaluate set-based label selectors, and returns the filter to list them with.
+ */
+ @SuppressWarnings("unchecked")
+ private static FilterWatchListDeletable<Pod, PodList, PodResource> stubPodList(
+ KubernetesClient client) {
+ MixedOperation<Pod, PodList, PodResource> pods = mock(MixedOperation.class);
+ NonNamespaceOperation<Pod, PodList, PodResource> namespacedPods =
+ mock(NonNamespaceOperation.class);
+ FilterWatchListDeletable<Pod, PodList, PodResource> labeledPods =
+ mock(FilterWatchListDeletable.class);
+ FilterWatchListDeletable<Pod, PodList, PodResource> rolePods =
+ mock(FilterWatchListDeletable.class);
+ doReturn(pods).when(client).pods();
+ when(pods.inNamespace("default")).thenReturn(namespacedPods);
+ when(namespacedPods.withLabel(Constants.LABEL_SPARK_CLUSTER_NAME, "cluster1"))
+ .thenReturn(labeledPods);
+ when(labeledPods.withLabelIn(
+ Constants.LABEL_SPARK_ROLE_NAME,
+ Constants.LABEL_SPARK_ROLE_MASTER_VALUE,
+ Constants.LABEL_SPARK_ROLE_WORKER_VALUE))
+ .thenReturn(rolePods);
+ return rolePods;
+ }
+
+ private void createRunningCluster() {
+ kubernetesClient.resource(masterStatefulSetSpec).create();
+ kubernetesClient.resource(workerStatefulSetSpec).create();
+ kubernetesClient
+ .resource(pod("cluster1-master-0", Constants.LABEL_SPARK_ROLE_MASTER_VALUE))
+ .create();
+ kubernetesClient
+ .resource(pod("cluster1-worker-0", Constants.LABEL_SPARK_ROLE_WORKER_VALUE))
+ .create();
+ }
+
+ private ClusterState captureAppendedState() {
+ ArgumentCaptor<ClusterState> captor = ArgumentCaptor.forClass(ClusterState.class);
+ verify(recorder).appendNewStateAndPersist(eq(mockContext), captor.capture());
+ verify(recorder, never()).persistStatus(any(), any());
+ return captor.getValue();
+ }
+
+ private StatefulSet get(StatefulSet statefulSet) {
+ return kubernetesClient.resource(statefulSet).get();
+ }
+
+ private Workload getWorkload() {
+ return kubernetesClient
+ .resources(Workload.class)
+ .inNamespace("default")
+ .withName("sparkcluster-cluster1")
+ .get();
+ }
+
+ private static SparkCluster buildCluster(ClusterStateSummary summary, boolean suspend) {
+ SparkCluster cluster = new SparkCluster();
+ cluster.setMetadata(
+ new ObjectMetaBuilder()
+ .withName("cluster1")
+ .withNamespace("default")
+ .withUid("cluster-uid")
+ .build());
+ cluster.setSpec(
+ ClusterSpec.builder()
+ .runtimeVersions(RuntimeVersions.builder().sparkVersion("4.2.0").build())
+ .clusterTolerations(
+ ClusterTolerations.builder()
+ .instanceConfig(
+ WorkerInstanceConfig.builder()
+ .initWorkers(1)
+ .minWorkers(1)
+ .maxWorkers(1)
+ .build())
+ .build())
+ .build());
+ cluster.getSpec().setSuspend(suspend);
+ cluster.setStatus(cluster.getStatus().appendNewState(new ClusterState(summary, "")));
+ return cluster;
+ }
+
+ private static SparkCluster buildKueueCluster(ClusterStateSummary summary, boolean suspend) {
+ SparkCluster cluster = buildCluster(summary, suspend);
+ cluster.getMetadata().setLabels(Map.of(Constants.LABEL_QUEUE_NAME, "cluster-queue"));
+ return cluster;
+ }
+
+ private static StatefulSet statefulSet(String name) {
+ return new StatefulSetBuilder()
+ .withNewMetadata()
+ .withName(name)
+ .withNamespace("default")
+ .endMetadata()
+ .withNewSpec()
+ .withReplicas(1)
+ .endSpec()
+ .build();
+ }
+
+ private static Pod pod(String name, String role) {
+ return new PodBuilder()
+ .withNewMetadata()
+ .withName(name)
+ .withNamespace("default")
+ .addToLabels(Constants.LABEL_SPARK_CLUSTER_NAME, "cluster1")
+ .addToLabels(Constants.LABEL_SPARK_ROLE_NAME, role)
+ .endMetadata()
+ .build();
+ }
+
+ private static Service service(String name) {
+ return new ServiceBuilder()
+ .withNewMetadata()
+ .withName(name)
+ .withNamespace("default")
+ .endMetadata()
+ .build();
+ }
+}
diff --git a/spark-submission-worker/src/main/java/org/apache/spark/k8s/operator/SparkClusterResourceSpec.java b/spark-submission-worker/src/main/java/org/apache/spark/k8s/operator/SparkClusterResourceSpec.java
index e7319b14..b28b3edc 100644
--- a/spark-submission-worker/src/main/java/org/apache/spark/k8s/operator/SparkClusterResourceSpec.java
+++ b/spark-submission-worker/src/main/java/org/apache/spark/k8s/operator/SparkClusterResourceSpec.java
@@ -219,7 +219,7 @@ public class SparkClusterResourceSpec {
var partialStatefulSet =
new StatefulSetBuilder()
.withNewMetadataLike(objectMeta)
- .withName(name + "-master")
+ .withName(getMasterStatefulSetName(name))
.addToLabels(LABEL_SPARK_ROLE_NAME, LABEL_SPARK_ROLE_MASTER_VALUE)
.addToLabels(LABEL_SPARK_VERSION_NAME, version)
.withNamespace(namespace)
@@ -300,7 +300,7 @@ public class SparkClusterResourceSpec {
var partialStatefulSet =
new StatefulSetBuilder()
.withNewMetadataLike(metadata)
- .withName(name + "-worker")
+ .withName(getWorkerStatefulSetName(name))
.addToLabels(LABEL_SPARK_ROLE_NAME, LABEL_SPARK_ROLE_WORKER_VALUE)
.addToLabels(LABEL_SPARK_VERSION_NAME, version)
.withNamespace(namespace)
@@ -356,6 +356,48 @@ public class SparkClusterResourceSpec {
.build();
}
+ /**
+ * Returns the name of the StatefulSet of the Spark master, which does not depend on the spec.
+ *
+ * @param clusterName The name of the cluster.
+ * @return The name of the master StatefulSet.
+ */
+ public static String getMasterStatefulSetName(String clusterName) {
+ return clusterName + "-master";
+ }
+
+ /**
+ * Returns the name of the StatefulSet of the Spark workers, which does not depend on the spec.
+ *
+ * @param clusterName The name of the cluster.
+ * @return The name of the worker StatefulSet.
+ */
+ public static String getWorkerStatefulSetName(String clusterName) {
+ return clusterName + "-worker";
+ }
+
+ /**
+ * Returns the name of the HorizontalPodAutoscaler of the Spark workers, which does not depend on
+ * whether the current spec asks for one.
+ *
+ * @param clusterName The name of the cluster.
+ * @return The name of the HorizontalPodAutoscaler.
+ */
+ public static String getWorkerHorizontalPodAutoscalerName(String clusterName) {
+ return clusterName + "-worker-hpa";
+ }
+
+ /**
+ * Returns the name of the PodDisruptionBudget of the Spark workers, which does not depend on
+ * whether the current spec asks for one.
+ *
+ * @param clusterName The name of the cluster.
+ * @return The name of the PodDisruptionBudget.
+ */
+ public static String getWorkerPodDisruptionBudgetName(String clusterName) {
+ return clusterName + "-worker-pdb";
+ }
+
/**
* Builds a Kubernetes HorizontalPodAutoscaler for the Spark workers.
*
@@ -410,14 +452,14 @@ public class SparkClusterResourceSpec {
new HorizontalPodAutoscalerBuilder()
.withNewMetadata()
.withNamespace(namespace)
- .withName(clusterName + "-worker-hpa")
+ .withName(getWorkerHorizontalPodAutoscalerName(clusterName))
.addToLabels(LABEL_SPARK_VERSION_NAME, spec.getRuntimeVersions().getSparkVersion())
.endMetadata()
.withNewSpecLike(horizontalPodAutoscalerSpec)
.withNewScaleTargetRef()
.withApiVersion("apps/v1")
.withKind("StatefulSet")
- .withName(clusterName + "-worker")
+ .withName(getWorkerStatefulSetName(clusterName))
.endScaleTargetRef()
.withMinReplicas(instanceConfig.getMinWorkers())
.withMaxReplicas(instanceConfig.getMaxWorkers())
@@ -441,7 +483,7 @@ public class SparkClusterResourceSpec {
return Optional.of(
new PodDisruptionBudgetBuilder()
.withNewMetadata()
- .withName(clusterName + "-worker-pdb")
+ .withName(getWorkerPodDisruptionBudgetName(clusterName))
.withNamespace(namespace)
.addToLabels(LABEL_SPARK_VERSION_NAME, spec.getRuntimeVersions().getSparkVersion())
.endMetadata()
diff --git a/tests/e2e/kueue/chainsaw-test.yaml b/tests/e2e/kueue/chainsaw-test.yaml
index f939474c..40bbcb07 100644
--- a/tests/e2e/kueue/chainsaw-test.yaml
+++ b/tests/e2e/kueue/chainsaw-test.yaml
@@ -539,6 +539,82 @@ spec:
type: Normal
reason: KueueAdmitted
(contains(message || '', 'sparkcluster-spark-cluster-kueue-queued-test')): true
+ # Suspending the running cluster releases its master and workers, and then its quota
+ - script:
+ timeout: 30s
+ content: |
+ kubectl patch sparkcluster spark-cluster-kueue-queued-test -n default --type merge -p '{"spec":{"suspend":true}}'
+ - assert:
+ timeout: 60s
+ resource:
+ apiVersion: spark.apache.org/v1
+ kind: SparkCluster
+ metadata:
+ name: spark-cluster-kueue-queued-test
+ namespace: default
+ status:
+ currentState:
+ currentStateSummary: Suspended
+ - error:
+ timeout: 120s
+ resource:
+ apiVersion: v1
+ kind: Pod
+ metadata:
+ namespace: default
+ labels:
+ spark.operator/spark-cluster-name: spark-cluster-kueue-queued-test
+ - error:
+ timeout: 120s
+ resource:
+ apiVersion: kueue.x-k8s.io/v1beta2
+ kind: Workload
+ metadata:
+ name: sparkcluster-spark-cluster-kueue-queued-test
+ namespace: default
+ - assert:
+ timeout: 60s
+ resource:
+ apiVersion: kueue.x-k8s.io/v1beta2
+ kind: ClusterQueue
+ metadata:
+ name: ($CLUSTER_QUEUE)
+ status:
+ admittedWorkloads: 0
+ reservingWorkloads: 0
+ # Resuming queues the cluster again, and Kueue admits it into the quota it gave back
+ - script:
+ timeout: 30s
+ content: |
+ kubectl patch sparkcluster spark-cluster-kueue-queued-test -n default --type merge -p '{"spec":{"suspend":false}}'
+ - assert:
+ timeout: 5m
+ resource:
+ apiVersion: spark.apache.org/v1
+ kind: SparkCluster
+ metadata:
+ name: spark-cluster-kueue-queued-test
+ namespace: default
+ status:
+ currentState:
+ currentStateSummary: RunningHealthy
+ - assert:
+ timeout: 60s
+ resource:
+ apiVersion: kueue.x-k8s.io/v1beta2
+ kind: ClusterQueue
+ metadata:
+ name: ($CLUSTER_QUEUE)
+ status:
+ admittedWorkloads: 1
+ - assert:
+ bindings:
+ - name: SPARK_APP_NAMESPACE
+ value: default
+ - name: SPARK_CLUSTER_NAME
+ value: spark-cluster-kueue-queued-test
+ timeout: 5m
+ file: "../assertions/spark-cluster/spark-cluster-worker-statefulset.yaml"
catch:
- describe:
apiVersion: spark.apache.org/v1
diff --git a/tests/e2e/suspend/chainsaw-test.yaml b/tests/e2e/suspend/chainsaw-test.yaml
index 0bc2b370..a70842c5 100644
--- a/tests/e2e/suspend/chainsaw-test.yaml
+++ b/tests/e2e/suspend/chainsaw-test.yaml
@@ -130,6 +130,77 @@ spec:
value: spark-cluster-suspend-test
timeout: 120s
file: "../assertions/spark-cluster/spark-cluster-worker-statefulset.yaml"
+ - script:
+ timeout: 30s
+ content: |
+ kubectl patch sparkcluster spark-cluster-suspend-test -n default --type merge -p '{"spec":{"suspend":true}}'
+ - assert:
+ timeout: 60s
+ resource:
+ apiVersion: spark.apache.org/v1
+ kind: SparkCluster
+ metadata:
+ name: spark-cluster-suspend-test
+ namespace: default
+ status:
+ currentState:
+ currentStateSummary: Suspended
+ - error:
+ timeout: 60s
+ resource:
+ apiVersion: apps/v1
+ kind: StatefulSet
+ metadata:
+ name: spark-cluster-suspend-test-master
+ namespace: default
+ - error:
+ timeout: 60s
+ resource:
+ apiVersion: apps/v1
+ kind: StatefulSet
+ metadata:
+ name: spark-cluster-suspend-test-worker
+ namespace: default
+ - error:
+ timeout: 120s
+ resource:
+ apiVersion: v1
+ kind: Pod
+ metadata:
+ namespace: default
+ labels:
+ spark.operator/spark-cluster-name: spark-cluster-suspend-test
+ - assert:
+ timeout: 10s
+ resource:
+ apiVersion: v1
+ kind: Service
+ metadata:
+ name: spark-cluster-suspend-test-master-svc
+ namespace: default
+ - script:
+ timeout: 30s
+ content: |
+ kubectl patch sparkcluster spark-cluster-suspend-test -n default --type merge -p '{"spec":{"suspend":false}}'
+ - assert:
+ timeout: 120s
+ resource:
+ apiVersion: spark.apache.org/v1
+ kind: SparkCluster
+ metadata:
+ name: spark-cluster-suspend-test
+ namespace: default
+ status:
+ currentState:
+ currentStateSummary: RunningHealthy
+ - assert:
+ bindings:
+ - name: SPARK_APP_NAMESPACE
+ value: default
+ - name: SPARK_CLUSTER_NAME
+ value: spark-cluster-suspend-test
+ timeout: 120s
+ file: "../assertions/spark-cluster/spark-cluster-worker-statefulset.yaml"
catch:
- describe:
apiVersion: spark.apache.org/v1
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org