You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dongjoon-hyun (via GitHub)" <gi...@apache.org> on 2026/09/29 16:14:42 UTC
[PR] [SPARK-59870] Clarify the `Failed` message of `SparkCluster` and the K8s events docs [spark-kubernetes-operator]
dongjoon-hyun opened a new pull request, #908:
URL: https://github.com/apache/spark-kubernetes-operator/pull/908
### What changes were proposed in this pull request?
This PR aims to clarify the following messages.
- The `Failed` state after `SchedulingFailure` of `SparkCluster` points to `SchedulingFailure` for the cause, instead of `Cannot process cluster status.`.
- The description of `spark.kubernetes.operator.events.enabled` refers to `configuration.md` instead of listing only some of the events.
- `configuration.md` documents the RBAC of K8s events: `get`, `create` and `patch` on `events`.
### Why are the changes needed?
- `Cannot process cluster status.` reads as another, unexplained failure.
- The description misses 5 of the 13 non-state event reasons.
- Users who manage the operator RBAC without the Helm chart need to know these permissions.
### Does this PR introduce _any_ user-facing change?
Yes, but only the message of the `Failed` state after `SchedulingFailure` of `SparkCluster` changes from [1.0.0](https://github.com/apache/spark-kubernetes-operator/releases/tag/1.0.0) (2026-07-23). The rest is documentation.
### How was this patch tested?
Pass the CIs with the updated test case.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59870] Clarify the `Failed` message of `SparkCluster` and the K8s events docs [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #908:
URL: https://github.com/apache/spark-kubernetes-operator/pull/908#discussion_r4136129895
##########
spark-operator-api/src/main/java/org/apache/spark/k8s/operator/Constants.java:
##########
@@ -273,6 +273,14 @@ public final class Constants {
public static final String CLUSTER_REQUEUED_MESSAGE =
"Cluster is submitted again after its eviction by Kueue.";
+ /**
+ * Message indicating that the cluster failed after SchedulingFailure, whose message tells the
+ * cause.
+ */
+ public static final String CLUSTER_FAILED_AFTER_SCHEDULING_FAILURE_MESSAGE =
+ "Cluster failed, since its master and workers could not be requested. See the preceding "
+ + "SchedulingFailure state for the cause.";
Review Comment:
Thank you. Right, the master and worker `StatefulSet`s can be applied already when a later request fails. I applied your suggestion via da952c96.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59870] Clarify the `Failed` message of `SparkCluster` and the K8s events docs [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #908:
URL: https://github.com/apache/spark-kubernetes-operator/pull/908#issuecomment-5894795098
Thank you for the review, @peter-toth.
1. I updated the answer to the user-facing change question in the PR description, since the `Failed` state message after `SchedulingFailure` is released in 1.0.0.
2. I applied your suggestion via da952c96.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59870] Clarify the `Failed` message of `SparkCluster` and the K8s events docs [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #908: [SPARK-59870] Clarify the `Failed` message of `SparkCluster` and the K8s events docs
URL: https://github.com/apache/spark-kubernetes-operator/pull/908
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59870] Clarify the `Failed` message of `SparkCluster` and the K8s events docs [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #908:
URL: https://github.com/apache/spark-kubernetes-operator/pull/908#issuecomment-5894903262
Thank you again!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59870] Clarify the `Failed` message of `SparkCluster` and the K8s events docs [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #908:
URL: https://github.com/apache/spark-kubernetes-operator/pull/908#issuecomment-5894910802
Merged to main
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59870] Clarify the `Failed` message of `SparkCluster` and the K8s events docs [spark-kubernetes-operator]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #908:
URL: https://github.com/apache/spark-kubernetes-operator/pull/908#discussion_r4135953065
##########
spark-operator-api/src/main/java/org/apache/spark/k8s/operator/Constants.java:
##########
@@ -273,6 +273,14 @@ public final class Constants {
public static final String CLUSTER_REQUEUED_MESSAGE =
"Cluster is submitted again after its eviction by Kueue.";
+ /**
+ * Message indicating that the cluster failed after SchedulingFailure, whose message tells the
+ * cause.
+ */
+ public static final String CLUSTER_FAILED_AFTER_SCHEDULING_FAILURE_MESSAGE =
+ "Cluster failed, since its master and workers could not be requested. See the preceding "
+ + "SchedulingFailure state for the cause.";
Review Comment:
**Finding 2.** "its master and workers could not be requested" doesn't hold on every path into `SchedulingFailure`. `ClusterInitStep` applies the master and worker StatefulSets first (`ClusterInitStep.java:112-127`), then the NetworkPolicy, the HPA and the PDB. A non-retryable failure of one of the later requests still ends in `failScheduling` (`ClusterInitStep.java:182`). Examples are a 403 from an operator RBAC without `networkpolicies`, or an admission policy which rejects the PDB.
I checked it with a throwaway test on `ClusterInitStep` whose NetworkPolicy apply throws a 403:
- both StatefulSets were applied;
- the state became `SchedulingFailure`;
- `ClusterUnknownStateStep` then appended `Failed` with this message.
Nothing releases a `Failed` cluster, so its master and workers keep running while the status says they were never requested. Maybe word it without that claim:
```suggestion
"Cluster failed, since requesting its resources failed. See the preceding "
+ "SchedulingFailure state for the cause.";
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org