You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "jiangzho (via GitHub)" <gi...@apache.org> on 2026/02/21 01:25:08 UTC
[PR] [SPARK-55623] Add granular restart control with consecutive failure tracking [spark-kubernetes-operator]
jiangzho opened a new pull request, #514:
URL: https://github.com/apache/spark-kubernetes-operator/pull/514
### What changes were proposed in this pull request?
This PR implements granular restart control for Spark applications with support for consecutive failure / scheduling failures tracking.
### Why are the changes needed?
Consecutive failure tracking enables operators to allow more total restarts for apps with occasional transient failuresm, and to stop quickly on persistent failures. It also apply special handling for scheduling failures to mitigate API server stress.
This is an enhancement based on our current configuration for maximal restarts and backoff interval.
### Does this PR introduce any user-facing change?
Yes. New optional configuration fields availale in RestartConfig:
```yaml
restartConfig:
restartPolicy: Always
maxRestartAttempts: 5 # existing field
restartBackoffMillis: 30000 # existing field
restartCounterResetMillis: 3600000 # existing field
# New: consecutive failure limits
maxRestartOnFailure: 3
restartBackoffMillisForFailure: 60000
maxRestartOnSchedulingFailure: 2
restartBackoffMillisForSchedulingFailure: 300000
```
This should be backwards compartible. All new fields are optional. When failure-specific limits are not set, the operator uses maxRestartAttempts as before.
### How was this patch tested?
Unit tests added to validate the limit evaluation flow, with & without the new fields.
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55623] Add granular restart control with consecutive failure tracking [spark-kubernetes-operator]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #514:
URL: https://github.com/apache/spark-kubernetes-operator/pull/514#discussion_r2845393287
##########
docs/spark_custom_resources.md:
##########
@@ -234,14 +234,111 @@ restartConfig:
restartBackoffMillis: 30000
```
+### Granular Restart Control
+
+For more fine-grained control over restart behavior, you can configure different retry limits
+and backoff times for specific failure types. This allows you to handle different failure
+scenarios with appropriate strategies.
+
+The operator maintains multiple counters to track different types of restarts:
+- General restart counter: Tracks all restarts
+- Consecutive failure counter: Tracks consecutive failures
+- Consecutive scheduling failure counter: Tracks consecutive scheduling failures only
+
+#### Restart Behavior Control
+
+- Consecutive failure tracking: The failure-specific counters track consecutive failures
+ of the app, distinguishing between persistent failures (requiring intervention) and
+ transient issues (safe for retry).
+ - For Example: With `restartPolicy=Always`, `maxRestartAttempts=5` and `maxRestartOnFailure=2`:
+ - The app would tolerate at maximum of 3 consecutive failures, with maximal of 5 restarts
+ - In other words, sequence F -> F -> F would stop.
+ - sequence F -> S -> F -> S -> F would continue with the 5th restart as the succeeded attempts
+ reset the failure counter
+- Granular control over `SchedulingFailure`: similarly, it's possible to control the maximal
+ restart and backoff interval for consecutive `SchedulingFailure` attempts, as it can be highly
+ associated with API server rejections, quota exceeded, resource constraints.
+
+#### Restart Limit Evaluation
+
+When an attempt ends, limits are checked in order:
+ 1. General limit (`maxRestartAttempts`) is checked for every restart
+ 2. For failures, the most specific applicable limit is also checked:
+ - Scheduling failures (SchedulingFailure) → `maxRestartOnSchedulingFailure` (if set)
+ - Other failures → `maxRestartOnFailure` (if set)
+ 3. The application stops if any applicable limit is exceeded
+
+
+#### Configuration Fields
+
+```yaml
+restartConfig:
+ restartPolicy: Always
+ # Default restart configuration (applies to all restarts)
+ maxRestartAttempts: 5
+ restartBackoffMillis: 30000 # 30 seconds
+
+ # Override for consecutive general failures (application crashes, driver failures, etc.)
+ # This counter resets to 0 on success
+ maxRestartOnFailure: 3
+ restartBackoffMillisForFailure: 60000 # 1 minute
+
+ # Override for consecutive scheduling failures
+ maxRestartOnSchedulingFailure: 1
+ restartBackoffMillisForSchedulingFailure: 300000 # 5 minutes
+```
+
+#### Example Use Cases
+
+Tolerate transient failures but stop on persistent issues:
+
+```yaml
+restartConfig:
+ restartPolicy: Always
+ maxRestartAttempts: 100 # Allow many total attempts
+ restartBackoffMillis: 30000
+ # But stop after 3 consecutive failures (indicates persistent problem)
+ maxRestartOnFailure: 3
+ restartBackoffMillisForFailure: 60000
+```
+
+Mitigate API server stress during scheduling failures:
+
+```yaml
+restartConfig:
+ restartPolicy: Always
+ maxRestartAttempts: 50
+ restartBackoffMillis: 30000
+ # Stop quickly on scheduling failures to avoid overwhelming API server
+ maxRestartOnSchedulingFailure: 2
+ restartBackoffMillisForSchedulingFailure: 600000 # 10 minutes
+```
+
+
+| Field | Type | Default Value | Description |
+|-----------------------------------------------------------------------------------------|---------|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| .spec.applicationTolerations.restartConfig.restartPolicy | string | Never | Restart policy: `Never`, `Always`, `OnFailure`, or `OnInfrastructureFailure` |
+| .spec.applicationTolerations.restartConfig.maxRestartAttempts | integer | 3 | Maximum number of restart attempts for all scenarios (always checked) |
+| .spec.applicationTolerations.restartConfig.restartBackoffMillis | integer | 30000 | Default backoff time in milliseconds between restart attempts |
+| .spec.applicationTolerations.restartConfig.maxRestartOnFailure | integer | null | Maximum consecutive failures before stopping. Resets to 0 on success. If null, uses maxRestartAttempts |
Review Comment:
Should we use `-1` default value for the not specified case? It seems that's the value we use in other fields as well (e.g. `restartCounterResetMillis`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55623] Add granular restart control with consecutive failure tracking [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #514:
URL: https://github.com/apache/spark-kubernetes-operator/pull/514#issuecomment-4012835083
Merged to main. Thank you, @jiangzho and @peter-toth .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55623] Add granular restart control with consecutive failure tracking [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #514:
URL: https://github.com/apache/spark-kubernetes-operator/pull/514#discussion_r2891356993
##########
spark-operator-api/src/main/java/org/apache/spark/k8s/operator/status/ApplicationAttemptInfo.java:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.status;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import lombok.EqualsAndHashCode;
+import lombok.Getter;
+import lombok.Setter;
+import lombok.ToString;
+
+/**
+ * Information about an attempt.
+ *
+ * <p>Maintains counters for different restart limit checks:
+ * <ul>
+ * <li><b>failureRestartCounter</b>: Consecutive failure count, checked against
+ * maxRestartOnFailure</li>
+ * <li><b>schedulingFailureRestartCounter</b>: Consecutive scheduling failure count, checked
+ * against maxRestartOnSchedulingFailure</li>
+ * </ul>
+ *
+ */
+@Setter
+@Getter
+@EqualsAndHashCode(callSuper = true)
+@ToString(callSuper = true)
+@JsonInclude(JsonInclude.Include.NON_NULL)
+@JsonIgnoreProperties(ignoreUnknown = true)
+public class ApplicationAttemptInfo extends BaseAttemptInfo {
+ protected long failureRestartCounter;
+ protected long schedulingFailureRestartCounter;
+
+ public ApplicationAttemptInfo() {
+ super();
+ failureRestartCounter = 0L;
Review Comment:
For the counter, `int` is enough already, isn't it?
```
int 32 -2,147,483,648 to 2,147,483,647
long 64 -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55623] Add granular restart control with consecutive failure tracking [spark-kubernetes-operator]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #514:
URL: https://github.com/apache/spark-kubernetes-operator/pull/514#discussion_r2845393287
##########
docs/spark_custom_resources.md:
##########
@@ -234,14 +234,111 @@ restartConfig:
restartBackoffMillis: 30000
```
+### Granular Restart Control
+
+For more fine-grained control over restart behavior, you can configure different retry limits
+and backoff times for specific failure types. This allows you to handle different failure
+scenarios with appropriate strategies.
+
+The operator maintains multiple counters to track different types of restarts:
+- General restart counter: Tracks all restarts
+- Consecutive failure counter: Tracks consecutive failures
+- Consecutive scheduling failure counter: Tracks consecutive scheduling failures only
+
+#### Restart Behavior Control
+
+- Consecutive failure tracking: The failure-specific counters track consecutive failures
+ of the app, distinguishing between persistent failures (requiring intervention) and
+ transient issues (safe for retry).
+ - For Example: With `restartPolicy=Always`, `maxRestartAttempts=5` and `maxRestartOnFailure=2`:
+ - The app would tolerate at maximum of 3 consecutive failures, with maximal of 5 restarts
+ - In other words, sequence F -> F -> F would stop.
+ - sequence F -> S -> F -> S -> F would continue with the 5th restart as the succeeded attempts
+ reset the failure counter
+- Granular control over `SchedulingFailure`: similarly, it's possible to control the maximal
+ restart and backoff interval for consecutive `SchedulingFailure` attempts, as it can be highly
+ associated with API server rejections, quota exceeded, resource constraints.
+
+#### Restart Limit Evaluation
+
+When an attempt ends, limits are checked in order:
+ 1. General limit (`maxRestartAttempts`) is checked for every restart
+ 2. For failures, the most specific applicable limit is also checked:
+ - Scheduling failures (SchedulingFailure) → `maxRestartOnSchedulingFailure` (if set)
+ - Other failures → `maxRestartOnFailure` (if set)
+ 3. The application stops if any applicable limit is exceeded
+
+
+#### Configuration Fields
+
+```yaml
+restartConfig:
+ restartPolicy: Always
+ # Default restart configuration (applies to all restarts)
+ maxRestartAttempts: 5
+ restartBackoffMillis: 30000 # 30 seconds
+
+ # Override for consecutive general failures (application crashes, driver failures, etc.)
+ # This counter resets to 0 on success
+ maxRestartOnFailure: 3
+ restartBackoffMillisForFailure: 60000 # 1 minute
+
+ # Override for consecutive scheduling failures
+ maxRestartOnSchedulingFailure: 1
+ restartBackoffMillisForSchedulingFailure: 300000 # 5 minutes
+```
+
+#### Example Use Cases
+
+Tolerate transient failures but stop on persistent issues:
+
+```yaml
+restartConfig:
+ restartPolicy: Always
+ maxRestartAttempts: 100 # Allow many total attempts
+ restartBackoffMillis: 30000
+ # But stop after 3 consecutive failures (indicates persistent problem)
+ maxRestartOnFailure: 3
+ restartBackoffMillisForFailure: 60000
+```
+
+Mitigate API server stress during scheduling failures:
+
+```yaml
+restartConfig:
+ restartPolicy: Always
+ maxRestartAttempts: 50
+ restartBackoffMillis: 30000
+ # Stop quickly on scheduling failures to avoid overwhelming API server
+ maxRestartOnSchedulingFailure: 2
+ restartBackoffMillisForSchedulingFailure: 600000 # 10 minutes
+```
+
+
+| Field | Type | Default Value | Description |
+|-----------------------------------------------------------------------------------------|---------|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| .spec.applicationTolerations.restartConfig.restartPolicy | string | Never | Restart policy: `Never`, `Always`, `OnFailure`, or `OnInfrastructureFailure` |
+| .spec.applicationTolerations.restartConfig.maxRestartAttempts | integer | 3 | Maximum number of restart attempts for all scenarios (always checked) |
+| .spec.applicationTolerations.restartConfig.restartBackoffMillis | integer | 30000 | Default backoff time in milliseconds between restart attempts |
+| .spec.applicationTolerations.restartConfig.maxRestartOnFailure | integer | null | Maximum consecutive failures before stopping. Resets to 0 on success. If null, uses maxRestartAttempts |
Review Comment:
Shall we use `-1` default value for the not specified case? It seems that's the value we use in other fields as well (e.g. `restartCounterResetMillis`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55623] Add granular restart control with consecutive failure tracking [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #514: [SPARK-55623] Add granular restart control with consecutive failure tracking
URL: https://github.com/apache/spark-kubernetes-operator/pull/514
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55623] Add granular restart control with consecutive failure tracking [spark-kubernetes-operator]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #514:
URL: https://github.com/apache/spark-kubernetes-operator/pull/514#discussion_r2845393287
##########
docs/spark_custom_resources.md:
##########
@@ -234,14 +234,111 @@ restartConfig:
restartBackoffMillis: 30000
```
+### Granular Restart Control
+
+For more fine-grained control over restart behavior, you can configure different retry limits
+and backoff times for specific failure types. This allows you to handle different failure
+scenarios with appropriate strategies.
+
+The operator maintains multiple counters to track different types of restarts:
+- General restart counter: Tracks all restarts
+- Consecutive failure counter: Tracks consecutive failures
+- Consecutive scheduling failure counter: Tracks consecutive scheduling failures only
+
+#### Restart Behavior Control
+
+- Consecutive failure tracking: The failure-specific counters track consecutive failures
+ of the app, distinguishing between persistent failures (requiring intervention) and
+ transient issues (safe for retry).
+ - For Example: With `restartPolicy=Always`, `maxRestartAttempts=5` and `maxRestartOnFailure=2`:
+ - The app would tolerate at maximum of 3 consecutive failures, with maximal of 5 restarts
+ - In other words, sequence F -> F -> F would stop.
+ - sequence F -> S -> F -> S -> F would continue with the 5th restart as the succeeded attempts
+ reset the failure counter
+- Granular control over `SchedulingFailure`: similarly, it's possible to control the maximal
+ restart and backoff interval for consecutive `SchedulingFailure` attempts, as it can be highly
+ associated with API server rejections, quota exceeded, resource constraints.
+
+#### Restart Limit Evaluation
+
+When an attempt ends, limits are checked in order:
+ 1. General limit (`maxRestartAttempts`) is checked for every restart
+ 2. For failures, the most specific applicable limit is also checked:
+ - Scheduling failures (SchedulingFailure) → `maxRestartOnSchedulingFailure` (if set)
+ - Other failures → `maxRestartOnFailure` (if set)
+ 3. The application stops if any applicable limit is exceeded
+
+
+#### Configuration Fields
+
+```yaml
+restartConfig:
+ restartPolicy: Always
+ # Default restart configuration (applies to all restarts)
+ maxRestartAttempts: 5
+ restartBackoffMillis: 30000 # 30 seconds
+
+ # Override for consecutive general failures (application crashes, driver failures, etc.)
+ # This counter resets to 0 on success
+ maxRestartOnFailure: 3
+ restartBackoffMillisForFailure: 60000 # 1 minute
+
+ # Override for consecutive scheduling failures
+ maxRestartOnSchedulingFailure: 1
+ restartBackoffMillisForSchedulingFailure: 300000 # 5 minutes
+```
+
+#### Example Use Cases
+
+Tolerate transient failures but stop on persistent issues:
+
+```yaml
+restartConfig:
+ restartPolicy: Always
+ maxRestartAttempts: 100 # Allow many total attempts
+ restartBackoffMillis: 30000
+ # But stop after 3 consecutive failures (indicates persistent problem)
+ maxRestartOnFailure: 3
+ restartBackoffMillisForFailure: 60000
+```
+
+Mitigate API server stress during scheduling failures:
+
+```yaml
+restartConfig:
+ restartPolicy: Always
+ maxRestartAttempts: 50
+ restartBackoffMillis: 30000
+ # Stop quickly on scheduling failures to avoid overwhelming API server
+ maxRestartOnSchedulingFailure: 2
+ restartBackoffMillisForSchedulingFailure: 600000 # 10 minutes
+```
+
+
+| Field | Type | Default Value | Description |
+|-----------------------------------------------------------------------------------------|---------|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| .spec.applicationTolerations.restartConfig.restartPolicy | string | Never | Restart policy: `Never`, `Always`, `OnFailure`, or `OnInfrastructureFailure` |
+| .spec.applicationTolerations.restartConfig.maxRestartAttempts | integer | 3 | Maximum number of restart attempts for all scenarios (always checked) |
+| .spec.applicationTolerations.restartConfig.restartBackoffMillis | integer | 30000 | Default backoff time in milliseconds between restart attempts |
+| .spec.applicationTolerations.restartConfig.maxRestartOnFailure | integer | null | Maximum consecutive failures before stopping. Resets to 0 on success. If null, uses maxRestartAttempts |
Review Comment:
Should we use `-1` default value for the not specified case? It seems that's the value we use in other fields as well (e..g. `restartCounterResetMillis`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55623] Add granular restart control with consecutive failure tracking [spark-kubernetes-operator]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #514:
URL: https://github.com/apache/spark-kubernetes-operator/pull/514#discussion_r2845583246
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/AppCleanUpStep.java:
##########
@@ -174,6 +174,10 @@ public ReconcileProgress reconcile(
return appendStateAndRequeueAfter(
context, statusRecorder, state, Duration.ofMillis(requeueAfterMillis));
} else {
+ // Check if current state is a failure before restarting
Review Comment:
Seems like you can move this into the if just before `requeueAfterMillis = ...`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55623] Add granular restart control with consecutive failure tracking [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #514:
URL: https://github.com/apache/spark-kubernetes-operator/pull/514#issuecomment-3953859845
Ack, @peter-toth . I'll take a look today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55623] Add granular restart control with consecutive failure tracking [spark-kubernetes-operator]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #514:
URL: https://github.com/apache/spark-kubernetes-operator/pull/514#discussion_r2890712038
##########
docs/spark_custom_resources.md:
##########
@@ -234,14 +234,111 @@ restartConfig:
restartBackoffMillis: 30000
```
+### Granular Restart Control
+
+For more fine-grained control over restart behavior, you can configure different retry limits
+and backoff times for specific failure types. This allows you to handle different failure
+scenarios with appropriate strategies.
+
+The operator maintains multiple counters to track different types of restarts:
+- General restart counter: Tracks all restarts
+- Consecutive failure counter: Tracks consecutive failures
+- Consecutive scheduling failure counter: Tracks consecutive scheduling failures only
+
+#### Restart Behavior Control
+
+- Consecutive failure tracking: The failure-specific counters track consecutive failures
+ of the app, distinguishing between persistent failures (requiring intervention) and
+ transient issues (safe for retry).
+ - For Example: With `restartPolicy=Always`, `maxRestartAttempts=5` and `maxRestartOnFailure=2`:
+ - The app would tolerate at maximum of 3 consecutive failures, with maximal of 5 restarts
+ - In other words, sequence F -> F -> F would stop.
+ - sequence F -> S -> F -> S -> F would continue with the 5th restart as the succeeded attempts
+ reset the failure counter
+- Granular control over `SchedulingFailure`: similarly, it's possible to control the maximal
+ restart and backoff interval for consecutive `SchedulingFailure` attempts, as it can be highly
+ associated with API server rejections, quota exceeded, resource constraints.
+
+#### Restart Limit Evaluation
+
+When an attempt ends, limits are checked in order:
+ 1. General limit (`maxRestartAttempts`) is checked for every restart
+ 2. For failures, the most specific applicable limit is also checked:
+ - Scheduling failures (SchedulingFailure) → `maxRestartOnSchedulingFailure` (if set)
+ - Other failures → `maxRestartOnFailure` (if set)
+ 3. The application stops if any applicable limit is exceeded
+
+
+#### Configuration Fields
+
+```yaml
+restartConfig:
+ restartPolicy: Always
+ # Default restart configuration (applies to all restarts)
+ maxRestartAttempts: 5
+ restartBackoffMillis: 30000 # 30 seconds
+
+ # Override for consecutive general failures (application crashes, driver failures, etc.)
+ # This counter resets to 0 on success
+ maxRestartOnFailure: 3
+ restartBackoffMillisForFailure: 60000 # 1 minute
+
+ # Override for consecutive scheduling failures
+ maxRestartOnSchedulingFailure: 1
+ restartBackoffMillisForSchedulingFailure: 300000 # 5 minutes
+```
+
+#### Example Use Cases
+
+Tolerate transient failures but stop on persistent issues:
+
+```yaml
+restartConfig:
+ restartPolicy: Always
+ maxRestartAttempts: 100 # Allow many total attempts
+ restartBackoffMillis: 30000
+ # But stop after 3 consecutive failures (indicates persistent problem)
+ maxRestartOnFailure: 3
+ restartBackoffMillisForFailure: 60000
+```
+
+Mitigate API server stress during scheduling failures:
+
+```yaml
+restartConfig:
+ restartPolicy: Always
+ maxRestartAttempts: 50
+ restartBackoffMillis: 30000
+ # Stop quickly on scheduling failures to avoid overwhelming API server
+ maxRestartOnSchedulingFailure: 2
+ restartBackoffMillisForSchedulingFailure: 600000 # 10 minutes
+```
+
+
+| Field | Type | Default Value | Description |
+|-----------------------------------------------------------------------------------------|---------|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| .spec.applicationTolerations.restartConfig.restartPolicy | string | Never | Restart policy: `Never`, `Always`, `OnFailure`, or `OnInfrastructureFailure` |
+| .spec.applicationTolerations.restartConfig.maxRestartAttempts | integer | 3 | Maximum number of restart attempts for all scenarios (always checked) |
+| .spec.applicationTolerations.restartConfig.restartBackoffMillis | integer | 30000 | Default backoff time in milliseconds between restart attempts |
+| .spec.applicationTolerations.restartConfig.maxRestartOnFailure | integer | null | Maximum consecutive failures before stopping. Resets to 0 on success. If null, uses maxRestartAttempts |
Review Comment:
Yeah, I see your point. These configs are kind of overrides for the default `maxRestartAttempts` / `restartBackoffMillis` configs in a special case, and as such, not overriding or setting a special override value (e.g. -1 for unlimited) can be different.
Yes, maybe 1 makes sense as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55623] Add granular restart control with consecutive failure tracking [spark-kubernetes-operator]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #514:
URL: https://github.com/apache/spark-kubernetes-operator/pull/514#discussion_r2845643638
##########
spark-operator-api/src/main/java/org/apache/spark/k8s/operator/status/AttemptInfo.java:
##########
@@ -40,13 +53,41 @@
public class AttemptInfo {
@Getter @Builder.Default protected final long id = 0L;
@Getter @Setter protected long restartCounter;
+ @Getter @Setter protected long failureRestartCounter;
+ @Getter @Setter protected long schedulingFailureRestartCounter;
/**
* Creates a new AttemptInfo object representing the next attempt.
- *
- * @return A new AttemptInfo with an incremented ID.
*/
- public AttemptInfo createNextAttemptInfo(boolean resetRestartCounter) {
- return new AttemptInfo(id + 1L, resetRestartCounter ? 1L : restartCounter + 1);
+ public AttemptInfo createNextAttemptInfo(
+ boolean resetRestartCounter, ApplicationStateSummary currentStateSummary) {
+ long newRestartCounter = resetRestartCounter ? 1L : restartCounter + 1;
+ long newFailureCounter;
+ long newSchedulingFailureCounter;
+
+ if (resetRestartCounter) {
+ // Reset all counters when restart counter is reset
+ newRestartCounter = 1L;
Review Comment:
I wonder if it would make sense to merge this `if ... else if ...` logic with the logic in `terminateOrRestart()`?
I mean, you are calculating `exceededLimit` / `stateMessage` based on `ApplicationStateSummary` there, and you are kind of calculating the new counters based on `ApplicationStateSummary` here in `AttemptInfo.createNextAttemptInfo()`. But `createNextAttemptInfo()` gets invoked only from `terminateOrRestart()`, so would it make sense to drop `createNextAttemptInfo()` entirelly, calculate everyting in `terminateOrRestart()` and keep `AttemptInfo` as a simple data class?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55623] Add granular restart control with consecutive failure tracking [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #514:
URL: https://github.com/apache/spark-kubernetes-operator/pull/514#discussion_r2891352997
##########
spark-operator-api/src/main/java/org/apache/spark/k8s/operator/spec/RestartConfig.java:
##########
@@ -42,4 +44,23 @@ public class RestartConfig {
@Default("-1")
@Builder.Default
protected Long restartCounterResetMillis = -1L;
+ @Builder.Default protected Long maxRestartOnFailure = null;
Review Comment:
~Can we use `Integer` instead of `Long` , @jiangzho ?~
##########
spark-operator-api/src/main/java/org/apache/spark/k8s/operator/status/ApplicationAttemptInfo.java:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.status;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import lombok.EqualsAndHashCode;
+import lombok.Getter;
+import lombok.Setter;
+import lombok.ToString;
+
+/**
+ * Information about an attempt.
+ *
+ * <p>Maintains counters for different restart limit checks:
+ * <ul>
+ * <li><b>failureRestartCounter</b>: Consecutive failure count, checked against
+ * maxRestartOnFailure</li>
+ * <li><b>schedulingFailureRestartCounter</b>: Consecutive scheduling failure count, checked
+ * against maxRestartOnSchedulingFailure</li>
+ * </ul>
+ *
+ */
+@Setter
+@Getter
+@EqualsAndHashCode(callSuper = true)
+@ToString(callSuper = true)
+@JsonInclude(JsonInclude.Include.NON_NULL)
+@JsonIgnoreProperties(ignoreUnknown = true)
+public class ApplicationAttemptInfo extends BaseAttemptInfo {
+ protected long failureRestartCounter;
+ protected long schedulingFailureRestartCounter;
+
+ public ApplicationAttemptInfo() {
+ super();
+ failureRestartCounter = 0L;
Review Comment:
~For the counter, `int` is enough already, isn't it?~
```
int 32 -2,147,483,648 to 2,147,483,647
long 64 -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55623] Add granular restart control with consecutive failure tracking [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #514:
URL: https://github.com/apache/spark-kubernetes-operator/pull/514#discussion_r2891352997
##########
spark-operator-api/src/main/java/org/apache/spark/k8s/operator/spec/RestartConfig.java:
##########
@@ -42,4 +44,23 @@ public class RestartConfig {
@Default("-1")
@Builder.Default
protected Long restartCounterResetMillis = -1L;
+ @Builder.Default protected Long maxRestartOnFailure = null;
Review Comment:
Can we use `Integer` instead of `Long` , @jiangzho ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55623] Add granular restart control with consecutive failure tracking [spark-kubernetes-operator]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #514:
URL: https://github.com/apache/spark-kubernetes-operator/pull/514#discussion_r2845393287
##########
docs/spark_custom_resources.md:
##########
@@ -234,14 +234,111 @@ restartConfig:
restartBackoffMillis: 30000
```
+### Granular Restart Control
+
+For more fine-grained control over restart behavior, you can configure different retry limits
+and backoff times for specific failure types. This allows you to handle different failure
+scenarios with appropriate strategies.
+
+The operator maintains multiple counters to track different types of restarts:
+- General restart counter: Tracks all restarts
+- Consecutive failure counter: Tracks consecutive failures
+- Consecutive scheduling failure counter: Tracks consecutive scheduling failures only
+
+#### Restart Behavior Control
+
+- Consecutive failure tracking: The failure-specific counters track consecutive failures
+ of the app, distinguishing between persistent failures (requiring intervention) and
+ transient issues (safe for retry).
+ - For Example: With `restartPolicy=Always`, `maxRestartAttempts=5` and `maxRestartOnFailure=2`:
+ - The app would tolerate at maximum of 3 consecutive failures, with maximal of 5 restarts
+ - In other words, sequence F -> F -> F would stop.
+ - sequence F -> S -> F -> S -> F would continue with the 5th restart as the succeeded attempts
+ reset the failure counter
+- Granular control over `SchedulingFailure`: similarly, it's possible to control the maximal
+ restart and backoff interval for consecutive `SchedulingFailure` attempts, as it can be highly
+ associated with API server rejections, quota exceeded, resource constraints.
+
+#### Restart Limit Evaluation
+
+When an attempt ends, limits are checked in order:
+ 1. General limit (`maxRestartAttempts`) is checked for every restart
+ 2. For failures, the most specific applicable limit is also checked:
+ - Scheduling failures (SchedulingFailure) → `maxRestartOnSchedulingFailure` (if set)
+ - Other failures → `maxRestartOnFailure` (if set)
+ 3. The application stops if any applicable limit is exceeded
+
+
+#### Configuration Fields
+
+```yaml
+restartConfig:
+ restartPolicy: Always
+ # Default restart configuration (applies to all restarts)
+ maxRestartAttempts: 5
+ restartBackoffMillis: 30000 # 30 seconds
+
+ # Override for consecutive general failures (application crashes, driver failures, etc.)
+ # This counter resets to 0 on success
+ maxRestartOnFailure: 3
+ restartBackoffMillisForFailure: 60000 # 1 minute
+
+ # Override for consecutive scheduling failures
+ maxRestartOnSchedulingFailure: 1
+ restartBackoffMillisForSchedulingFailure: 300000 # 5 minutes
+```
+
+#### Example Use Cases
+
+Tolerate transient failures but stop on persistent issues:
+
+```yaml
+restartConfig:
+ restartPolicy: Always
+ maxRestartAttempts: 100 # Allow many total attempts
+ restartBackoffMillis: 30000
+ # But stop after 3 consecutive failures (indicates persistent problem)
+ maxRestartOnFailure: 3
+ restartBackoffMillisForFailure: 60000
+```
+
+Mitigate API server stress during scheduling failures:
+
+```yaml
+restartConfig:
+ restartPolicy: Always
+ maxRestartAttempts: 50
+ restartBackoffMillis: 30000
+ # Stop quickly on scheduling failures to avoid overwhelming API server
+ maxRestartOnSchedulingFailure: 2
+ restartBackoffMillisForSchedulingFailure: 600000 # 10 minutes
+```
+
+
+| Field | Type | Default Value | Description |
+|-----------------------------------------------------------------------------------------|---------|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| .spec.applicationTolerations.restartConfig.restartPolicy | string | Never | Restart policy: `Never`, `Always`, `OnFailure`, or `OnInfrastructureFailure` |
+| .spec.applicationTolerations.restartConfig.maxRestartAttempts | integer | 3 | Maximum number of restart attempts for all scenarios (always checked) |
+| .spec.applicationTolerations.restartConfig.restartBackoffMillis | integer | 30000 | Default backoff time in milliseconds between restart attempts |
+| .spec.applicationTolerations.restartConfig.maxRestartOnFailure | integer | null | Maximum consecutive failures before stopping. Resets to 0 on success. If null, uses maxRestartAttempts |
Review Comment:
Shall we use `-1` default value for the not specified case? It seems that's the value we use with other fields as well (e.g. `restartCounterResetMillis`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55623] Add granular restart control with consecutive failure tracking [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #514:
URL: https://github.com/apache/spark-kubernetes-operator/pull/514#discussion_r2852788994
##########
build-tools/helm/spark-kubernetes-operator/crds/sparkclusters.spark.apache.org-v1.yaml:
##########
@@ -21996,10 +21996,14 @@ spec:
properties:
attemptInfo:
properties:
+ failureRestartCounter:
+ type: integer
Review Comment:
Given that the PR description and code, please revert this change on `SparkCluster`. I don't think this PR has the corresponding implementation, @jiangzho .
> This PR implements granular restart control for Spark applications with support for consecutive failure / scheduling failures tracking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55623] Add granular restart control with consecutive failure tracking [spark-kubernetes-operator]
Posted by "jiangzho (via GitHub)" <gi...@apache.org>.
jiangzho commented on PR #514:
URL: https://github.com/apache/spark-kubernetes-operator/pull/514#issuecomment-3947692122
cc @peter-toth may I get review on this please ? Thanks a lot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55623] Add granular restart control with consecutive failure tracking [spark-kubernetes-operator]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #514:
URL: https://github.com/apache/spark-kubernetes-operator/pull/514#discussion_r2845476254
##########
spark-operator-api/src/main/java/org/apache/spark/k8s/operator/status/AttemptInfo.java:
##########
@@ -40,13 +53,41 @@
public class AttemptInfo {
@Getter @Builder.Default protected final long id = 0L;
@Getter @Setter protected long restartCounter;
+ @Getter @Setter protected long failureRestartCounter;
+ @Getter @Setter protected long schedulingFailureRestartCounter;
/**
* Creates a new AttemptInfo object representing the next attempt.
- *
- * @return A new AttemptInfo with an incremented ID.
*/
- public AttemptInfo createNextAttemptInfo(boolean resetRestartCounter) {
- return new AttemptInfo(id + 1L, resetRestartCounter ? 1L : restartCounter + 1);
+ public AttemptInfo createNextAttemptInfo(
+ boolean resetRestartCounter, ApplicationStateSummary currentStateSummary) {
+ long newRestartCounter = resetRestartCounter ? 1L : restartCounter + 1;
+ long newFailureCounter;
+ long newSchedulingFailureCounter;
+
+ if (resetRestartCounter) {
+ // Reset all counters when restart counter is reset
+ newRestartCounter = 1L;
Review Comment:
This looks weird, you have already set `newRestartCounter` based on `resetRestartCounter` a few lines above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55623] Add granular restart control with consecutive failure tracking [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #514:
URL: https://github.com/apache/spark-kubernetes-operator/pull/514#discussion_r2891363077
##########
spark-operator-api/src/main/java/org/apache/spark/k8s/operator/status/BaseAttemptSummary.java:
##########
@@ -33,6 +33,6 @@
@RequiredArgsConstructor
@JsonInclude(JsonInclude.Include.NON_NULL)
@JsonIgnoreProperties(ignoreUnknown = true)
-public class BaseAttemptSummary {
- protected final AttemptInfo attemptInfo;
+public class BaseAttemptSummary<AI extends BaseAttemptInfo> {
+ protected final AI attemptInfo;
Review Comment:
Oh, `AI` looks like a bad naming. Please choose something else, @jiangzho .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55623] Add granular restart control with consecutive failure tracking [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #514:
URL: https://github.com/apache/spark-kubernetes-operator/pull/514#issuecomment-4012847305
Oh, @jiangzho . It seems that you are using a wrong parent when you create a JIRA ID.
<img width="1026" height="310" alt="Screenshot 2026-03-06 at 08 47 38" src="https://github.com/user-attachments/assets/b78ad748-ac89-44e1-a029-4193aa99a4bd" />
Do you have any reason not to use SPARK-55000?
<img width="534" height="82" alt="Screenshot 2026-03-06 at 08 48 47" src="https://github.com/user-attachments/assets/77ad3586-7060-4317-bddd-6f470d390c43" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55623] Add granular restart control with consecutive failure tracking [spark-kubernetes-operator]
Posted by "jiangzho (via GitHub)" <gi...@apache.org>.
jiangzho commented on code in PR #514:
URL: https://github.com/apache/spark-kubernetes-operator/pull/514#discussion_r2892302488
##########
spark-operator-api/src/main/java/org/apache/spark/k8s/operator/status/BaseAttemptSummary.java:
##########
@@ -33,6 +33,6 @@
@RequiredArgsConstructor
@JsonInclude(JsonInclude.Include.NON_NULL)
@JsonIgnoreProperties(ignoreUnknown = true)
-public class BaseAttemptSummary {
- protected final AttemptInfo attemptInfo;
+public class BaseAttemptSummary<AI extends BaseAttemptInfo> {
+ protected final AI attemptInfo;
Review Comment:
Updated to `I` to indicate the `Info` semantic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org