You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "jiangzho (via GitHub)" <gi...@apache.org> on 2025/07/22 01:37:31 UTC
[PR] [SPARK-52915] Support TTL for Spark apps [spark-kubernetes-operator]
jiangzho opened a new pull request, #290:
URL: https://github.com/apache/spark-kubernetes-operator/pull/290
### What changes were proposed in this pull request?
This PR adds support for configuring the ttl for Spark apps after it stops. Working with the `resourceRetainPolicy` and `resourceRetainDurationMillis`, it enhances the garbage collection mechanism at the custom resource level.
### Why are the changes needed?
Introducing TTL helps user to more effectively configure the garbage collection for apps.
### Does this PR introduce _any_ user-facing change?
New configurable field spec.applicationTolerations.ttlAfterStopMillis added to SparkApplication CRD
### How was this patch tested?
CIs - including new unit test and revised e2e scenario
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52915] Support TTL for Spark apps [spark-kubernetes-operator]
Posted by "jiangzho (via GitHub)" <gi...@apache.org>.
jiangzho commented on code in PR #290:
URL: https://github.com/apache/spark-kubernetes-operator/pull/290#discussion_r2243837781
##########
docs/spark_custom_resources.md:
##########
@@ -293,6 +293,8 @@ applicationTolerations:
resourceRetainPolicy: OnFailure
# Secondary resources would be garbage collected 10 minutes after app termination
resourceRetainDurationMillis: 600000
+ # Garbage collect the SparkApplication custom resource itself 30 minutes after termination
+ ttlAfterStopMillis: 1800000
Review Comment:
Yes, it is. This is only an example value placed in this snippet. I can add a chart for the actual default values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52915] Support TTL for Spark apps [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #290:
URL: https://github.com/apache/spark-kubernetes-operator/pull/290#discussion_r2241065150
##########
docs/spark_custom_resources.md:
##########
@@ -302,7 +304,11 @@ possible to configure `resourceRetainDurationMillis` to define the maximal retai
these resources. Note that this applies only to operator-created resources (driver pod, SparkConf
configmap .etc). You may also want to tune `spark.kubernetes.driver.service.deleteOnTermination`
and `spark.kubernetes.executor.deleteOnTermination` to control the behavior of driver-created
-resources.
+resources. `ttlAfterStopMillis` controls the garbage collection behavior at the SparkApplication
+level after it stops. When set to a non-negative value, Spark operator would garbage collect the
+application (and therefore all its associated resources) after given timeout. If the application
+is configured to restart, `resourceRetainPolicy`, `resourceRetainDurationMillis` and
+`ttlAfterStopMillis` would be applied only to the last attempt.
Review Comment:
I read the code. So, `Math.min(resourceRetainDurationMillis, ttlAfterStopMillis)` is applied, right? I believe we need to rewrite this sentence. Given that the logic, this looks inaccurate to me.
> `ttlAfterStopMillis` would be applied only to the last attempt.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52915] Support TTL for Spark apps [spark-kubernetes-operator]
Posted by "jiangzho (via GitHub)" <gi...@apache.org>.
jiangzho commented on PR #290:
URL: https://github.com/apache/spark-kubernetes-operator/pull/290#issuecomment-3114223003
cc @peter-toth can you please take a look ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52915] Support TTL for Spark apps [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #290:
URL: https://github.com/apache/spark-kubernetes-operator/pull/290#discussion_r2241060160
##########
docs/spark_custom_resources.md:
##########
@@ -302,7 +304,11 @@ possible to configure `resourceRetainDurationMillis` to define the maximal retai
these resources. Note that this applies only to operator-created resources (driver pod, SparkConf
configmap .etc). You may also want to tune `spark.kubernetes.driver.service.deleteOnTermination`
and `spark.kubernetes.executor.deleteOnTermination` to control the behavior of driver-created
-resources.
+resources. `ttlAfterStopMillis` controls the garbage collection behavior at the SparkApplication
+level after it stops. When set to a non-negative value, Spark operator would garbage collect the
+application (and therefore all its associated resources) after given timeout. If the application
+is configured to restart, `resourceRetainPolicy`, `resourceRetainDurationMillis` and
+`ttlAfterStopMillis` would be applied only to the last attempt.
Review Comment:
What does this mean, @jiangzho ?
- `ttlAfterStopMillis` is ignored when `resourceRetainDurationMillis` configuration exists?
- Or, `max(ttlAfterStopMillis , resourceRetainDurationMillis)` is applied?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52915] Support TTL for Spark apps [spark-kubernetes-operator]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #290:
URL: https://github.com/apache/spark-kubernetes-operator/pull/290#issuecomment-3187512778
Thanks @jiangzho for the fix and @dongjoon-hyun for the review!
Merged to `main` (0.5.0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52915] Support TTL for Spark apps [spark-kubernetes-operator]
Posted by "jiangzho (via GitHub)" <gi...@apache.org>.
jiangzho commented on code in PR #290:
URL: https://github.com/apache/spark-kubernetes-operator/pull/290#discussion_r2244010488
##########
docs/spark_custom_resources.md:
##########
@@ -302,7 +304,11 @@ possible to configure `resourceRetainDurationMillis` to define the maximal retai
these resources. Note that this applies only to operator-created resources (driver pod, SparkConf
configmap .etc). You may also want to tune `spark.kubernetes.driver.service.deleteOnTermination`
and `spark.kubernetes.executor.deleteOnTermination` to control the behavior of driver-created
-resources.
+resources. `ttlAfterStopMillis` controls the garbage collection behavior at the SparkApplication
+level after it stops. When set to a non-negative value, Spark operator would garbage collect the
+application (and therefore all its associated resources) after given timeout. If the application
+is configured to restart, `resourceRetainPolicy`, `resourceRetainDurationMillis` and
+`ttlAfterStopMillis` would be applied only to the last attempt.
Review Comment:
Ah sorry for the slightly misleading statement, it actually refers to another perspective.
* All 3 fields `ttlAfterStopMillis` , `resourceRetainDurationMillis` and `resourceRetainPolicy` apply to the last attempt only if the app is configured to restart
- When an app is configured to restart, all resources related to one single attempt(driver, executor, ...) would be released before making the next attempt, regardless of the values configured in above 3 fields - except for the last attempt, when no restart expected.
For example, if I do configure
```
applicationTolerations:
restartConfig:
restartPolicy: OnFailure
maxRestartAttempts: 1
resourceRetainPolicy: Always
resourceRetainDurationMillis: 30000
ttlAfterStopMillis: 60000
```
and my app ends up with status like
```
status:
... # the 1st attempt
"5":
currentStateSummary: Failed
lastTransitionTime: "2025-07-30T22:43:15.293414Z"
"6":
currentStateSummary: ScheduledToRestart
lastTransitionTime: "2025-07-30T22:43:15.406645Z"
... # the 2nd attempt
"11":
currentStateSummary: Succeeded
lastTransitionTime: "2025-07-30T22:44:15.503645Z"
"12":
currentStateSummary: TerminatedWithoutReleaseResources
lastTransitionTime: "2025-07-30T22:44:15.503645Z"
```
The retain policy only takes effect after state `12` - resources are always released between attempts between `5` and `6`
Thanks for calling out the `Math.min(resourceRetainDurationMillis, ttlAfterStopMillis)` prt - forgot to mention it in doc. I'll add it in chart mentioned above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52915] Support TTL for Spark apps [spark-kubernetes-operator]
Posted by "jiangzho (via GitHub)" <gi...@apache.org>.
jiangzho commented on PR #290:
URL: https://github.com/apache/spark-kubernetes-operator/pull/290#issuecomment-3161413997
@dongjoon-hyun gentle nudge on this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52915] Support TTL for Spark apps [spark-kubernetes-operator]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #290:
URL: https://github.com/apache/spark-kubernetes-operator/pull/290#issuecomment-3114290506
@jiangzho , I will try to take a look at this PR tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52915] Support TTL for Spark apps [spark-kubernetes-operator]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth closed pull request #290: [SPARK-52915] Support TTL for Spark apps
URL: https://github.com/apache/spark-kubernetes-operator/pull/290
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52915] Support TTL for Spark apps [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #290:
URL: https://github.com/apache/spark-kubernetes-operator/pull/290#discussion_r2241056867
##########
docs/spark_custom_resources.md:
##########
@@ -293,6 +293,8 @@ applicationTolerations:
resourceRetainPolicy: OnFailure
# Secondary resources would be garbage collected 10 minutes after app termination
resourceRetainDurationMillis: 600000
+ # Garbage collect the SparkApplication custom resource itself 30 minutes after termination
+ ttlAfterStopMillis: 1800000
Review Comment:
The default value is `-1` in the code, isn't it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52915] Support TTL for Spark apps [spark-kubernetes-operator]
Posted by "jiangzho (via GitHub)" <gi...@apache.org>.
jiangzho commented on PR #290:
URL: https://github.com/apache/spark-kubernetes-operator/pull/290#issuecomment-3146006000
@dongjoon-hyun - would you mind give a second round of review please ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org