You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dongjoon-hyun (via GitHub)" <gi...@apache.org> on 2026/10/01 04:56:25 UTC
[PR] [SPARK-59918][K8S] Support `deployment` pods allocator in `ExecutorResizePlugin` [spark]
dongjoon-hyun opened a new pull request, #59180:
URL: https://github.com/apache/spark/pull/59180
### What changes were proposed in this pull request?
This PR aims to support `ExecutorResizePlugin` with the `deployment` pods allocator in addition to the default `direct` pods allocator.
### Why are the changes needed?
[SPARK-56670](https://github.com/apache/spark/pull/55615) restricted `ExecutorResizePlugin` to the `direct` pods allocator because the higher-level controller's pod template is the source of truth for the other allocators. However, this is not a concern for `Deployment`. Spark never updates its pod template, and the `ReplicaSet` controller neither reverts in-place pod resizes nor rolls out a new revision for them.
### Does this PR introduce _any_ user-facing change?
Yes. In Apache Spark 4.2.0, `ExecutorResizePlugin` logs a warning and does not start with the `deployment` pods allocator. After this PR, it works with the `deployment` pods allocator.
### How was this patch tested?
Pass the CIs with the updated test case.
I also verified manually on K8s v1.37.0 that the executor pods of the `Deployment` are resized in place up to `spark.kubernetes.executor.resizeMaxMemory` without restarts, while the `Deployment` keeps its pod template and a single `ReplicaSet`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59918][K8S] Support `deployment` pods allocator in `ExecutorResizePlugin` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #59180:
URL: https://github.com/apache/spark/pull/59180#issuecomment-5925069196
cc @peter-toth
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59918][K8S] Support `deployment` pods allocator in `ExecutorResizePlugin` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #59180:
URL: https://github.com/apache/spark/pull/59180#issuecomment-5925080004
Thank you so much, @HyukjinKwon !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59918][K8S] Support `deployment` pods allocator in `ExecutorResizePlugin` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #59180:
URL: https://github.com/apache/spark/pull/59180#issuecomment-5925296167
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/913175ed22f616bb229d9f4533a398f832680cd5
- merged into branch-4.x https://github.com/apache/spark/commit/d4c47c69c0dd9de2400861017688fb542af898f9
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59918][K8S] Support `deployment` pods allocator in `ExecutorResizePlugin` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #59180: [SPARK-59918][K8S] Support `deployment` pods allocator in `ExecutorResizePlugin`
URL: https://github.com/apache/spark/pull/59180
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org