[jira] [Commented] (SPARK-27963) Allow dynamic allocation without an external shuffle service

["Marcelo Vanzin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-27963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16857146#comment-16857146 ] 

Marcelo Vanzin commented on SPARK-27963:
----------------------------------------

FYI I have a WIP patch to implement this that I plan to post soon (although I'll be out for a couple of weeks and won't be able to update it).

> Allow dynamic allocation without an external shuffle service
> ------------------------------------------------------------
>
>                 Key: SPARK-27963
>                 URL: https://issues.apache.org/jira/browse/SPARK-27963
>             Project: Spark
>          Issue Type: New Feature
>          Components: Spark Core
>    Affects Versions: 3.0.0
>            Reporter: Marcelo Vanzin
>            Priority: Major
>
> It would be useful for users to be able to enable dynamic allocation without the need to provision an external shuffle service. One immediate use case is the ability to use dynamic allocation on Kubernetes, which doesn't yet have that service.
> This has been suggested before (e.g. https://github.com/apache/spark/pull/24083, which was attached to the k8s-specific SPARK-24432), and can actually be done without affecting the internals of the Spark scheduler (aside from the dynamic allocation code). 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org