[jira] [Created] (SPARK-44951) Improve Spark Dynamic Allocation

["Holden Karau (Jira)" <ji...@apache.org>]

Holden Karau created SPARK-44951:
------------------------------------

             Summary: Improve Spark Dynamic Allocation
                 Key: SPARK-44951
                 URL: https://issues.apache.org/jira/browse/SPARK-44951
             Project: Spark
          Issue Type: Improvement
          Components: Kubernetes, Spark Core, YARN
    Affects Versions: 4.0.0
            Reporter: Holden Karau


For Spark 4 we should aim to improve Spark's dynamic allocation. Some potential ideas here includes the following:
 * Plug-gable DEA algorithms
 * How to reduce wastage on the RM side? Sometimes the driver asks for some units of resources. But when RM provisions them, the driver cancels it. 
 * Support for "warm" executor pools which are not tied to a particular driver but start and wait for a driver to connect to them to "claim" them.
 * More explicit Cost Vs AppRunTime confiugration: A good DEA algo should allow the developer to choose between cost and runtime. Sometimes developers might be ok to pay higher costs for faster execution.
 * Use previous run information to inform future runs
 * Better selection of executors to be scaled down



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org