[jira] [Created] (SPARK-38447) Using Available Resources in Yarn Cluster Information in Spark Dynamic Allocation

["Abhishek Dixit (Jira)" <ji...@apache.org>]

Abhishek Dixit created SPARK-38447:
--------------------------------------

             Summary: Using Available Resources in Yarn Cluster Information in Spark Dynamic Allocation
                 Key: SPARK-38447
                 URL: https://issues.apache.org/jira/browse/SPARK-38447
             Project: Spark
          Issue Type: Improvement
          Components: YARN
    Affects Versions: 3.2.1
            Reporter: Abhishek Dixit


Yarn Cluster Manager Provides information on available resources (VCores / Memory) in the Cluster via AM-RM heartbeat. In autoscaling, usually, latency to start executor containers on nodes that are immediately available in the cluster is comparatively lesser as compared to latency in adding new nodes to the cluster. Dynamic Allocation can leverage this information of the number of resources immediately available and latency in adding new nodes in deciding the number of executors to be requested from Yarn RM.

This improvement can be built in two parts:
 #  Infra to Send Available VCores and Memory information from Yarn AM-RM heartbeat response to ExecutorAllocationClient.
 # Leveraging Available VCores and Memory information in ExecutorAllocationManger to decide the number of executors to be requested from Yarn RM.

I'll create PRs for both the tasks one by one.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org