[jira] [Commented] (SPARK-36964) Reuse CachedDNSToSwitchMapping for yarn container requests

["Apache Spark (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-36964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17426521#comment-17426521 ] 

Apache Spark commented on SPARK-36964:
--------------------------------------

User 'gaoyajun02' has created a pull request for this issue:
https://github.com/apache/spark/pull/34231

> Reuse CachedDNSToSwitchMapping for yarn  container requests
> -----------------------------------------------------------
>
>                 Key: SPARK-36964
>                 URL: https://issues.apache.org/jira/browse/SPARK-36964
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core, YARN
>    Affects Versions: 3.0.3, 3.1.2
>            Reporter: gaoyajun02
>            Priority: Major
>
> Similar to SPARK-13704​, In some cases, YarnAllocator add or remove container requests can be expensive, it may call the topology script for rack awareness.
> When submit a very large job in a very large Yarn cluster, the topology script may take signifiant time to run. And this blocks receiving YarnSchedulerBackend's RequestExecutors rpc calls, This request comes from spark dynamic executor allocation thread, which may blocks the ExecutorAllocationListener, and then result in executorManagement queue backlog.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org