You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Thomas Graves (Jira)" <ji...@apache.org> on 2023/03/20 19:05:00 UTC
[jira] [Resolved] (SPARK-41585) The Spark exclude node functionality for YARN should work independently of dynamic allocation
[ https://issues.apache.org/jira/browse/SPARK-41585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves resolved SPARK-41585.
-----------------------------------
Fix Version/s: 3.5.0
Target Version/s: 3.5.0
Assignee: Luca Canali
Resolution: Fixed
> The Spark exclude node functionality for YARN should work independently of dynamic allocation
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-41585
> URL: https://issues.apache.org/jira/browse/SPARK-41585
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 3.0.3, 3.1.3, 3.2.2, 3.3.1
> Reporter: Luca Canali
> Assignee: Luca Canali
> Priority: Minor
> Fix For: 3.5.0
>
>
> The Spark exclude node functionality for Spark on YARN, introduced in SPARK-26688, allows users to specify a list of node names that are excluded from resource allocation. This is done using the configuration parameter: {{spark.yarn.exclude.nodes}}
> The feature currently works only for executors allocated via dynamic allocation. To use the feature on Spark 3.3.1, for example, one may set the configurations {{{}spark.dynamicAllocation.enabled{}}}=true, spark.dynamicAllocation.minExecutors=0 and spark.executor.instances=0, thus making Spark spawning executors only via dynamic allocation.
> This proposes to document this behavior for the current Spark release and also proposes an improvement of this feature by extending the scope of Spark exclude node functionality for YARN beyond dynamic allocation, which I believe makes it more generally useful.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org