You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Wan Kun (Jira)" <ji...@apache.org> on 2024/02/01 08:47:00 UTC
[jira] [Created] (SPARK-46942) Ignore --num-executors config if DYN_ALLOCATION_ENABLED is true and allow remove idle executors
Wan Kun created SPARK-46942:
-------------------------------
Summary: Ignore --num-executors config if DYN_ALLOCATION_ENABLED is true and allow remove idle executors
Key: SPARK-46942
URL: https://issues.apache.org/jira/browse/SPARK-46942
Project: Spark
Issue Type: Improvement
Components: YARN
Affects Versions: 4.0.0
Reporter: Wan Kun
In https://issues.apache.org/jira/browse/SPARK-13723 , conf --num-executors will be used to compute initialExecutors and initial # of executors.
I think that may conflict to DYN_ALLOCATION_MIN_EXECUTORS and will not free the idle executors.
{code}
...
private def removeExecutors(executors: Seq[(String, Int)]): Seq[String] = synchronized {
if (newExecutorTotal - 1 < minNumExecutors) {
logDebug(s"Not removing idle executor $executorIdToBeRemoved because there " +
s"are only $newExecutorTotal executor(s) left (minimum number of executor limit " +
s"$minNumExecutors)")
} else if (newExecutorTotal - 1 < numExecutorsTargetPerResourceProfileId(rpId)) {
logDebug(s"Not removing idle executor $executorIdToBeRemoved because there " +
s"are only $newExecutorTotal executor(s) left (number of executor " +
s"target ${numExecutorsTargetPerResourceProfileId(rpId)})")
} else {
executorIdsToBeRemoved += executorIdToBeRemoved
numExecutorsTotalPerRpId(rpId) -= 1
}
}
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org