You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Wan Kun (Jira)" <ji...@apache.org> on 2024/02/01 08:47:00 UTC

[jira] [Created] (SPARK-46942) Ignore --num-executors config if DYN_ALLOCATION_ENABLED is true and allow remove idle executors

Wan Kun created SPARK-46942:
-------------------------------

             Summary: Ignore --num-executors config if DYN_ALLOCATION_ENABLED is true and allow remove idle executors
                 Key: SPARK-46942
                 URL: https://issues.apache.org/jira/browse/SPARK-46942
             Project: Spark
          Issue Type: Improvement
          Components: YARN
    Affects Versions: 4.0.0
            Reporter: Wan Kun


In https://issues.apache.org/jira/browse/SPARK-13723 , conf --num-executors will be used to compute initialExecutors and initial # of executors.

I think that may conflict to DYN_ALLOCATION_MIN_EXECUTORS and will not free the idle executors.

{code}
... 
  private def removeExecutors(executors: Seq[(String, Int)]): Seq[String] = synchronized {
        if (newExecutorTotal - 1 < minNumExecutors) {
          logDebug(s"Not removing idle executor $executorIdToBeRemoved because there " +
            s"are only $newExecutorTotal executor(s) left (minimum number of executor limit " +
            s"$minNumExecutors)")
        } else if (newExecutorTotal - 1 < numExecutorsTargetPerResourceProfileId(rpId)) {
          logDebug(s"Not removing idle executor $executorIdToBeRemoved because there " +
            s"are only $newExecutorTotal executor(s) left (number of executor " +
            s"target ${numExecutorsTargetPerResourceProfileId(rpId)})")
        } else {
          executorIdsToBeRemoved += executorIdToBeRemoved
          numExecutorsTotalPerRpId(rpId) -= 1
        }
  }
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org