You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "wankunde (via GitHub)" <gi...@apache.org> on 2024/02/01 08:57:05 UTC
[PR] [SPARK-46942][CORE] Ignore --num-executors config if DYN_ALLOCATION_ENABLED is true and allow remove idle executors [spark]
wankunde opened a new pull request, #44981:
URL: https://github.com/apache/spark/pull/44981
### What changes were proposed in this pull request?
In https://issues.apache.org/jira/browse/SPARK-13723 , conf --num-executors will be used to compute initialExecutors and initial # of executors.
I think that may conflict to DYN_ALLOCATION_MIN_EXECUTORS and will not free the idle executors.
```
private def removeExecutors(executors: Seq[(String, Int)]): Seq[String] = synchronized {
if (newExecutorTotal - 1 < minNumExecutors) {
logDebug(s"Not removing idle executor $executorIdToBeRemoved because there " +
s"are only $newExecutorTotal executor(s) left (minimum number of executor limit " +
s"$minNumExecutors)")
} else if (newExecutorTotal - 1 < numExecutorsTargetPerResourceProfileId(rpId)) {
logDebug(s"Not removing idle executor $executorIdToBeRemoved because there " +
s"are only $newExecutorTotal executor(s) left (number of executor " +
s"target ${numExecutorsTargetPerResourceProfileId(rpId)})")
} else {
executorIdsToBeRemoved += executorIdToBeRemoved
numExecutorsTotalPerRpId(rpId) -= 1
}
}
```
### Why are the changes needed?
Remove the idle executors and improve the cluster resources utilization.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Local test.
Before this PR:
```
24/01/27 02:41:20 INFO Utils: Using initial executors = 1000, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances
```
After this PR:
```
24/01/27 02:51:20 INFO Utils: Using initial executors = 0, max of spark.dynamicAllocation.initialExecutors and spark.dynamicAllocation.minExecutors
```
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46942][CORE] Ignore --num-executors config if DYN_ALLOCATION_ENABLED is true and allow remove idle executors [spark]
Posted by "wankunde (via GitHub)" <gi...@apache.org>.
wankunde closed pull request #44981: [SPARK-46942][CORE] Ignore --num-executors config if DYN_ALLOCATION_ENABLED is true and allow remove idle executors
URL: https://github.com/apache/spark/pull/44981
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org