You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by ch...@apache.org on 2023/06/08 12:17:50 UTC
[kyuubi] 01/01: [KYUUBI#4935] More than target num of executors may survive after FinalStageResourceManager did kill
This is an automated email from the ASF dual-hosted git repository.
chengpan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
commit d8415aa3862de7136bce7fa6c6fc258b8bc8ccc2
Author: zhouyifan279 <zh...@gmail.com>
AuthorDate: Thu Jun 8 20:16:39 2023 +0800
[KYUUBI#4935] More than target num of executors may survive after FinalStageResourceManager did kill
### _Why are the changes needed?_
When FinalStageResourceManager chooses executors to be killed, it may add dead executors to the kill list.
This will leave more than target num of executors survived and cause resource waste.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes #4936 from zhouyifan279/kill-executor.
Closes #4936
2aaa84cb1 [zhouyifan279] [KYUUBI#4935][Improvement] More than target num of executors may survive after FinalStageResourceManager did kill
Authored-by: zhouyifan279 <zh...@gmail.com>
Signed-off-by: Cheng Pan <ch...@apache.org>
---
.../src/main/scala/org/apache/spark/sql/FinalStageResourceManager.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/extensions/spark/kyuubi-extension-spark-3-3/src/main/scala/org/apache/spark/sql/FinalStageResourceManager.scala b/extensions/spark/kyuubi-extension-spark-3-3/src/main/scala/org/apache/spark/sql/FinalStageResourceManager.scala
index 7a0ae1592..dc573f838 100644
--- a/extensions/spark/kyuubi-extension-spark-3-3/src/main/scala/org/apache/spark/sql/FinalStageResourceManager.scala
+++ b/extensions/spark/kyuubi-extension-spark-3-3/src/main/scala/org/apache/spark/sql/FinalStageResourceManager.scala
@@ -170,7 +170,7 @@ case class FinalStageResourceManager(session: SparkSession)
// Evict the rest executors according to the shuffle block size
executorToBlockSize.toSeq.sortBy(_._2).foreach { case (id, _) =>
- if (executorIdsToKill.length < expectedNumExecutorToKill) {
+ if (executorIdsToKill.length < expectedNumExecutorToKill && existedExecutors.contains(id)) {
executorIdsToKill.append(id)
}
}