You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@celeborn.apache.org by zh...@apache.org on 2023/06/16 10:38:18 UTC
[incubator-celeborn] branch branch-0.3 updated: [CELEBORN-687] Fix shuffleResourceExists, reduce unexpected slot release request
This is an automated email from the ASF dual-hosted git repository.
zhouky pushed a commit to branch branch-0.3
in repository https://gitbox.apache.org/repos/asf/incubator-celeborn.git
The following commit(s) were added to refs/heads/branch-0.3 by this push:
new 380badb81 [CELEBORN-687] Fix shuffleResourceExists, reduce unexpected slot release request
380badb81 is described below
commit 380badb81668c4af98a517f65da5125cb027cf73
Author: Shuang <lv...@gmail.com>
AuthorDate: Fri Jun 16 18:37:48 2023 +0800
[CELEBORN-687] Fix shuffleResourceExists, reduce unexpected slot release request
### What changes were proposed in this pull request?
Check ShufflePartitionLocationInfo whether empty or not for every worker
### Why are the changes needed?
Actually shuffleResources would only remove related partitionLocations after stageEnd , then workers with empty partitionLocations will left(for speculative task), so shuffleResourceExists need check ShufflePartitionLocationInfo for every worker otherwise it would print wrong message and send release slot requests twice.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manual test
### Before this pr
<img width="1252" alt="image" src="https://github.com/apache/incubator-celeborn/assets/28799061/06b71162-e78b-4163-8f52-24b50bc6c540">
### After this pr
Closes #1599 from RexXiong/CELEBORN-687.
Authored-by: Shuang <lv...@gmail.com>
Signed-off-by: zky.zhoukeyong <zk...@alibaba-inc.com>
(cherry picked from commit 28a99ded8dce6a1a20f258d8f5d00e0bb37c8bb5)
Signed-off-by: zky.zhoukeyong <zk...@alibaba-inc.com>
---
.../src/main/scala/org/apache/celeborn/client/LifecycleManager.scala | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/client/src/main/scala/org/apache/celeborn/client/LifecycleManager.scala b/client/src/main/scala/org/apache/celeborn/client/LifecycleManager.scala
index 6100c75b0..77dcfc559 100644
--- a/client/src/main/scala/org/apache/celeborn/client/LifecycleManager.scala
+++ b/client/src/main/scala/org/apache/celeborn/client/LifecycleManager.scala
@@ -1133,8 +1133,8 @@ class LifecycleManager(appId: String, val conf: CelebornConf) extends RpcEndpoin
}
private def shuffleResourceExists(shuffleId: Int): Boolean = {
- val workers = workerSnapshots(shuffleId)
- workers != null && !workers.isEmpty
+ val workerPartitionInfos = workerSnapshots(shuffleId)
+ workerPartitionInfos != null && workerPartitionInfos.values().asScala.exists(!_.isEmpty())
}
// Once a partition is released, it will be never needed anymore