You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@celeborn.apache.org by ch...@apache.org on 2024/01/17 12:01:12 UTC
(incubator-celeborn) branch branch-0.4 updated: [CELEBORN-1218] Optimize dataPusher to get partitionLocationMap only once
This is an automated email from the ASF dual-hosted git repository.
chengpan pushed a commit to branch branch-0.4
in repository https://gitbox.apache.org/repos/asf/incubator-celeborn.git
The following commit(s) were added to refs/heads/branch-0.4 by this push:
new 94a344c2c [CELEBORN-1218] Optimize dataPusher to get partitionLocationMap only once
94a344c2c is described below
commit 94a344c2c479a30a42f73a7f09361eae06d19afe
Author: liangyongyuan <li...@xiaomi.com>
AuthorDate: Wed Jan 17 20:00:44 2024 +0800
[CELEBORN-1218] Optimize dataPusher to get partitionLocationMap only once
### What changes were proposed in this pull request?
dataPusher fetches partitionLocationMap only once outside the loop.
### Why are the changes needed?
If an exception occurs while obtaining partitionLocationMap in dataPusher.takePushTasks, it will result in attempting to fetch partitionLocationMap for each iteration in the workingQueue traversal. This leads to an unusually prolonged job execution time.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Through existing uts
Closes #2220 from lyy-pineapple/CELEBORN-1218.
Authored-by: liangyongyuan <li...@xiaomi.com>
Signed-off-by: Cheng Pan <ch...@apache.org>
(cherry picked from commit 78b77dc27f94061453a78381872c8908c45cbc8f)
Signed-off-by: Cheng Pan <ch...@apache.org>
---
.../apache/celeborn/client/write/DataPushQueue.java | 20 ++++++++++----------
1 file changed, 10 insertions(+), 10 deletions(-)
diff --git a/client/src/main/java/org/apache/celeborn/client/write/DataPushQueue.java b/client/src/main/java/org/apache/celeborn/client/write/DataPushQueue.java
index 5d4759f7e..84fb1ab81 100644
--- a/client/src/main/java/org/apache/celeborn/client/write/DataPushQueue.java
+++ b/client/src/main/java/org/apache/celeborn/client/write/DataPushQueue.java
@@ -90,13 +90,16 @@ public class DataPushQueue {
// takeTaskWaitTimeMs
// in last loop
workerCapacity.clear();
- Iterator<PushTask> iterator = workingQueue.iterator();
- while (iterator.hasNext()) {
- PushTask task = iterator.next();
- int partitionId = task.getPartitionId();
- Map<Integer, PartitionLocation> partitionLocationMap =
- client.getPartitionLocation(shuffleId, numMappers, numPartitions);
- if (partitionLocationMap != null) {
+ Map<Integer, PartitionLocation> partitionLocationMap =
+ client.getPartitionLocation(shuffleId, numMappers, numPartitions);
+ if (partitionLocationMap == null) {
+ tasks.addAll(workingQueue);
+ workingQueue.clear();
+ } else {
+ Iterator<PushTask> iterator = workingQueue.iterator();
+ while (iterator.hasNext()) {
+ PushTask task = iterator.next();
+ int partitionId = task.getPartitionId();
PartitionLocation loc = partitionLocationMap.get(partitionId);
// According to CELEBORN-560, call rerun task and speculative task after LifecycleManager
// handle StageEnd will return empty PartitionLocation map, here loc can be null
@@ -122,9 +125,6 @@ public class DataPushQueue {
iterator.remove();
tasks.add(task);
}
- } else {
- iterator.remove();
- tasks.add(task);
}
}
if (!tasks.isEmpty()) {