You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@celeborn.apache.org by et...@apache.org on 2023/06/20 10:43:24 UTC
[incubator-celeborn] branch branch-0.3 updated: [CELEBORN-701][COMPATIBILITY] Fix compatibility issue caused by pushdata timeout
This is an automated email from the ASF dual-hosted git repository.
ethanfeng pushed a commit to branch branch-0.3
in repository https://gitbox.apache.org/repos/asf/incubator-celeborn.git
The following commit(s) were added to refs/heads/branch-0.3 by this push:
new 162aaa8d2 [CELEBORN-701][COMPATIBILITY] Fix compatibility issue caused by pushdata timeout
162aaa8d2 is described below
commit 162aaa8d210066467a6e0bfd55f34dc9be669920
Author: zky.zhoukeyong <zk...@alibaba-inc.com>
AuthorDate: Tue Jun 20 18:43:02 2023 +0800
[CELEBORN-701][COMPATIBILITY] Fix compatibility issue caused by pushdata timeout
### What changes were proposed in this pull request?
Fixes compatibility issue caused by change of push data timeout.
### Why are the changes needed?
When I test with branch-0.2 client with main server side, I got the following error:
```
23/06/20 17:42:34,538 ERROR [push-timeout-checker-12] PushDataHandler: PushData replication failed for partitionLocation: PartitionLocation[
id-epoch:767-0
host-rpcPort-pushPort-fetchPort-replicatePort:192.168.1.17-37687-42605-42891-41319
mode:MASTER
peer:(host-rpcPort-pushPort-fetchPort-replicatePort:192.168.1.18-45201-35749-41831-46853)
storage hint:StorageInfo{type=MEMORY, mountPoint='/mnt/disk4', finalResult=false, filePath=}
mapIdBitMap:null]
org.apache.celeborn.common.exception.CelebornIOException: PUSH_DATA_TIMEOUT_SLAVE
at org.apache.celeborn.common.network.client.TransportResponseHandler.failExpiredPushRequest(TransportResponseHandler.java:125)
at org.apache.celeborn.common.network.client.TransportResponseHandler.lambda$new$0(TransportResponseHandler.java:96)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
```
The error is because in main branch ReserveSlots added a new field ```pushDataTimeout```, so when client is branch-0.2, the default value will be 0, and always trigger timeout.
### Does this PR introduce _any_ user-facing change?
Yes, this PR fixes a bug when client is branch-0.2 and server is main.
### How was this patch tested?
Manual test.
Closes #1611 from waitinfuture/701.
Authored-by: zky.zhoukeyong <zk...@alibaba-inc.com>
Signed-off-by: mingji <fe...@alibaba-inc.com>
(cherry picked from commit 6ca2bb2c6f55c20a713a7e2de080814e82f930f4)
Signed-off-by: mingji <fe...@alibaba-inc.com>
---
.../scala/org/apache/celeborn/service/deploy/worker/Controller.scala | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/Controller.scala b/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/Controller.scala
index 57cba3425..ca4b6dc9e 100644
--- a/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/Controller.scala
+++ b/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/Controller.scala
@@ -60,6 +60,7 @@ private[deploy] class Controller(
var asyncReplyPool: ScheduledExecutorService = _
val minPartitionSizeToEstimate = conf.minPartitionSizeToEstimate
var shutdown: AtomicBoolean = _
+ val defaultPushdataTimeout = conf.pushDataTimeoutMs
val testRetryCommitFiles = conf.testRetryCommitFiles
@@ -242,7 +243,9 @@ private[deploy] class Controller(
partitionLocationInfo.addMasterPartitions(shuffleKey, masterLocs)
partitionLocationInfo.addSlavePartitions(shuffleKey, slaveLocs)
shufflePartitionType.put(shuffleKey, partitionType)
- shufflePushDataTimeout.put(shuffleKey, pushDataTimeout)
+ shufflePushDataTimeout.put(
+ shuffleKey,
+ if (pushDataTimeout <= 0) defaultPushdataTimeout else pushDataTimeout)
workerInfo.allocateSlots(shuffleKey, Utils.getSlotsPerDisk(requestMasterLocs, requestSlaveLocs))
logInfo(s"Reserved ${masterLocs.size()} master location" +