You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@celeborn.apache.org by et...@apache.org on 2023/06/20 10:43:24 UTC

[incubator-celeborn] branch branch-0.3 updated: [CELEBORN-701][COMPATIBILITY] Fix compatibility issue caused by pushdata timeout

This is an automated email from the ASF dual-hosted git repository.

ethanfeng pushed a commit to branch branch-0.3
in repository https://gitbox.apache.org/repos/asf/incubator-celeborn.git


The following commit(s) were added to refs/heads/branch-0.3 by this push:
     new 162aaa8d2 [CELEBORN-701][COMPATIBILITY] Fix compatibility issue caused by pushdata timeout
162aaa8d2 is described below

commit 162aaa8d210066467a6e0bfd55f34dc9be669920
Author: zky.zhoukeyong <zk...@alibaba-inc.com>
AuthorDate: Tue Jun 20 18:43:02 2023 +0800

    [CELEBORN-701][COMPATIBILITY] Fix compatibility issue caused by pushdata timeout
    
    ### What changes were proposed in this pull request?
    Fixes compatibility issue caused by change of push data timeout.
    
    ### Why are the changes needed?
    When I test with branch-0.2 client with main server side, I got the following error:
    ```
    23/06/20 17:42:34,538 ERROR [push-timeout-checker-12] PushDataHandler: PushData replication failed for partitionLocation: PartitionLocation[
      id-epoch:767-0
      host-rpcPort-pushPort-fetchPort-replicatePort:192.168.1.17-37687-42605-42891-41319
      mode:MASTER
      peer:(host-rpcPort-pushPort-fetchPort-replicatePort:192.168.1.18-45201-35749-41831-46853)
      storage hint:StorageInfo{type=MEMORY, mountPoint='/mnt/disk4', finalResult=false, filePath=}
      mapIdBitMap:null]
    org.apache.celeborn.common.exception.CelebornIOException: PUSH_DATA_TIMEOUT_SLAVE
            at org.apache.celeborn.common.network.client.TransportResponseHandler.failExpiredPushRequest(TransportResponseHandler.java:125)
            at org.apache.celeborn.common.network.client.TransportResponseHandler.lambda$new$0(TransportResponseHandler.java:96)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:750)
    ```
    The error is because in main branch ReserveSlots added a new field ```pushDataTimeout```, so when client is branch-0.2, the default value will be 0, and always trigger timeout.
    
    ### Does this PR introduce _any_ user-facing change?
    Yes, this PR fixes a bug when client is branch-0.2 and server is main.
    
    ### How was this patch tested?
    Manual test.
    
    Closes #1611 from waitinfuture/701.
    
    Authored-by: zky.zhoukeyong <zk...@alibaba-inc.com>
    Signed-off-by: mingji <fe...@alibaba-inc.com>
    (cherry picked from commit 6ca2bb2c6f55c20a713a7e2de080814e82f930f4)
    Signed-off-by: mingji <fe...@alibaba-inc.com>
---
 .../scala/org/apache/celeborn/service/deploy/worker/Controller.scala | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/Controller.scala b/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/Controller.scala
index 57cba3425..ca4b6dc9e 100644
--- a/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/Controller.scala
+++ b/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/Controller.scala
@@ -60,6 +60,7 @@ private[deploy] class Controller(
   var asyncReplyPool: ScheduledExecutorService = _
   val minPartitionSizeToEstimate = conf.minPartitionSizeToEstimate
   var shutdown: AtomicBoolean = _
+  val defaultPushdataTimeout = conf.pushDataTimeoutMs
 
   val testRetryCommitFiles = conf.testRetryCommitFiles
 
@@ -242,7 +243,9 @@ private[deploy] class Controller(
     partitionLocationInfo.addMasterPartitions(shuffleKey, masterLocs)
     partitionLocationInfo.addSlavePartitions(shuffleKey, slaveLocs)
     shufflePartitionType.put(shuffleKey, partitionType)
-    shufflePushDataTimeout.put(shuffleKey, pushDataTimeout)
+    shufflePushDataTimeout.put(
+      shuffleKey,
+      if (pushDataTimeout <= 0) defaultPushdataTimeout else pushDataTimeout)
     workerInfo.allocateSlots(shuffleKey, Utils.getSlotsPerDisk(requestMasterLocs, requestSlaveLocs))
 
     logInfo(s"Reserved ${masterLocs.size()} master location" +