You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "RexXiong (via GitHub)" <gi...@apache.org> on 2023/09/21 12:58:04 UTC

[GitHub] [incubator-celeborn] RexXiong commented on a diff in pull request #1924: [CELEBORN-955] Re-run Spark Stage for Celeborn Shuffle Fetch Failure

RexXiong commented on code in PR #1924:
URL: https://github.com/apache/incubator-celeborn/pull/1924#discussion_r1333011153


##########
client/src/main/scala/org/apache/celeborn/client/LifecycleManager.scala:
##########
@@ -556,6 +562,28 @@ class LifecycleManager(val appUniqueId: String, val conf: CelebornConf) extends
     commitManager.handleGetReducerFileGroup(context, shuffleId)
   }
 
+  private def handleGetShuffleId(
+      context: RpcCallContext,
+      appShuffleId: Int,
+      maxAttemptNum: Int,
+      attemptId: Int,
+      isShuffleWriter: Boolean): Unit = {
+
+    val newShuffleId = appShuffleId * maxAttemptNum + attemptId
+    val shuffleIds = shuffleIdMapping.computeIfAbsent(appShuffleId, new function.Function[Int, Seq[Int]]() {
+      override def apply(shuffleId: Int): Seq[Int] = Seq.apply(newShuffleId)
+    })
+
+    if (isShuffleWriter && !shuffleIds.contains(newShuffleId)) {
+      logInfo(s"[handleGetShuffleId]" +
+        s"record new shuffleId $newShuffleId with appShuffleId $appShuffleId attemptId $attemptId")
+      shuffleIdMapping.put(appShuffleId, shuffleIds :+ newShuffleId)
+    }
+
+    val shuffleId = shuffleIdMapping.get(appShuffleId).last

Review Comment:
   For Shuffle unregister/clean uselessShuffleId 
   1. When register a new shuffleId with same appShuffleId we can clear all out-dated info in lifecycleManager, such as shuffleIdMapping(appShuffleId->shuffleId), shuffleId related info...
   2. When shuffleId unregister, LifecycleManager alse can clear the useless shuffleId in shuffleIdMapping.



##########
client/src/main/scala/org/apache/celeborn/client/LifecycleManager.scala:
##########
@@ -556,6 +562,28 @@ class LifecycleManager(val appUniqueId: String, val conf: CelebornConf) extends
     commitManager.handleGetReducerFileGroup(context, shuffleId)
   }
 
+  private def handleGetShuffleId(

Review Comment:
   Another option:
   1. For writer, Add new field appShuffleId to registerShuffleRequest and generate uniqueShuffleId(used as celeborn shuffleId) at client side(spark), then lifecycleManager doesn't have to compute shuffleId according to spark stage rerun-logic, and writer needn't introduce a new rpc, btw shuffleIdMapping still need in lifecyceManager.
   2. For Reader, shuffleClient can directly get latest shuffleId related, this way we don't need the getShuffleId rpc, but we need change the shuffleId meaning in GetReducerFileGroup to appShuffleId, and response latest shuffle id in GetReducerFileGroupResponse @ErikFang @waitinfuture



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org