You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "waitinfuture (via GitHub)" <gi...@apache.org> on 2023/08/24 04:40:32 UTC

[GitHub] [incubator-celeborn] waitinfuture commented on a diff in pull request #1550: [CELEBORN-627][FLINK] Support split partitions.

waitinfuture commented on code in PR #1550:
URL: https://github.com/apache/incubator-celeborn/pull/1550#discussion_r1303791287


##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/PushDataHandler.scala:
##########
@@ -909,13 +901,26 @@ class PushDataHandler extends BaseMessageHandler with Logging {
         callback,
         wrappedCallback)) return
 
+    // During worker shutdown, worker will return HARD_SPLIT for all existed partition.

Review Comment:
   Better to rename `handleRpcRequestCore` to `handleMapPartitionRpcRequestCore` since it's only used for MapPartition.



##########
client-flink/common/src/main/java/org/apache/celeborn/plugin/flink/RemoteBufferStreamReader.java:
##########
@@ -146,4 +149,10 @@ public void dataReceived(ReadData readData) {
         readData.getFlinkBuffer().readableBytes());
     dataListener.accept(readData.getFlinkBuffer());
   }
+
+  public void onStreamEnd(BufferStreamEnd streamEnd) {
+    long streamId = streamEnd.getStreamId();
+    logger.debug("Rss buffer stream reader get stream end for {}", streamId);

Review Comment:
   remove `Rss`



##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/Worker.scala:
##########
@@ -682,6 +691,44 @@ private[celeborn] class Worker(
 
   @VisibleForTesting
   def getPushFetchServerPort: (Int, Int) = (pushPort, fetchPort)
+
+  def isRegionFinish: Boolean = {

Review Comment:
   better to rename to `isRegionFinished`



##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/Worker.scala:
##########
@@ -596,8 +597,9 @@ private[celeborn] class Worker(
     var waitTimes = 0
 
     def waitTime: Long = waitTimes * interval
-
-    while (!partitionLocationInfo.isEmpty && waitTime < timeout) {
+    // for mappartition, it should wait that datas of a region are completely received.
+    val isRegionFinished = isRegionFinish

Review Comment:
   This line is unnecessary



##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/Worker.scala:
##########
@@ -596,8 +597,9 @@ private[celeborn] class Worker(
     var waitTimes = 0
 
     def waitTime: Long = waitTimes * interval
-
-    while (!partitionLocationInfo.isEmpty && waitTime < timeout) {
+    // for mappartition, it should wait that datas of a region are completely received.
+    val isRegionFinished = isRegionFinish
+    while (!partitionLocationInfo.isEmpty && !isRegionFinished && waitTime < timeout) {

Review Comment:
   Should call `isRegionFinish()` instead of the cached `isRegionFinished`



##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/storage/MapDataPartitionReader.java:
##########
@@ -436,7 +436,11 @@ public void release() {
     // we can safely release if reader reaches error or (read/send finished)
     synchronized (lock) {
       if (!isReleased) {
-        logger.debug("release reader for stream {}", this.streamId);
+        logger.debug("release reader for stream {}", streamId);
+        // tell client that this stream is finished.
+        if (readFinished) {
+          associatedChannel.writeAndFlush(new BufferStreamEnd(streamId));

Review Comment:
   Will this be called in time?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org