You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "FMX (via GitHub)" <gi...@apache.org> on 2023/02/10 02:18:14 UTC

[GitHub] [incubator-celeborn] FMX opened a new pull request, #1214: [CELEBORN-278] Refactor open stream to support buffer stream.

FMX opened a new pull request, #1214:
URL: https://github.com/apache/incubator-celeborn/pull/1214

   ### What changes were proposed in this pull request?
   Refactor open stream to support buffer stream.
   
   
   ### Why are the changes needed?
   
   
   
   ### Does this PR introduce _any_ user-facing change?
   
   
   
   ### How was this patch tested?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-celeborn] zhongqiangczq commented on a diff in pull request #1214: [CELEBORN-278] Add openStreamWithCredit RPC.

Posted by "zhongqiangczq (via GitHub)" <gi...@apache.org>.
zhongqiangczq commented on code in PR #1214:
URL: https://github.com/apache/incubator-celeborn/pull/1214#discussion_r1108061250


##########
common/src/main/java/org/apache/celeborn/common/network/protocol/OpenStreamWithCredit.java:
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.celeborn.common.network.protocol;
+
+import static org.apache.celeborn.common.network.protocol.Message.Type.OPEN_STREAM_WITH_CREDIT;
+
+import java.nio.charset.StandardCharsets;
+
+import io.netty.buffer.ByteBuf;
+
+/** Buffer stream used in Map partition scenario. */
+public final class OpenStreamWithCredit extends RequestMessage {
+  public final byte[] shuffleKey;
+  public final byte[] fileName;
+  public final int startIndex;

Review Comment:
   is it more understanding by replacing member startIndex/endIndex with startReduceIndex/endReduceIndex like OpenStream?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-celeborn] waitinfuture commented on a diff in pull request #1214: [CELEBORN-278] Refactor open stream to support buffer stream.

Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture commented on code in PR #1214:
URL: https://github.com/apache/incubator-celeborn/pull/1214#discussion_r1103981942


##########
common/src/main/java/org/apache/celeborn/common/network/protocol/OpenStream.java:
##########
@@ -24,25 +24,13 @@
 import io.netty.buffer.ByteBuf;
 
 /** Request to read a set of blocks. Returns {@link StreamHandle}. */
-public final class OpenStream extends RequestMessage {
+public class OpenStream extends RequestMessage {
   public byte[] shuffleKey;
   public byte[] fileName;
-  public int startMapIndex;
-  public int endMapIndex;
 
-  public OpenStream(String shuffleKey, String fileName, int startMapIndex, int endMapIndex) {
-    this(
-        shuffleKey.getBytes(StandardCharsets.UTF_8),
-        fileName.getBytes(StandardCharsets.UTF_8),
-        startMapIndex,
-        endMapIndex);
-  }
-
-  public OpenStream(byte[] shuffleKey, byte[] fileName, int startMapIndex, int endMapIndex) {
+  public OpenStream(byte[] shuffleKey, byte[] fileName) {

Review Comment:
   ```encodedLength``` should also be changed as startMapIndex and endMapIndex are removed



##########
common/src/main/java/org/apache/celeborn/common/network/protocol/OpenStream.java:
##########
@@ -24,25 +24,13 @@
 import io.netty.buffer.ByteBuf;
 
 /** Request to read a set of blocks. Returns {@link StreamHandle}. */
-public final class OpenStream extends RequestMessage {
+public class OpenStream extends RequestMessage {
   public byte[] shuffleKey;
   public byte[] fileName;
-  public int startMapIndex;
-  public int endMapIndex;
 
-  public OpenStream(String shuffleKey, String fileName, int startMapIndex, int endMapIndex) {
-    this(
-        shuffleKey.getBytes(StandardCharsets.UTF_8),
-        fileName.getBytes(StandardCharsets.UTF_8),
-        startMapIndex,
-        endMapIndex);
-  }
-
-  public OpenStream(byte[] shuffleKey, byte[] fileName, int startMapIndex, int endMapIndex) {
+  public OpenStream(byte[] shuffleKey, byte[] fileName) {

Review Comment:
   I think refactoring OpenStream will cause incompatibility issue. I think we can try renaming OpenStream to OpenChunkStream, and we should run test case with newer client and older worker to verify.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-celeborn] waitinfuture merged pull request #1214: [CELEBORN-278] Add openStreamWithCredit RPC.

Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture merged PR #1214:
URL: https://github.com/apache/incubator-celeborn/pull/1214


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-celeborn] RexXiong commented on a diff in pull request #1214: [CELEBORN-278] Add openStreamWithCredit RPC.

Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on code in PR #1214:
URL: https://github.com/apache/incubator-celeborn/pull/1214#discussion_r1106679208


##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/FetchHandler.scala:
##########
@@ -77,25 +78,34 @@ class FetchHandler(val conf: TransportConf) extends BaseMessageHandler with Logg
         rpcSource.updateMessageMetrics(r, 0)
         handleChunkFetchRequest(client, r)
       case r: RpcRequest =>
-        handleOpenStream(client, r)
+        handleOpenStreamLikeRpcs(client, r)

Review Comment:
   suggest not modify the function name.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-celeborn] FMX commented on a diff in pull request #1214: [CELEBORN-278] Add openStreamWithCredit RPC.

Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on code in PR #1214:
URL: https://github.com/apache/incubator-celeborn/pull/1214#discussion_r1106707998


##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/FetchHandler.scala:
##########
@@ -77,25 +78,34 @@ class FetchHandler(val conf: TransportConf) extends BaseMessageHandler with Logg
         rpcSource.updateMessageMetrics(r, 0)
         handleChunkFetchRequest(client, r)
       case r: RpcRequest =>
-        handleOpenStream(client, r)
+        handleOpenStreamLikeRpcs(client, r)

Review Comment:
   OK.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-celeborn] codecov[bot] commented on pull request #1214: [CELEBORN-278] Refactor open stream to support buffer stream.

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #1214:
URL: https://github.com/apache/incubator-celeborn/pull/1214#issuecomment-1425097799

   # [Codecov](https://codecov.io/gh/apache/incubator-celeborn/pull/1214?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#1214](https://codecov.io/gh/apache/incubator-celeborn/pull/1214?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d849fb6) into [main](https://codecov.io/gh/apache/incubator-celeborn/commit/adb6592d316b28fdf26cc0364779b64de96c9e6a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (adb6592) will **decrease** coverage by `0.08%`.
   > The diff coverage is `8.99%`.
   
   ```diff
   @@             Coverage Diff              @@
   ##               main    #1214      +/-   ##
   ============================================
   - Coverage     27.18%   27.10%   -0.08%     
     Complexity      806      806              
   ============================================
     Files           212      214       +2     
     Lines         17974    18037      +63     
     Branches       1964     1965       +1     
   ============================================
   + Hits           4885     4887       +2     
   - Misses        12766    12827      +61     
     Partials        323      323              
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-celeborn/pull/1214?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...pache/celeborn/client/read/DfsPartitionReader.java](https://codecov.io/gh/apache/incubator-celeborn/pull/1214?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y2xpZW50L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9jZWxlYm9ybi9jbGllbnQvcmVhZC9EZnNQYXJ0aXRpb25SZWFkZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...he/celeborn/client/read/WorkerPartitionReader.java](https://codecov.io/gh/apache/incubator-celeborn/pull/1214?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y2xpZW50L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9jZWxlYm9ybi9jbGllbnQvcmVhZC9Xb3JrZXJQYXJ0aXRpb25SZWFkZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...born/common/network/protocol/OpenBufferStream.java](https://codecov.io/gh/apache/incubator-celeborn/pull/1214?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9jZWxlYm9ybi9jb21tb24vbmV0d29yay9wcm90b2NvbC9PcGVuQnVmZmVyU3RyZWFtLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...eborn/common/network/protocol/OpenChunkStream.java](https://codecov.io/gh/apache/incubator-celeborn/pull/1214?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9jZWxlYm9ybi9jb21tb24vbmV0d29yay9wcm90b2NvbC9PcGVuQ2h1bmtTdHJlYW0uamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...e/celeborn/common/network/protocol/OpenStream.java](https://codecov.io/gh/apache/incubator-celeborn/pull/1214?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9jZWxlYm9ybi9jb21tb24vbmV0d29yay9wcm90b2NvbC9PcGVuU3RyZWFtLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...ache/celeborn/common/network/protocol/Message.java](https://codecov.io/gh/apache/incubator-celeborn/pull/1214?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9jZWxlYm9ybi9jb21tb24vbmV0d29yay9wcm90b2NvbC9NZXNzYWdlLmphdmE=) | `50.00% <42.86%> (-1.19%)` | :arrow_down: |
   | [.../celeborn/service/deploy/worker/FetchHandler.scala](https://codecov.io/gh/apache/incubator-celeborn/pull/1214?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-d29ya2VyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvY2VsZWJvcm4vc2VydmljZS9kZXBsb3kvd29ya2VyL0ZldGNoSGFuZGxlci5zY2FsYQ==) | `36.89% <71.43%> (-0.61%)` | :arrow_down: |
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org