You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/02/25 07:29:41 UTC

[GitHub] [spark] seayoun opened a new pull request #31643: 【SPARK-34534】fix blockIds order when use FetchShuffleBlocks to fetch blocks

seayoun opened a new pull request #31643:
URL: https://github.com/apache/spark/pull/31643


   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
     2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
     7. If you want to add a new configuration, please read the guideline first for naming configurations in
        'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   Fix a problems which can lead to data correctness after part blocks retry in `OneForOneBlockFetcher` when use `FetchShuffleBlocks` .
   
   
   ### Why are the changes needed?
   This is a data correctness bug, It's is no problems when use old protocol to send `OpenBlocks` before fetch chunks in `OneForOneBlockFetcher`; 
   In latest branch, `OpenBlocks`  has been replaced to `FetchShuffleBlocks`. Howerver, `FetchShuffleBlocks` read shuffle blocks order is not the same as `blockIds` in `OneForOneBlockFetcher; the `blockIds` is used to match blockId with shuffle data with index, now it is out of order;
   It will lead to read wrong block chunk when some blocks fetch failed in `OneForOneBlockFetcher`, it will retry the rest of the blocks in `blockIds`  based on the `blockIds`'s order.
   
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   -->
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-788741414


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135646/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786529644


   **[Test build #135497 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135497/testReport)** for PR 31643 at commit [`006a5ec`](https://github.com/apache/spark/commit/006a5ec12a93ed641c3ef91d3d7adc6e6deb7553).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] otterc commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r583790528



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -117,17 +117,23 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     boolean batchFetchEnabled = firstBlock.length == 5;
 
     HashMap<Long, ArrayList<Integer>> mapIdToReduceIds = new HashMap<>();
+    ArrayList<Long> orderedMapId = new ArrayList<>();
     for (String blockId : blockIds) {
       String[] blockIdParts = splitBlockId(blockId);
       if (Integer.parseInt(blockIdParts[1]) != shuffleId) {
         throw new IllegalArgumentException("Expected shuffleId=" + shuffleId +
           ", got:" + blockId);
       }
       long mapId = Long.parseLong(blockIdParts[2]);
+      assert(orderedMapId.isEmpty() || mapId >= orderedMapId.get(orderedMapId.size() - 1));

Review comment:
       I actually don't think that the bug reveals that order in blockIds is important. Just in `OneForOneBlockFetcher`, it never expects the provided `blockIds` are in certain order. Also at this layer why should it expect that?
   It is given an array of blockIds which may be in any order and would fetch them. 
   IMO, the bug is just that the  member `blockIds` of OneForOneBlockFetcher is out of sync with the FetchShuffleBlocks.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-787631583






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] mridulm commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786008067


   Can we add a test to validate and ensure this does not happen in future ?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-788792970






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786507753


   **[Test build #135499 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135499/testReport)** for PR 31643 at commit [`a08ca1d`](https://github.com/apache/spark/commit/a08ca1da29103ce8b16bc81fc208992f4561fb1f).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786684965


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135509/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] seayoun commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
seayoun commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r583487717



##########
File path: common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/OneForOneBlockFetcherSuite.java
##########
@@ -201,6 +201,50 @@ public void testEmptyBlockFetch() {
     }
   }
 
+  @Test
+  public void testFetchShuffleBlocksOrder() {
+    LinkedHashMap<String, ManagedBuffer> blocks = Maps.newLinkedHashMap();
+    blocks.put("shuffle_0_0_0", new NioManagedBuffer(ByteBuffer.wrap(new byte[1])));
+    blocks.put("shuffle_0_2_1", new NioManagedBuffer(ByteBuffer.wrap(new byte[2])));
+    blocks.put("shuffle_0_10_2", new NettyManagedBuffer(Unpooled.wrappedBuffer(new byte[3])));
+    String[] blockIds = blocks.keySet().toArray(new String[blocks.size()]);
+
+    BlockFetchingListener listener = fetchBlocks(
+      blocks,
+      blockIds,
+      new FetchShuffleBlocks("app-id", "exec-id", 0, new long[]{0}, new int[][]{{0, 3}}, false),
+      conf);
+
+    for (int chunkIndex = 0; chunkIndex < blockIds.length; chunkIndex++) {
+      String blockId = blockIds[chunkIndex];
+      verify(listener).onBlockFetchSuccess(blockId, blocks.get(blockId));
+    }
+  }
+
+  @Test
+  public void testBatchFetchShuffleBlocksOrder() {
+    LinkedHashMap<String, ManagedBuffer> blocks = Maps.newLinkedHashMap();
+    blocks.put("shuffle_0_0_1_2", new NioManagedBuffer(ByteBuffer.wrap(new byte[1])));
+    blocks.put("shuffle_0_2_2_3", new NioManagedBuffer(ByteBuffer.wrap(new byte[2])));
+    blocks.put("shuffle_0_10_3_4", new NettyManagedBuffer(Unpooled.wrappedBuffer(new byte[3])));
+    String[] blockIds = blocks.keySet().toArray(new String[blocks.size()]);
+
+    BlockFetchingListener listener = fetchBlocks(
+      blocks,
+      blockIds,
+      new OpenBlocks("app-id", "exec-id", blockIds),

Review comment:
       > This doesn't follow the existing test `testBatchFetchThreeShuffleBlocks`, what's the reason?
   
   I will add test case for FetchShuffleBlocks using batch mode later




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] seayoun commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
seayoun commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r585372950



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -138,9 +138,24 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     }
     long[] mapIds = Longs.toArray(mapIdToReduceIds.keySet());
     int[][] reduceIdArr = new int[mapIds.length][];
+    int blockIdIndex = 0;
     for (int i = 0; i < mapIds.length; i++) {
       reduceIdArr[i] = Ints.toArray(mapIdToReduceIds.get(mapIds[i]));
+      // The `blockIds`'s order must be same with the read order specified in in FetchShuffleBlocks
+      // because the shuffle data's return order should match the `blockIds`'s order to ensure
+      // blockId and data match.
+      if (!batchFetchEnabled) {
+        for (int j = 0; j < reduceIdArr[i].length; j++) {
+          this.blockIds[blockIdIndex++] = "shuffle_" + shuffleId + "_" + mapIds[i] + "_"
+                  + reduceIdArr[i][j];

Review comment:
       We can consider the proposal from @cloud-fan . I think it maybe more flexible.
   (`LinkedHashMap` is used to ensure `mapIdToBlockIds.keySet()` and `mapIdToReduceIds.keySet()` keep same key order. I will use one map instead of two, so ignore this, haha)




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786414359


   **[Test build #135488 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135488/testReport)** for PR 31643 at commit [`384f43c`](https://github.com/apache/spark/commit/384f43c32bfc54b93035ad48b1ae8b297101aec7).
    * This patch **fails Java style tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Ngone51 commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r585353311



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -138,9 +138,24 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     }
     long[] mapIds = Longs.toArray(mapIdToReduceIds.keySet());
     int[][] reduceIdArr = new int[mapIds.length][];
+    int blockIdIndex = 0;
     for (int i = 0; i < mapIds.length; i++) {
       reduceIdArr[i] = Ints.toArray(mapIdToReduceIds.get(mapIds[i]));
+      // The `blockIds`'s order must be same with the read order specified in in FetchShuffleBlocks
+      // because the shuffle data's return order should match the `blockIds`'s order to ensure
+      // blockId and data match.
+      if (!batchFetchEnabled) {
+        for (int j = 0; j < reduceIdArr[i].length; j++) {
+          this.blockIds[blockIdIndex++] = "shuffle_" + shuffleId + "_" + mapIds[i] + "_"
+                  + reduceIdArr[i][j];

Review comment:
       Then, how about we carry the raw `blockId` with the splited block parts? For example, we put the raw `blockId` at index 0 (we can override the useless prefix "shuffle") to be compatible between batchfetch and non-batchfetch. 
   
   I think using one map instead of two would simplify the codes and be easier to maintain.
   
   BTW, I saw your latest updates, and I dont' think the `LinkedHashMap` helps anything in the current way(option1), while it does help in option2. 

##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -138,9 +138,24 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     }
     long[] mapIds = Longs.toArray(mapIdToReduceIds.keySet());
     int[][] reduceIdArr = new int[mapIds.length][];
+    int blockIdIndex = 0;
     for (int i = 0; i < mapIds.length; i++) {
       reduceIdArr[i] = Ints.toArray(mapIdToReduceIds.get(mapIds[i]));
+      // The `blockIds`'s order must be same with the read order specified in in FetchShuffleBlocks
+      // because the shuffle data's return order should match the `blockIds`'s order to ensure
+      // blockId and data match.
+      if (!batchFetchEnabled) {
+        for (int j = 0; j < reduceIdArr[i].length; j++) {
+          this.blockIds[blockIdIndex++] = "shuffle_" + shuffleId + "_" + mapIds[i] + "_"
+                  + reduceIdArr[i][j];

Review comment:
       Then, how about we carry the raw `blockId` with the splited block parts? For example, we can put the raw `blockId` at index 0 (we can override the useless prefix "shuffle") to be compatible between batchfetch and non-batchfetch. 
   
   I think using one map instead of two would simplify the codes and be easier to maintain.
   
   BTW, I saw your latest updates, and I dont' think the `LinkedHashMap` helps anything in the current way(option1), while it does help in option2. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] otterc commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
otterc commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-788474846


   +1 for Option 1


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] mridulm commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786007281


   +CC @otterc 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Ngone51 commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r585324316



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -138,9 +138,24 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     }
     long[] mapIds = Longs.toArray(mapIdToReduceIds.keySet());
     int[][] reduceIdArr = new int[mapIds.length][];
+    int blockIdIndex = 0;
     for (int i = 0; i < mapIds.length; i++) {
       reduceIdArr[i] = Ints.toArray(mapIdToReduceIds.get(mapIds[i]));
+      // The `blockIds`'s order must be same with the read order specified in in FetchShuffleBlocks
+      // because the shuffle data's return order should match the `blockIds`'s order to ensure
+      // blockId and data match.
+      if (!batchFetchEnabled) {
+        for (int j = 0; j < reduceIdArr[i].length; j++) {
+          this.blockIds[blockIdIndex++] = "shuffle_" + shuffleId + "_" + mapIds[i] + "_"
+                  + reduceIdArr[i][j];

Review comment:
       We don't need an extra `mapIdToBlockIds`. We should just replace `mapIdToReduceIds: HashMap<Long, ArrayList<Integer>>` to `mapIdToBlockIds: HashMap<Long, ArrayList<String>>`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31643: [SPARK-34534]fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-785789064


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40043/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] seayoun commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
seayoun commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r585322045



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -138,9 +138,24 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     }
     long[] mapIds = Longs.toArray(mapIdToReduceIds.keySet());
     int[][] reduceIdArr = new int[mapIds.length][];
+    int blockIdIndex = 0;
     for (int i = 0; i < mapIds.length; i++) {
       reduceIdArr[i] = Ints.toArray(mapIdToReduceIds.get(mapIds[i]));
+      // The `blockIds`'s order must be same with the read order specified in in FetchShuffleBlocks
+      // because the shuffle data's return order should match the `blockIds`'s order to ensure
+      // blockId and data match.
+      if (!batchFetchEnabled) {
+        for (int j = 0; j < reduceIdArr[i].length; j++) {
+          this.blockIds[blockIdIndex++] = "shuffle_" + shuffleId + "_" + mapIds[i] + "_"
+                  + reduceIdArr[i][j];

Review comment:
       Well, I also think reconstruct is not well, a new `mapIdToBlockIds` which has the same key order with `mapIdToReduceIds` is needed, what's your opinion?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786684151


   **[Test build #135509 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135509/testReport)** for PR 31643 at commit [`1234253`](https://github.com/apache/spark/commit/123425312f76b3364bfa74e09e0c9b4fec18a21a).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-788654098


   **[Test build #135623 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135623/testReport)** for PR 31643 at commit [`0748d20`](https://github.com/apache/spark/commit/0748d205ed03e005f7e6ad152483bb0a5fab522a).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-788774861






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xuanyuanking commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r583374620



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -117,17 +117,23 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     boolean batchFetchEnabled = firstBlock.length == 5;
 
     HashMap<Long, ArrayList<Integer>> mapIdToReduceIds = new HashMap<>();
+    ArrayList<Long> orderedMapId = new ArrayList<>();

Review comment:
       Please add more comments here to emphasize why we need ordered map id here.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-788558102


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40201/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] seayoun commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
seayoun commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r585340480



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -138,9 +138,24 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     }
     long[] mapIds = Longs.toArray(mapIdToReduceIds.keySet());
     int[][] reduceIdArr = new int[mapIds.length][];
+    int blockIdIndex = 0;
     for (int i = 0; i < mapIds.length; i++) {
       reduceIdArr[i] = Ints.toArray(mapIdToReduceIds.get(mapIds[i]));
+      // The `blockIds`'s order must be same with the read order specified in in FetchShuffleBlocks
+      // because the shuffle data's return order should match the `blockIds`'s order to ensure
+      // blockId and data match.
+      if (!batchFetchEnabled) {
+        for (int j = 0; j < reduceIdArr[i].length; j++) {
+          this.blockIds[blockIdIndex++] = "shuffle_" + shuffleId + "_" + mapIds[i] + "_"
+                  + reduceIdArr[i][j];

Review comment:
       I see. It seems to be feasible.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-787099380


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40125/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-788849640


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40233/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-787107627


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135544/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786560326


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40080/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786552221


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40080/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-788566576


   **[Test build #135623 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135623/testReport)** for PR 31643 at commit [`0748d20`](https://github.com/apache/spark/commit/0748d205ed03e005f7e6ad152483bb0a5fab522a).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-787604272


   **[Test build #135573 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135573/testReport)** for PR 31643 at commit [`e823930`](https://github.com/apache/spark/commit/e823930db1093e8e536b2117b62894160f7b2820).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786525903


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40078/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786443205


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40069/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786443205


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40069/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Ngone51 commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r584030032



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -117,17 +117,23 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     boolean batchFetchEnabled = firstBlock.length == 5;
 
     HashMap<Long, ArrayList<Integer>> mapIdToReduceIds = new HashMap<>();
+    ArrayList<Long> orderedMapId = new ArrayList<>();
     for (String blockId : blockIds) {
       String[] blockIdParts = splitBlockId(blockId);
       if (Integer.parseInt(blockIdParts[1]) != shuffleId) {
         throw new IllegalArgumentException("Expected shuffleId=" + shuffleId +
           ", got:" + blockId);
       }
       long mapId = Long.parseLong(blockIdParts[2]);
+      assert(orderedMapId.isEmpty() || mapId >= orderedMapId.get(orderedMapId.size() - 1));

Review comment:
       I see. Just feel a little bit uncomfortable about "rebuild"..




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] otterc commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r584028482



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -117,17 +117,23 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     boolean batchFetchEnabled = firstBlock.length == 5;
 
     HashMap<Long, ArrayList<Integer>> mapIdToReduceIds = new HashMap<>();
+    ArrayList<Long> orderedMapId = new ArrayList<>();
     for (String blockId : blockIds) {
       String[] blockIdParts = splitBlockId(blockId);
       if (Integer.parseInt(blockIdParts[1]) != shuffleId) {
         throw new IllegalArgumentException("Expected shuffleId=" + shuffleId +
           ", got:" + blockId);
       }
       long mapId = Long.parseLong(blockIdParts[2]);
+      assert(orderedMapId.isEmpty() || mapId >= orderedMapId.get(orderedMapId.size() - 1));

Review comment:
       The initial fix that was proposed was to create the member `blockIds` in OneForOneBlockFetcher in accordance with the FetchShuffleBlocks message which should work even if blockIds are random.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r583473213



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -117,30 +117,40 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     boolean batchFetchEnabled = firstBlock.length == 5;
 
     HashMap<Long, ArrayList<Integer>> mapIdToReduceIds = new HashMap<>();
+    // The mapIds and reducesId must be same order with blockIds because the shuffle data's
+    // return order will match the `blockIds` order to ensure blockId and data match.

Review comment:
       `will` -> `should`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-788743704


   **[Test build #135647 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135647/testReport)** for PR 31643 at commit [`c73c316`](https://github.com/apache/spark/commit/c73c31620b99a17555928f6615887ae9619b3c8a).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-789413203


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40268/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-788578764


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40203/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-787618959


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40154/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786615218


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40090/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786468172


   **[Test build #135497 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135497/testReport)** for PR 31643 at commit [`006a5ec`](https://github.com/apache/spark/commit/006a5ec12a93ed641c3ef91d3d7adc6e6deb7553).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-788743704


   **[Test build #135647 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135647/testReport)** for PR 31643 at commit [`c73c316`](https://github.com/apache/spark/commit/c73c31620b99a17555928f6615887ae9619b3c8a).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534][CORE] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-789442325


   **[Test build #135686 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135686/testReport)** for PR 31643 at commit [`9cf23df`](https://github.com/apache/spark/commit/9cf23dffd76d149b49c6681460a1fb91418af314).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] seayoun commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
seayoun commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r585400491



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -138,9 +138,24 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     }
     long[] mapIds = Longs.toArray(mapIdToReduceIds.keySet());
     int[][] reduceIdArr = new int[mapIds.length][];
+    int blockIdIndex = 0;
     for (int i = 0; i < mapIds.length; i++) {
       reduceIdArr[i] = Ints.toArray(mapIdToReduceIds.get(mapIds[i]));
+      // The `blockIds`'s order must be same with the read order specified in in FetchShuffleBlocks
+      // because the shuffle data's return order should match the `blockIds`'s order to ensure
+      // blockId and data match.
+      if (!batchFetchEnabled) {
+        for (int j = 0; j < reduceIdArr[i].length; j++) {
+          this.blockIds[blockIdIndex++] = "shuffle_" + shuffleId + "_" + mapIds[i] + "_"
+                  + reduceIdArr[i][j];

Review comment:
       Do you mean replace `mapIdToReduceIds` from `HashMap<Long, ArrayList<Integer>>` to `HashMap<Long, Pair<Integer, String>>`, add extral blockId in a pair?

##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -106,41 +107,53 @@ private boolean isShuffleBlocks(String[] blockIds) {
   }
 
   /**
-   * Analyze the pass in blockIds and create FetchShuffleBlocks message.
-   * The blockIds has been sorted by mapId and reduceId. It's produced in
-   * org.apache.spark.MapOutputTracker.convertMapStatuses.
+   * Create FetchShuffleBlocks message and rebuild internal blockIds by
+   * analyzing the pass in blockIds.
    */
-  private FetchShuffleBlocks createFetchShuffleBlocksMsg(
+  private FetchShuffleBlocks createFetchShuffleBlocksMsgAndBuildBlockIds(
       String appId, String execId, String[] blockIds) {
     String[] firstBlock = splitBlockId(blockIds[0]);
     int shuffleId = Integer.parseInt(firstBlock[1]);
     boolean batchFetchEnabled = firstBlock.length == 5;
 
-    HashMap<Long, ArrayList<Integer>> mapIdToReduceIds = new HashMap<>();
+    HashMap<Long, BlocksInfo> mapIdToBlocksInfo = new HashMap<>();

Review comment:
       Seems to be better.

##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -138,9 +138,24 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     }
     long[] mapIds = Longs.toArray(mapIdToReduceIds.keySet());
     int[][] reduceIdArr = new int[mapIds.length][];
+    int blockIdIndex = 0;
     for (int i = 0; i < mapIds.length; i++) {
       reduceIdArr[i] = Ints.toArray(mapIdToReduceIds.get(mapIds[i]));
+      // The `blockIds`'s order must be same with the read order specified in in FetchShuffleBlocks
+      // because the shuffle data's return order should match the `blockIds`'s order to ensure
+      // blockId and data match.
+      if (!batchFetchEnabled) {
+        for (int j = 0; j < reduceIdArr[i].length; j++) {
+          this.blockIds[blockIdIndex++] = "shuffle_" + shuffleId + "_" + mapIds[i] + "_"
+                  + reduceIdArr[i][j];

Review comment:
       I understand your design and it's a smart design, I think it maybe a little complex for reading; single class may be more easier to read.
   What's your options?

##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -81,7 +81,6 @@ public OneForOneBlockFetcher(
       TransportConf transportConf,
       DownloadFileManager downloadFileManager) {
     this.client = client;
-    this.blockIds = blockIds;

Review comment:
       The blockIds will be used to create OpenBlocks or ShuffleFetchBlocks later in contructor.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r583579332



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -117,30 +117,40 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     boolean batchFetchEnabled = firstBlock.length == 5;
 
     HashMap<Long, ArrayList<Integer>> mapIdToReduceIds = new HashMap<>();
+    // The mapIds and reducesIds must be same order with blockIds because the shuffle data's

Review comment:
       `reducesIds` -> `reduceIds `




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-788575882


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40203/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31643: [SPARK-34534][CORE] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-789426934


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40268/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Ngone51 commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r585353311



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -138,9 +138,24 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     }
     long[] mapIds = Longs.toArray(mapIdToReduceIds.keySet());
     int[][] reduceIdArr = new int[mapIds.length][];
+    int blockIdIndex = 0;
     for (int i = 0; i < mapIds.length; i++) {
       reduceIdArr[i] = Ints.toArray(mapIdToReduceIds.get(mapIds[i]));
+      // The `blockIds`'s order must be same with the read order specified in in FetchShuffleBlocks
+      // because the shuffle data's return order should match the `blockIds`'s order to ensure
+      // blockId and data match.
+      if (!batchFetchEnabled) {
+        for (int j = 0; j < reduceIdArr[i].length; j++) {
+          this.blockIds[blockIdIndex++] = "shuffle_" + shuffleId + "_" + mapIds[i] + "_"
+                  + reduceIdArr[i][j];

Review comment:
       Then, how about we carry the raw `blockId` with the splited block parts? For example, using a String array and always put the raw `blockId` at index 0 to be compatible between batchfetch and non-batchfetch. 
   
   BTW, I saw your latest updates, and I dont' think the `LinkedHashMap` helps anything in the current way(option1), while it does help in option2. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786515835


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40078/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-787107627


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135544/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Ngone51 commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r585380050



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -138,9 +138,24 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     }
     long[] mapIds = Longs.toArray(mapIdToReduceIds.keySet());
     int[][] reduceIdArr = new int[mapIds.length][];
+    int blockIdIndex = 0;
     for (int i = 0; i < mapIds.length; i++) {
       reduceIdArr[i] = Ints.toArray(mapIdToReduceIds.get(mapIds[i]));
+      // The `blockIds`'s order must be same with the read order specified in in FetchShuffleBlocks
+      // because the shuffle data's return order should match the `blockIds`'s order to ensure
+      // blockId and data match.
+      if (!batchFetchEnabled) {
+        for (int j = 0; j < reduceIdArr[i].length; j++) {
+          this.blockIds[blockIdIndex++] = "shuffle_" + shuffleId + "_" + mapIds[i] + "_"
+                  + reduceIdArr[i][j];

Review comment:
       I think my way is similar to @cloud-fan's in general, where both two ways suggest using one map by storing the raw `blockId` together with splited block id parts(the difference is whether it's parsed).
   
   
   Besides, my way avoid introducing the new class `BlocksInfo` by reusing the `blockIdParts` which is more efficient and simpler. And I don't see the flexible requirements for this function now,  we can add `BlocksInfo` whenever we needs it.
   
   @cloud-fan WDYT?

##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -138,9 +138,24 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     }
     long[] mapIds = Longs.toArray(mapIdToReduceIds.keySet());
     int[][] reduceIdArr = new int[mapIds.length][];
+    int blockIdIndex = 0;
     for (int i = 0; i < mapIds.length; i++) {
       reduceIdArr[i] = Ints.toArray(mapIdToReduceIds.get(mapIds[i]));
+      // The `blockIds`'s order must be same with the read order specified in in FetchShuffleBlocks
+      // because the shuffle data's return order should match the `blockIds`'s order to ensure
+      // blockId and data match.
+      if (!batchFetchEnabled) {
+        for (int j = 0; j < reduceIdArr[i].length; j++) {
+          this.blockIds[blockIdIndex++] = "shuffle_" + shuffleId + "_" + mapIds[i] + "_"
+                  + reduceIdArr[i][j];

Review comment:
       I think my way is similar to @cloud-fan's in general, where both two ways suggest using one map by storing the raw `blockId` together with splited block id parts(parsed or not).
   
   
   Besides, my way avoid introducing the new class `BlocksInfo` by reusing the `blockIdParts` which is more efficient and simpler. And I don't see the flexible requirements for this function now,  we can add `BlocksInfo` whenever we needs it.
   
   @cloud-fan WDYT?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] otterc commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r583790528



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -117,17 +117,23 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     boolean batchFetchEnabled = firstBlock.length == 5;
 
     HashMap<Long, ArrayList<Integer>> mapIdToReduceIds = new HashMap<>();
+    ArrayList<Long> orderedMapId = new ArrayList<>();
     for (String blockId : blockIds) {
       String[] blockIdParts = splitBlockId(blockId);
       if (Integer.parseInt(blockIdParts[1]) != shuffleId) {
         throw new IllegalArgumentException("Expected shuffleId=" + shuffleId +
           ", got:" + blockId);
       }
       long mapId = Long.parseLong(blockIdParts[2]);
+      assert(orderedMapId.isEmpty() || mapId >= orderedMapId.get(orderedMapId.size() - 1));

Review comment:
       I actually don't think that the bug reveals that order in blockIds is important. Just in `OneForOneBlockFetcher`, it never expects the provided `blockIds` are in certain order. Also at this layer why should it care about the order of blockIds?
   It is given an array of blockIds which may be in any order and would fetch them. 
   IMO, the bug is just that the  member `blockIds` of OneForOneBlockFetcher is out of sync with the FetchShuffleBlocks.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786560326


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40080/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r583476111



##########
File path: common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/OneForOneBlockFetcherSuite.java
##########
@@ -201,6 +201,50 @@ public void testEmptyBlockFetch() {
     }
   }
 
+  @Test
+  public void testFetchShuffleBlocksOrder() {
+    LinkedHashMap<String, ManagedBuffer> blocks = Maps.newLinkedHashMap();
+    blocks.put("shuffle_0_0_0", new NioManagedBuffer(ByteBuffer.wrap(new byte[1])));
+    blocks.put("shuffle_0_2_1", new NioManagedBuffer(ByteBuffer.wrap(new byte[2])));
+    blocks.put("shuffle_0_10_2", new NettyManagedBuffer(Unpooled.wrappedBuffer(new byte[3])));
+    String[] blockIds = blocks.keySet().toArray(new String[blocks.size()]);
+
+    BlockFetchingListener listener = fetchBlocks(
+      blocks,
+      blockIds,
+      new FetchShuffleBlocks("app-id", "exec-id", 0, new long[]{0}, new int[][]{{0, 3}}, false),
+      conf);
+
+    for (int chunkIndex = 0; chunkIndex < blockIds.length; chunkIndex++) {
+      String blockId = blockIds[chunkIndex];
+      verify(listener).onBlockFetchSuccess(blockId, blocks.get(blockId));
+    }
+  }
+
+  @Test
+  public void testBatchFetchShuffleBlocksOrder() {
+    LinkedHashMap<String, ManagedBuffer> blocks = Maps.newLinkedHashMap();
+    blocks.put("shuffle_0_0_1_2", new NioManagedBuffer(ByteBuffer.wrap(new byte[1])));
+    blocks.put("shuffle_0_2_2_3", new NioManagedBuffer(ByteBuffer.wrap(new byte[2])));
+    blocks.put("shuffle_0_10_3_4", new NettyManagedBuffer(Unpooled.wrappedBuffer(new byte[3])));
+    String[] blockIds = blocks.keySet().toArray(new String[blocks.size()]);
+
+    BlockFetchingListener listener = fetchBlocks(
+      blocks,
+      blockIds,
+      new OpenBlocks("app-id", "exec-id", blockIds),

Review comment:
       This doesn't follow the existing test `testBatchFetchThreeShuffleBlocks`, what's the reason?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Ngone51 commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r584030032



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -117,17 +117,23 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     boolean batchFetchEnabled = firstBlock.length == 5;
 
     HashMap<Long, ArrayList<Integer>> mapIdToReduceIds = new HashMap<>();
+    ArrayList<Long> orderedMapId = new ArrayList<>();
     for (String blockId : blockIds) {
       String[] blockIdParts = splitBlockId(blockId);
       if (Integer.parseInt(blockIdParts[1]) != shuffleId) {
         throw new IllegalArgumentException("Expected shuffleId=" + shuffleId +
           ", got:" + blockId);
       }
       long mapId = Long.parseLong(blockIdParts[2]);
+      assert(orderedMapId.isEmpty() || mapId >= orderedMapId.get(orderedMapId.size() - 1));

Review comment:
       I see. that way looks feasible.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-788920801


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135653/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] otterc commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r583802260



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -136,11 +142,12 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
         mapIdToReduceIds.get(mapId).add(Integer.parseInt(blockIdParts[4]));
       }
     }
-    long[] mapIds = Longs.toArray(mapIdToReduceIds.keySet());
+    long[] mapIds = Longs.toArray(orderedMapId);

Review comment:
       ok. In that case, we could recompute the member `blockIds` to be in sync with `FetchShuffleBlocks`. I think that was the initial fix.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] seayoun commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
seayoun commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786391637


   > @seayoun Can this be fixed by using a LinkedHashMap for `mapIdToReduceIds` in `createFetchShuffleBlocksMsg`?
   > https://github.com/apache/spark/blob/f7ac2d655c756100c33e652402cefc507d2493b7/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java#L119
   > 
   > LinkedHashMap will return the keys in insertion order and the values are already ordered.
   > Or is the problem something different?
   
   Yes, and we shoud add an assert when blockIds parsed, so I use a arrayList to build mapIds and to check mapId order in blockIds. cc @cloud-fan 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Ngone51 commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r584019467



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -117,30 +117,40 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     boolean batchFetchEnabled = firstBlock.length == 5;
 
     HashMap<Long, ArrayList<Integer>> mapIdToReduceIds = new HashMap<>();
+    // The mapIds and reducesIds must be same order with blockIds because the shuffle data's
+    // return order should match the `blockIds` order to ensure blockId and data match.
+    ArrayList<Long> orderedMapId = new ArrayList<>();
     for (String blockId : blockIds) {
       String[] blockIdParts = splitBlockId(blockId);
       if (Integer.parseInt(blockIdParts[1]) != shuffleId) {
         throw new IllegalArgumentException("Expected shuffleId=" + shuffleId +
           ", got:" + blockId);
       }
       long mapId = Long.parseLong(blockIdParts[2]);
+      assert(orderedMapId.isEmpty() || mapId >= orderedMapId.get(orderedMapId.size() - 1));
       if (!mapIdToReduceIds.containsKey(mapId)) {
         mapIdToReduceIds.put(mapId, new ArrayList<>());
+        orderedMapId.add(mapId);
       }
-      mapIdToReduceIds.get(mapId).add(Integer.parseInt(blockIdParts[3]));
+      ArrayList<Integer> reduceIdsByMapId = mapIdToReduceIds.get(mapId);
+      int reduceId = Integer.parseInt(blockIdParts[3]);
+      assert(reduceIdsByMapId.isEmpty()
+              || reduceId > reduceIdsByMapId.get(reduceIdsByMapId.size() - 1));

Review comment:
       nit: 2 indentations




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Ngone51 commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786986201


   LGTM


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534]fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-785784827


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40043/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-785853407


   **[Test build #135463 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135463/testReport)** for PR 31643 at commit [`0af3b01`](https://github.com/apache/spark/commit/0af3b01708a66bd5bf84c1b665841f10c6d69788).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-787631582






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-788792970


   **[Test build #135653 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135653/testReport)** for PR 31643 at commit [`6377217`](https://github.com/apache/spark/commit/6377217b1efbbbce0f40232d40eae66112c4fc4b).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Ngone51 commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r584025854



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -117,17 +117,23 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     boolean batchFetchEnabled = firstBlock.length == 5;
 
     HashMap<Long, ArrayList<Integer>> mapIdToReduceIds = new HashMap<>();
+    ArrayList<Long> orderedMapId = new ArrayList<>();
     for (String blockId : blockIds) {
       String[] blockIdParts = splitBlockId(blockId);
       if (Integer.parseInt(blockIdParts[1]) != shuffleId) {
         throw new IllegalArgumentException("Expected shuffleId=" + shuffleId +
           ", got:" + blockId);
       }
       long mapId = Long.parseLong(blockIdParts[2]);
+      assert(orderedMapId.isEmpty() || mapId >= orderedMapId.get(orderedMapId.size() - 1));

Review comment:
       Ah, that's a good point. I agree the assertion is not necessary. But if we assume the random `blockIds` here, the current fix won't work with duplicate `mapIds`. (Because we should be able to save duplicate `mapIds` in `FetchShuffleBlocks` to ensure the server side serves chunks in the same order with `blockIds` )
   
   But since the `blockIds` is always ordered in reality, I think the current fix should be fine.
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786421087


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40069/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-787096518


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40125/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-789394133


   **[Test build #135686 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135686/testReport)** for PR 31643 at commit [`9cf23df`](https://github.com/apache/spark/commit/9cf23dffd76d149b49c6681460a1fb91418af314).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan closed pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #31643:
URL: https://github.com/apache/spark/pull/31643


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] seayoun commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
seayoun commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r585348178



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -138,9 +138,24 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     }
     long[] mapIds = Longs.toArray(mapIdToReduceIds.keySet());
     int[][] reduceIdArr = new int[mapIds.length][];
+    int blockIdIndex = 0;
     for (int i = 0; i < mapIds.length; i++) {
       reduceIdArr[i] = Ints.toArray(mapIdToReduceIds.get(mapIds[i]));
+      // The `blockIds`'s order must be same with the read order specified in in FetchShuffleBlocks
+      // because the shuffle data's return order should match the `blockIds`'s order to ensure
+      // blockId and data match.
+      if (!batchFetchEnabled) {
+        for (int j = 0; j < reduceIdArr[i].length; j++) {
+          this.blockIds[blockIdIndex++] = "shuffle_" + shuffleId + "_" + mapIds[i] + "_"
+                  + reduceIdArr[i][j];

Review comment:
       We should splits blockId to fill `mapIdToBlockIds` and then split blockId again to fill the `reduceIdArr` in `ShuffleFetchBlocks`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786534218


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40080/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] otterc commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r583382485



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -117,17 +117,23 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     boolean batchFetchEnabled = firstBlock.length == 5;
 
     HashMap<Long, ArrayList<Integer>> mapIdToReduceIds = new HashMap<>();
+    ArrayList<Long> orderedMapId = new ArrayList<>();
     for (String blockId : blockIds) {
       String[] blockIdParts = splitBlockId(blockId);
       if (Integer.parseInt(blockIdParts[1]) != shuffleId) {
         throw new IllegalArgumentException("Expected shuffleId=" + shuffleId +
           ", got:" + blockId);
       }
       long mapId = Long.parseLong(blockIdParts[2]);
+      assert(orderedMapId.isEmpty() || mapId >= orderedMapId.get(orderedMapId.size() - 1));

Review comment:
       Why is this needed? `mapId >= orderedMapId.get(orderedMapId.size() - 1)` is validating that the `blockIds` are in increasing order of mapIds but this validation didn't exist earlier so why is this being done now in reference to this bug?

##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -117,17 +117,23 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     boolean batchFetchEnabled = firstBlock.length == 5;
 
     HashMap<Long, ArrayList<Integer>> mapIdToReduceIds = new HashMap<>();
+    ArrayList<Long> orderedMapId = new ArrayList<>();
     for (String blockId : blockIds) {
       String[] blockIdParts = splitBlockId(blockId);
       if (Integer.parseInt(blockIdParts[1]) != shuffleId) {
         throw new IllegalArgumentException("Expected shuffleId=" + shuffleId +
           ", got:" + blockId);
       }
       long mapId = Long.parseLong(blockIdParts[2]);
+      assert(orderedMapId.isEmpty() || mapId >= orderedMapId.get(orderedMapId.size() - 1));
       if (!mapIdToReduceIds.containsKey(mapId)) {
         mapIdToReduceIds.put(mapId, new ArrayList<>());
+        orderedMapId.add(mapId);
       }
-      mapIdToReduceIds.get(mapId).add(Integer.parseInt(blockIdParts[3]));
+      ArrayList<Integer> reduceIdsByMapId = mapIdToReduceIds.get(mapId);
+      int reduceId = Integer.parseInt(blockIdParts[3]);
+      assert(reduceIdsByMapId.isEmpty() || reduceId > reduceIdsByMapId.get(reduceIdsByMapId.size() - 1));

Review comment:
       Same here, why is this validation `reduceId > reduceIdsByMapId.get(reduceIdsByMapId.size() - 1))` needed now in reference to this bug?

##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -136,11 +142,12 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
         mapIdToReduceIds.get(mapId).add(Integer.parseInt(blockIdParts[4]));
       }
     }
-    long[] mapIds = Longs.toArray(mapIdToReduceIds.keySet());
+    long[] mapIds = Longs.toArray(orderedMapId);

Review comment:
       If we use a LinkedHashMap for `mapIdToReduceIds`, then `mapIdToReduceIds.keySet` will return the mapIds in the same order as they were inserted. So the order will be same as blockIds.

##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -117,17 +117,23 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     boolean batchFetchEnabled = firstBlock.length == 5;
 
     HashMap<Long, ArrayList<Integer>> mapIdToReduceIds = new HashMap<>();
+    ArrayList<Long> orderedMapId = new ArrayList<>();

Review comment:
       Why can't we just change `mapIdToReduceIds` to a LinkedHashMap?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r583371936



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -136,11 +142,12 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
         mapIdToReduceIds.get(mapId).add(Integer.parseInt(blockIdParts[4]));

Review comment:
       This line should be `reduceIdsByMapId.add...`, as well as the assert.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Ngone51 commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r585428610



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -138,9 +138,24 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     }
     long[] mapIds = Longs.toArray(mapIdToReduceIds.keySet());
     int[][] reduceIdArr = new int[mapIds.length][];
+    int blockIdIndex = 0;
     for (int i = 0; i < mapIds.length; i++) {
       reduceIdArr[i] = Ints.toArray(mapIdToReduceIds.get(mapIds[i]));
+      // The `blockIds`'s order must be same with the read order specified in in FetchShuffleBlocks
+      // because the shuffle data's return order should match the `blockIds`'s order to ensure
+      // blockId and data match.
+      if (!batchFetchEnabled) {
+        for (int j = 0; j < reduceIdArr[i].length; j++) {
+          this.blockIds[blockIdIndex++] = "shuffle_" + shuffleId + "_" + mapIds[i] + "_"
+                  + reduceIdArr[i][j];

Review comment:
       @seayoun @cloud-fan  I mean: https://github.com/Ngone51/spark/pull/7/files
    




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-787101514


   **[Test build #135544 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135544/testReport)** for PR 31643 at commit [`7987a25`](https://github.com/apache/spark/commit/7987a25becc5321963b5c8f982ad8a5d5628f0a7).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] seayoun commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
seayoun commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-788560144


   Revert initial fix @cloud-fan @otterc @Ngone51 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31643: [SPARK-34534][CORE] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-789453951


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135686/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534]fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-785781756


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40043/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-788566576


   **[Test build #135623 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135623/testReport)** for PR 31643 at commit [`0748d20`](https://github.com/apache/spark/commit/0748d205ed03e005f7e6ad152483bb0a5fab522a).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #31643: [SPARK-34534][CORE] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-789394133


   **[Test build #135686 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135686/testReport)** for PR 31643 at commit [`9cf23df`](https://github.com/apache/spark/commit/9cf23dffd76d149b49c6681460a1fb91418af314).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-788558128


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40201/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] otterc commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
otterc commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786297319


   @seayoun Can this be fixed by using a LinkedHashMap for `mapIdToReduceIds` in `createFetchShuffleBlocksMsg`?
   https://github.com/apache/spark/blob/f7ac2d655c756100c33e652402cefc507d2493b7/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java#L119
   
   LinkedHashMap will return the keys in insertion order and the values are already ordered. 
   Or is the problem is something different?
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-788741414


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135646/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-788774860






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786430908


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40069/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] seayoun commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
seayoun commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r584423100



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -117,17 +117,23 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     boolean batchFetchEnabled = firstBlock.length == 5;
 
     HashMap<Long, ArrayList<Integer>> mapIdToReduceIds = new HashMap<>();
+    ArrayList<Long> orderedMapId = new ArrayList<>();
     for (String blockId : blockIds) {
       String[] blockIdParts = splitBlockId(blockId);
       if (Integer.parseInt(blockIdParts[1]) != shuffleId) {
         throw new IllegalArgumentException("Expected shuffleId=" + shuffleId +
           ", got:" + blockId);
       }
       long mapId = Long.parseLong(blockIdParts[2]);
+      assert(orderedMapId.isEmpty() || mapId >= orderedMapId.get(orderedMapId.size() - 1));

Review comment:
       I think we should reduce randomness in different task attempt and try to ensure the consistence of different task attempt.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786507753


   **[Test build #135499 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135499/testReport)** for PR 31643 at commit [`a08ca1d`](https://github.com/apache/spark/commit/a08ca1da29103ce8b16bc81fc208992f4561fb1f).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786649447


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40090/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-785769589


   **[Test build #135463 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135463/testReport)** for PR 31643 at commit [`0af3b01`](https://github.com/apache/spark/commit/0af3b01708a66bd5bf84c1b665841f10c6d69788).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31643: [SPARK-34534]fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-785683561






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786591593


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135499/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-785859456


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135463/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Ngone51 commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r585254991



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -138,9 +138,24 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     }
     long[] mapIds = Longs.toArray(mapIdToReduceIds.keySet());
     int[][] reduceIdArr = new int[mapIds.length][];
+    int blockIdIndex = 0;
     for (int i = 0; i < mapIds.length; i++) {
       reduceIdArr[i] = Ints.toArray(mapIdToReduceIds.get(mapIds[i]));
+      // The `blockIds`'s order must be same with the read order specified in in FetchShuffleBlocks
+      // because the shuffle data's return order should match the `blockIds`'s order to ensure
+      // blockId and data match.
+      if (!batchFetchEnabled) {
+        for (int j = 0; j < reduceIdArr[i].length; j++) {
+          this.blockIds[blockIdIndex++] = "shuffle_" + shuffleId + "_" + mapIds[i] + "_"
+                  + reduceIdArr[i][j];

Review comment:
       How about we put `blockId` to the `mapIdToReduceIds` first and parse it later here? So we only reorder the `blockIds` and don't need to reconstruct it.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786684965


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135509/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Ngone51 commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r585528211



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -138,9 +138,24 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     }
     long[] mapIds = Longs.toArray(mapIdToReduceIds.keySet());
     int[][] reduceIdArr = new int[mapIds.length][];
+    int blockIdIndex = 0;
     for (int i = 0; i < mapIds.length; i++) {
       reduceIdArr[i] = Ints.toArray(mapIdToReduceIds.get(mapIds[i]));
+      // The `blockIds`'s order must be same with the read order specified in in FetchShuffleBlocks
+      // because the shuffle data's return order should match the `blockIds`'s order to ensure
+      // blockId and data match.
+      if (!batchFetchEnabled) {
+        for (int j = 0; j < reduceIdArr[i].length; j++) {
+          this.blockIds[blockIdIndex++] = "shuffle_" + shuffleId + "_" + mapIds[i] + "_"
+                  + reduceIdArr[i][j];

Review comment:
       I think our original motivation was to avoid the reconstruction of `blockId`, which avoids the intermediate object as well. And now we have avoided the reconstruction but introduced another new intermediate object - `BlocksInfo`. Does it become more efficient or more readable?
   
   As for you think https://github.com/Ngone51/spark/pull/7/files is less readable...I personally don't really think it's hard to read..perhaps adding more comments should help.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] seayoun commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
seayoun commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r585340480



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -138,9 +138,24 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     }
     long[] mapIds = Longs.toArray(mapIdToReduceIds.keySet());
     int[][] reduceIdArr = new int[mapIds.length][];
+    int blockIdIndex = 0;
     for (int i = 0; i < mapIds.length; i++) {
       reduceIdArr[i] = Ints.toArray(mapIdToReduceIds.get(mapIds[i]));
+      // The `blockIds`'s order must be same with the read order specified in in FetchShuffleBlocks
+      // because the shuffle data's return order should match the `blockIds`'s order to ensure
+      // blockId and data match.
+      if (!batchFetchEnabled) {
+        for (int j = 0; j < reduceIdArr[i].length; j++) {
+          this.blockIds[blockIdIndex++] = "shuffle_" + shuffleId + "_" + mapIds[i] + "_"
+                  + reduceIdArr[i][j];

Review comment:
       the block id will be parse twice if do this.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r585383459



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -157,6 +170,18 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     return blockIdParts;
   }
 
+  /** The reduceIds and blocks in a single mapId */
+  private class BlocksInfo {
+
+    ArrayList<Integer> reduceIds;
+    ArrayList<String> blockIds;
+
+    public BlocksInfo(ArrayList<Integer> reduceIds, ArrayList<String> blockIds) {

Review comment:
       nit
   ```
   public BlocksInfo() {
     this.reduceIds = new ArrayList<>();
     this.blockIds = new ArrayList<>();
   }
   ```

##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -157,6 +170,18 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     return blockIdParts;
   }
 
+  /** The reduceIds and blocks in a single mapId */
+  private class BlocksInfo {
+
+    ArrayList<Integer> reduceIds;

Review comment:
       nit: final

##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -106,41 +107,53 @@ private boolean isShuffleBlocks(String[] blockIds) {
   }
 
   /**
-   * Analyze the pass in blockIds and create FetchShuffleBlocks message.
-   * The blockIds has been sorted by mapId and reduceId. It's produced in
-   * org.apache.spark.MapOutputTracker.convertMapStatuses.
+   * Create FetchShuffleBlocks message and rebuild internal blockIds by
+   * analyzing the pass in blockIds.
    */
-  private FetchShuffleBlocks createFetchShuffleBlocksMsg(
+  private FetchShuffleBlocks createFetchShuffleBlocksMsgAndBuildBlockIds(
       String appId, String execId, String[] blockIds) {
     String[] firstBlock = splitBlockId(blockIds[0]);
     int shuffleId = Integer.parseInt(firstBlock[1]);
     boolean batchFetchEnabled = firstBlock.length == 5;
 
-    HashMap<Long, ArrayList<Integer>> mapIdToReduceIds = new HashMap<>();
+    HashMap<Long, BlocksInfo> mapIdToBlocksInfo = new HashMap<>();

Review comment:
       shall we use linked hash map to eliminate the randomness of map id array? It's not a hard requirement but it seems better if the block ids are the same with and without the new shuffle protocol.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-787099380


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40125/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-788556790


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40201/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786468172


   **[Test build #135497 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135497/testReport)** for PR 31643 at commit [`006a5ec`](https://github.com/apache/spark/commit/006a5ec12a93ed641c3ef91d3d7adc6e6deb7553).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r585480726



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -81,7 +81,6 @@ public OneForOneBlockFetcher(
       TransportConf transportConf,
       DownloadFileManager downloadFileManager) {
     this.client = client;
-    this.blockIds = blockIds;

Review comment:
       Do we need to remove this from the constructor, line 78 and line 69?

##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -81,7 +81,6 @@ public OneForOneBlockFetcher(
       TransportConf transportConf,
       DownloadFileManager downloadFileManager) {
     this.client = client;
-    this.blockIds = blockIds;

Review comment:
       Or, we cannot change it because it's a public class?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-787701759


   Seems like there are two proposals:
   1. regenerate the block ids when building the `FetchShuffleBlocks` request.
   2. leverage the fact the block ids are ordered already.
   
   I think the goal is the same: keep block ids and the fetch request in sync. And these 2 proposals have no actual difference if the block ids are indeed ordered (you will regenerate the same ordered block ids).
   
   I'm in favor of 2 because it ensures the things specified in the doc of `createFetchShuffleBlocksMsg`. I think it's better to not only keep block ids and the fetch request in sync, but also ensure the block ids are ordered, to eliminate randomness.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786399711


   We can add a UT in `OneForOneBlockFetcherSuite`. It doesn't have to test the retry behavior, but just test that the block id matches its data.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31643: [SPARK-34534][CORE] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-789453951


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135686/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786421441


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135488/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-788580884


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40203/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-788558128


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40201/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r583387151



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -136,11 +142,12 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
         mapIdToReduceIds.get(mapId).add(Integer.parseInt(blockIdParts[4]));
       }
     }
-    long[] mapIds = Longs.toArray(mapIdToReduceIds.keySet());
+    long[] mapIds = Longs.toArray(orderedMapId);

Review comment:
       This assumes the block ids are ordered by map id and reduce id. If the block ids are `shuffle_1_1_1, shuffle_1_2_1, shuffle_1_1_2`, then I don't think `FetchShuffleBlocks` can retain the order anymore.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] otterc commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r584021590



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -117,17 +117,23 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     boolean batchFetchEnabled = firstBlock.length == 5;
 
     HashMap<Long, ArrayList<Integer>> mapIdToReduceIds = new HashMap<>();
+    ArrayList<Long> orderedMapId = new ArrayList<>();
     for (String blockId : blockIds) {
       String[] blockIdParts = splitBlockId(blockId);
       if (Integer.parseInt(blockIdParts[1]) != shuffleId) {
         throw new IllegalArgumentException("Expected shuffleId=" + shuffleId +
           ", got:" + blockId);
       }
       long mapId = Long.parseLong(blockIdParts[2]);
+      assert(orderedMapId.isEmpty() || mapId >= orderedMapId.get(orderedMapId.size() - 1));

Review comment:
       My point is that the order of blockIds has been irrelevant for `OneForOneBlockFetcher` so far. At this layer, it doesn't have any dependency on the order of block Ids so why to introduce that at this layer now?
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-787082172


   **[Test build #135544 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135544/testReport)** for PR 31643 at commit [`7987a25`](https://github.com/apache/spark/commit/7987a25becc5321963b5c8f982ad8a5d5628f0a7).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-787082172


   **[Test build #135544 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135544/testReport)** for PR 31643 at commit [`7987a25`](https://github.com/apache/spark/commit/7987a25becc5321963b5c8f982ad8a5d5628f0a7).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] seayoun commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
seayoun commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r585344490



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -138,9 +138,24 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     }
     long[] mapIds = Longs.toArray(mapIdToReduceIds.keySet());
     int[][] reduceIdArr = new int[mapIds.length][];
+    int blockIdIndex = 0;
     for (int i = 0; i < mapIds.length; i++) {
       reduceIdArr[i] = Ints.toArray(mapIdToReduceIds.get(mapIds[i]));
+      // The `blockIds`'s order must be same with the read order specified in in FetchShuffleBlocks
+      // because the shuffle data's return order should match the `blockIds`'s order to ensure
+      // blockId and data match.
+      if (!batchFetchEnabled) {
+        for (int j = 0; j < reduceIdArr[i].length; j++) {
+          this.blockIds[blockIdIndex++] = "shuffle_" + shuffleId + "_" + mapIds[i] + "_"
+                  + reduceIdArr[i][j];

Review comment:
       It will parse mapId at first, then fill the `mapIdToBlockIds`; 
   second, parse the reduceId and fill `ShuffleFetchBlocks` `reduceIdArr`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] otterc commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r583802260



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -136,11 +142,12 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
         mapIdToReduceIds.get(mapId).add(Integer.parseInt(blockIdParts[4]));
       }
     }
-    long[] mapIds = Longs.toArray(mapIdToReduceIds.keySet());
+    long[] mapIds = Longs.toArray(orderedMapId);

Review comment:
       ok. In that case, we could recompute the member `blockIds`  from the `FetchShuffleBlocks`. I think that was the initial fix.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] otterc commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r583791753



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -136,11 +142,12 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
         mapIdToReduceIds.get(mapId).add(Integer.parseInt(blockIdParts[4]));
       }
     }
-    long[] mapIds = Longs.toArray(mapIdToReduceIds.keySet());
+    long[] mapIds = Longs.toArray(orderedMapId);

Review comment:
       That's right so we can not just use LinkedHashMap and maintain a separate arrayList of mapIds.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786601391


   **[Test build #135509 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135509/testReport)** for PR 31643 at commit [`1234253`](https://github.com/apache/spark/commit/123425312f76b3364bfa74e09e0c9b4fec18a21a).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] otterc commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r583791753



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -136,11 +142,12 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
         mapIdToReduceIds.get(mapId).add(Integer.parseInt(blockIdParts[4]));
       }
     }
-    long[] mapIds = Longs.toArray(mapIdToReduceIds.keySet());
+    long[] mapIds = Longs.toArray(orderedMapId);

Review comment:
       That's right so we can not just use LinkedHashMap and maintain a separate arrayList of mapIds.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Ngone51 commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r585353311



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -138,9 +138,24 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     }
     long[] mapIds = Longs.toArray(mapIdToReduceIds.keySet());
     int[][] reduceIdArr = new int[mapIds.length][];
+    int blockIdIndex = 0;
     for (int i = 0; i < mapIds.length; i++) {
       reduceIdArr[i] = Ints.toArray(mapIdToReduceIds.get(mapIds[i]));
+      // The `blockIds`'s order must be same with the read order specified in in FetchShuffleBlocks
+      // because the shuffle data's return order should match the `blockIds`'s order to ensure
+      // blockId and data match.
+      if (!batchFetchEnabled) {
+        for (int j = 0; j < reduceIdArr[i].length; j++) {
+          this.blockIds[blockIdIndex++] = "shuffle_" + shuffleId + "_" + mapIds[i] + "_"
+                  + reduceIdArr[i][j];

Review comment:
       Then, how about we carry the raw `blockId` with the splited block parts? For example, using a String array and always put the raw `blockId` at index 0 to be compatible between batchfetch and non-batchfetch. 
   
   I think using one map instead of two would simplify the codes and be easier to maintain.
   
   BTW, I saw your latest updates, and I dont' think the `LinkedHashMap` helps anything in the current way(option1), while it does help in option2. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786601391


   **[Test build #135509 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135509/testReport)** for PR 31643 at commit [`1234253`](https://github.com/apache/spark/commit/123425312f76b3364bfa74e09e0c9b4fec18a21a).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Ngone51 commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r585345760



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -138,9 +138,24 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     }
     long[] mapIds = Longs.toArray(mapIdToReduceIds.keySet());
     int[][] reduceIdArr = new int[mapIds.length][];
+    int blockIdIndex = 0;
     for (int i = 0; i < mapIds.length; i++) {
       reduceIdArr[i] = Ints.toArray(mapIdToReduceIds.get(mapIds[i]));
+      // The `blockIds`'s order must be same with the read order specified in in FetchShuffleBlocks
+      // because the shuffle data's return order should match the `blockIds`'s order to ensure
+      // blockId and data match.
+      if (!batchFetchEnabled) {
+        for (int j = 0; j < reduceIdArr[i].length; j++) {
+          this.blockIds[blockIdIndex++] = "shuffle_" + shuffleId + "_" + mapIds[i] + "_"
+                  + reduceIdArr[i][j];

Review comment:
       OK I see. You mean we need to retrieve mapId twice.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-788580884


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40203/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-787092276


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40125/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786649447


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40090/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-785859456


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135463/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786410718


   **[Test build #135488 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135488/testReport)** for PR 31643 at commit [`384f43c`](https://github.com/apache/spark/commit/384f43c32bfc54b93035ad48b1ae8b297101aec7).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786591593


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135499/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-788660562


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135623/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r583386370



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -117,17 +117,23 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     boolean batchFetchEnabled = firstBlock.length == 5;
 
     HashMap<Long, ArrayList<Integer>> mapIdToReduceIds = new HashMap<>();
+    ArrayList<Long> orderedMapId = new ArrayList<>();
     for (String blockId : blockIds) {
       String[] blockIdParts = splitBlockId(blockId);
       if (Integer.parseInt(blockIdParts[1]) != shuffleId) {
         throw new IllegalArgumentException("Expected shuffleId=" + shuffleId +
           ", got:" + blockId);
       }
       long mapId = Long.parseLong(blockIdParts[2]);
+      assert(orderedMapId.isEmpty() || mapId >= orderedMapId.get(orderedMapId.size() - 1));

Review comment:
       This bug reveals that the block ids order is very important. I think it makes sense to add asserts to guarantee it.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r583472798



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -117,30 +117,40 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     boolean batchFetchEnabled = firstBlock.length == 5;
 
     HashMap<Long, ArrayList<Integer>> mapIdToReduceIds = new HashMap<>();
+    // The mapIds and reducesId must be same order with blockIds because the shuffle data's

Review comment:
       nit: `reducesId` -> `reduceIds`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786496127


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40078/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Ngone51 commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r585353311



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -138,9 +138,24 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     }
     long[] mapIds = Longs.toArray(mapIdToReduceIds.keySet());
     int[][] reduceIdArr = new int[mapIds.length][];
+    int blockIdIndex = 0;
     for (int i = 0; i < mapIds.length; i++) {
       reduceIdArr[i] = Ints.toArray(mapIdToReduceIds.get(mapIds[i]));
+      // The `blockIds`'s order must be same with the read order specified in in FetchShuffleBlocks
+      // because the shuffle data's return order should match the `blockIds`'s order to ensure
+      // blockId and data match.
+      if (!batchFetchEnabled) {
+        for (int j = 0; j < reduceIdArr[i].length; j++) {
+          this.blockIds[blockIdIndex++] = "shuffle_" + shuffleId + "_" + mapIds[i] + "_"
+                  + reduceIdArr[i][j];

Review comment:
       Then, how about we carry the raw `blockId` with the splited block parts? For example, using a String array and always put the raw `blockId` at index 0 (we can override the useless prefix "shuffle") to be compatible between batchfetch and non-batchfetch. 
   
   I think using one map instead of two would simplify the codes and be easier to maintain.
   
   BTW, I saw your latest updates, and I dont' think the `LinkedHashMap` helps anything in the current way(option1), while it does help in option2. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Ngone51 commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r584025854



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -117,17 +117,23 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     boolean batchFetchEnabled = firstBlock.length == 5;
 
     HashMap<Long, ArrayList<Integer>> mapIdToReduceIds = new HashMap<>();
+    ArrayList<Long> orderedMapId = new ArrayList<>();
     for (String blockId : blockIds) {
       String[] blockIdParts = splitBlockId(blockId);
       if (Integer.parseInt(blockIdParts[1]) != shuffleId) {
         throw new IllegalArgumentException("Expected shuffleId=" + shuffleId +
           ", got:" + blockId);
       }
       long mapId = Long.parseLong(blockIdParts[2]);
+      assert(orderedMapId.isEmpty() || mapId >= orderedMapId.get(orderedMapId.size() - 1));

Review comment:
       Ah, that's a good point. I agree the assertion is not necessary. But if we assume the random `blockIds` here, the current fix won't work with duplicate `mapIds`. (Because we should be able to save duplicate `mapIds` in `FetchShuffleBlocks` to ensure the server side serves chunks in the same order with `blockIds` )
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Ngone51 commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r585344010



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -138,9 +138,24 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     }
     long[] mapIds = Longs.toArray(mapIdToReduceIds.keySet());
     int[][] reduceIdArr = new int[mapIds.length][];
+    int blockIdIndex = 0;
     for (int i = 0; i < mapIds.length; i++) {
       reduceIdArr[i] = Ints.toArray(mapIdToReduceIds.get(mapIds[i]));
+      // The `blockIds`'s order must be same with the read order specified in in FetchShuffleBlocks
+      // because the shuffle data's return order should match the `blockIds`'s order to ensure
+      // blockId and data match.
+      if (!batchFetchEnabled) {
+        for (int j = 0; j < reduceIdArr[i].length; j++) {
+          this.blockIds[blockIdIndex++] = "shuffle_" + shuffleId + "_" + mapIds[i] + "_"
+                  + reduceIdArr[i][j];

Review comment:
       No...we can move the parse step here.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-788893555


   **[Test build #135653 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135653/testReport)** for PR 31643 at commit [`6377217`](https://github.com/apache/spark/commit/6377217b1efbbbce0f40232d40eae66112c4fc4b).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534][CORE] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-789426914


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40268/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #31643: [SPARK-34534]fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-785730004






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-788920801


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135653/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786628331


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40090/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-789418130


   GA passed, merging to master/3.1/3.0!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] otterc commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r584024039



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -117,17 +117,23 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     boolean batchFetchEnabled = firstBlock.length == 5;
 
     HashMap<Long, ArrayList<Integer>> mapIdToReduceIds = new HashMap<>();
+    ArrayList<Long> orderedMapId = new ArrayList<>();
     for (String blockId : blockIds) {
       String[] blockIdParts = splitBlockId(blockId);
       if (Integer.parseInt(blockIdParts[1]) != shuffleId) {
         throw new IllegalArgumentException("Expected shuffleId=" + shuffleId +
           ", got:" + blockId);
       }
       long mapId = Long.parseLong(blockIdParts[2]);
+      assert(orderedMapId.isEmpty() || mapId >= orderedMapId.get(orderedMapId.size() - 1));

Review comment:
       > the order of blockIds could be random
   
   Well, this PR is adding the validation where it checks the mapIds in `blockIds` are in increasing order.  So the order of `blockIds` cannot be random. 
   `assert(orderedMapId.isEmpty() || mapId >= orderedMapId.get(orderedMapId.size() - 1));`
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] seayoun commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
seayoun commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r585484158



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -81,7 +81,6 @@ public OneForOneBlockFetcher(
       TransportConf transportConf,
       DownloadFileManager downloadFileManager) {
     this.client = client;
-    this.blockIds = blockIds;

Review comment:
       The blockIds will be used to create OpenBlocks or ShuffleFetchBlocks later in constructor.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Ngone51 commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r584023425



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -117,17 +117,23 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     boolean batchFetchEnabled = firstBlock.length == 5;
 
     HashMap<Long, ArrayList<Integer>> mapIdToReduceIds = new HashMap<>();
+    ArrayList<Long> orderedMapId = new ArrayList<>();
     for (String blockId : blockIds) {
       String[] blockIdParts = splitBlockId(blockId);
       if (Integer.parseInt(blockIdParts[1]) != shuffleId) {
         throw new IllegalArgumentException("Expected shuffleId=" + shuffleId +
           ", got:" + blockId);
       }
       long mapId = Long.parseLong(blockIdParts[2]);
+      assert(orderedMapId.isEmpty() || mapId >= orderedMapId.get(orderedMapId.size() - 1));

Review comment:
       So to be clarified, the order of `blockIds` could be random, but the `orderedMapId` should be in the same order with `blockIds`. And I think this PR fixes the latter.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Ngone51 commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r584018972



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -117,17 +117,23 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     boolean batchFetchEnabled = firstBlock.length == 5;
 
     HashMap<Long, ArrayList<Integer>> mapIdToReduceIds = new HashMap<>();
+    ArrayList<Long> orderedMapId = new ArrayList<>();
     for (String blockId : blockIds) {
       String[] blockIdParts = splitBlockId(blockId);
       if (Integer.parseInt(blockIdParts[1]) != shuffleId) {
         throw new IllegalArgumentException("Expected shuffleId=" + shuffleId +
           ", got:" + blockId);
       }
       long mapId = Long.parseLong(blockIdParts[2]);
+      assert(orderedMapId.isEmpty() || mapId >= orderedMapId.get(orderedMapId.size() - 1));

Review comment:
       > It is given an array of blockIds which may be in any order and would fetch them.
   
   I think it's ordered because of `MapOutputTracker.convertMapStatuses`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786530644


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135497/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r585359529



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -106,17 +107,17 @@ private boolean isShuffleBlocks(String[] blockIds) {
   }
 
   /**
-   * Analyze the pass in blockIds and create FetchShuffleBlocks message.
-   * The blockIds has been sorted by mapId and reduceId. It's produced in
-   * org.apache.spark.MapOutputTracker.convertMapStatuses.
+   * Create FetchShuffleBlocks message and rebuild internal blockIds by
+   * analyzing the pass in blockIds.
    */
-  private FetchShuffleBlocks createFetchShuffleBlocksMsg(
+  private FetchShuffleBlocks createFetchShuffleBlocksMsgAndBuildBlockIds(
       String appId, String execId, String[] blockIds) {
     String[] firstBlock = splitBlockId(blockIds[0]);
     int shuffleId = Integer.parseInt(firstBlock[1]);
     boolean batchFetchEnabled = firstBlock.length == 5;
 
-    HashMap<Long, ArrayList<Integer>> mapIdToReduceIds = new HashMap<>();
+    LinkedHashMap<Long, ArrayList<Integer>> mapIdToReduceIds = new LinkedHashMap<>();
+    LinkedHashMap<Long, ArrayList<String>> mapIdToBlockIds = new LinkedHashMap<>();

Review comment:
       Can we use a single map? we can create a `BlocksInfo` class here
   ```
   class BlocksInfo {
     ArrayList<Integer> reduceIds;
     ArrayList<String> blockIds;
   }
   
   LinkedHashMap<Long, BlocksInfo> mapIdToBlocksInfo = new LinkedHashMap<>();
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786421441


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135488/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] otterc edited a comment on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
otterc edited a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786297319


   @seayoun Can this be fixed by using a LinkedHashMap for `mapIdToReduceIds` in `createFetchShuffleBlocksMsg`?
   https://github.com/apache/spark/blob/f7ac2d655c756100c33e652402cefc507d2493b7/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java#L119
   
   LinkedHashMap will return the keys in insertion order and the values are already ordered. 
   Or is the problem something different?
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534]fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-785769589


   **[Test build #135463 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135463/testReport)** for PR 31643 at commit [`0af3b01`](https://github.com/apache/spark/commit/0af3b01708a66bd5bf84c1b665841f10c6d69788).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Ngone51 commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r584025854



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -117,17 +117,23 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     boolean batchFetchEnabled = firstBlock.length == 5;
 
     HashMap<Long, ArrayList<Integer>> mapIdToReduceIds = new HashMap<>();
+    ArrayList<Long> orderedMapId = new ArrayList<>();
     for (String blockId : blockIds) {
       String[] blockIdParts = splitBlockId(blockId);
       if (Integer.parseInt(blockIdParts[1]) != shuffleId) {
         throw new IllegalArgumentException("Expected shuffleId=" + shuffleId +
           ", got:" + blockId);
       }
       long mapId = Long.parseLong(blockIdParts[2]);
+      assert(orderedMapId.isEmpty() || mapId >= orderedMapId.get(orderedMapId.size() - 1));

Review comment:
       Ah, that's a good point. I agree the assertion is not necessary. But if we assume the random `blockIds` here, the current fix won't work with duplicate `mapIds`. (we should be able to save duplicate `mapIds` in `FetchShuffleBlocks` to ensure the server side serves chunks in the same order with `blockIds` )
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786570613


   **[Test build #135499 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135499/testReport)** for PR 31643 at commit [`a08ca1d`](https://github.com/apache/spark/commit/a08ca1da29103ce8b16bc81fc208992f4561fb1f).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786410718


   **[Test build #135488 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135488/testReport)** for PR 31643 at commit [`384f43c`](https://github.com/apache/spark/commit/384f43c32bfc54b93035ad48b1ae8b297101aec7).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786399333


   The change LGTM. Can we add a test?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786525903


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40078/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31643: 【SPARK-34534】fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-785683561


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-786530644


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135497/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31643: [SPARK-34534][CORE] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-789426934


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40268/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan edited a comment on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
cloud-fan edited a comment on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-787701759


   Seems like there are two proposals:
   1. regenerate the block ids when building the `FetchShuffleBlocks` request.
   2. leverage the fact the block ids are ordered already.
   
   I think the goal is the same: keep block ids and the fetch request in sync. And these 2 proposals have no actual difference if the block ids are indeed ordered (you will regenerate the same ordered block ids).
   
   I don't have a strong preference here. Option 1 seems more flexible, and we can remove the assumptions specified in the doc of `createFetchShuffleBlocksMsg`.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-788660562


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135623/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] seayoun commented on a change in pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
seayoun commented on a change in pull request #31643:
URL: https://github.com/apache/spark/pull/31643#discussion_r584406925



##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
##########
@@ -117,30 +117,40 @@ private FetchShuffleBlocks createFetchShuffleBlocksMsg(
     boolean batchFetchEnabled = firstBlock.length == 5;
 
     HashMap<Long, ArrayList<Integer>> mapIdToReduceIds = new HashMap<>();
+    // The mapIds and reducesIds must be same order with blockIds because the shuffle data's
+    // return order should match the `blockIds` order to ensure blockId and data match.
+    ArrayList<Long> orderedMapId = new ArrayList<>();
     for (String blockId : blockIds) {
       String[] blockIdParts = splitBlockId(blockId);
       if (Integer.parseInt(blockIdParts[1]) != shuffleId) {
         throw new IllegalArgumentException("Expected shuffleId=" + shuffleId +
           ", got:" + blockId);
       }
       long mapId = Long.parseLong(blockIdParts[2]);
+      assert(orderedMapId.isEmpty() || mapId >= orderedMapId.get(orderedMapId.size() - 1));
       if (!mapIdToReduceIds.containsKey(mapId)) {
         mapIdToReduceIds.put(mapId, new ArrayList<>());
+        orderedMapId.add(mapId);
       }
-      mapIdToReduceIds.get(mapId).add(Integer.parseInt(blockIdParts[3]));
+      ArrayList<Integer> reduceIdsByMapId = mapIdToReduceIds.get(mapId);
+      int reduceId = Integer.parseInt(blockIdParts[3]);
+      assert(reduceIdsByMapId.isEmpty()
+              || reduceId > reduceIdsByMapId.get(reduceIdsByMapId.size() - 1));

Review comment:
       done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-787628752


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40154/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31643:
URL: https://github.com/apache/spark/pull/31643#issuecomment-788755248






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org