[spark] branch master updated: [SPARK-38428][SHUFFLE] Check the FetchShuffleBlocks message only once to improve iteration in external shuffle service

[mr...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

mridulm80 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 8fabd5e  [SPARK-38428][SHUFFLE] Check the FetchShuffleBlocks message only once to improve iteration in external shuffle service
8fabd5e is described below

commit 8fabd5efe311347a4d1c441813dd0bd3148655df
Author: weixiuli <we...@jd.com>
AuthorDate: Tue Mar 8 12:59:34 2022 -0600

    [SPARK-38428][SHUFFLE] Check the FetchShuffleBlocks message only once to improve iteration in external shuffle service
    
    ### What changes were proposed in this pull request?
    
    Currently, the FetchShuffleBlocks is checked in each element of a ShuffleManagedBufferIterator, which is unnecessary and it only needs to be checked once in the ShuffleManagedBufferIterator constructor.
    
    ### Why are the changes needed?
    To improve performance.
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    Existing unittests.
    
    Closes #35743 from weixiuli/SPARK-38428-improve-iteration.
    
    Authored-by: weixiuli <we...@jd.com>
    Signed-off-by: Mridul Muralidharan <mridul<at>gmail.com>
---
 .../org/apache/spark/network/shuffle/ExternalBlockHandler.java    | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockHandler.java b/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockHandler.java
index 1e413f6..52bc0f9c 100644
--- a/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockHandler.java
+++ b/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockHandler.java
@@ -512,14 +512,14 @@ public class ExternalBlockHandler extends RpcHandler
       mapIds = msg.mapIds;
       reduceIds = msg.reduceIds;
       batchFetchEnabled = msg.batchFetchEnabled;
-    }
-
-    @Override
-    public boolean hasNext() {
       // mapIds.length must equal to reduceIds.length, and the passed in FetchShuffleBlocks
       // must have non-empty mapIds and reduceIds, see the checking logic in
       // OneForOneBlockFetcher.
       assert(mapIds.length != 0 && mapIds.length == reduceIds.length);
+    }
+
+    @Override
+    public boolean hasNext() {
       return mapIdx < mapIds.length && reduceIdx < reduceIds[mapIdx].length;
     }
 

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org