[GitHub] [spark] otterc commented on a change in pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

[GitBox <gi...@apache.org>]

otterc commented on a change in pull request #30062:
URL: https://github.com/apache/spark/pull/30062#discussion_r507985489



##########
File path: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
##########
@@ -363,4 +363,26 @@ public boolean useOldFetchProtocol() {
     return conf.getBoolean("spark.shuffle.useOldFetchProtocol", false);
   }
 
+  /**
+   * The minimum size of a chunk when dividing a merged shuffle file into multiple chunks during
+   * push-based shuffle.
+   * A merged shuffle file consists of multiple small shuffle blocks. Fetching the
+   * complete merged shuffle file in a single response increases the memory requirements for the

Review comment:
       Are you referring to the configuration `maxRemoteBlockSizeFetchToMem`? 
   We are aware that when this configuration is set and if a request is larger than this, the block will be saved to disk. 
   With push-based shuffle, data of a remote merged block is always large. If we don't divide it into chunks, the remote merged data will always be written to disk and then read from it again. This adds a lot more time.
   Also any failure during fetching an entire merged block will be much more costly. With the approach of dividing a merged block into size-able chunks
   - We don't have to write to the disk always so the runtime of jobs are shorter.
   - When fetch of a shuffle chunk fails, then we fallback to the original blocks corresponding to the mapIds which are part of this chunk.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org