[GitHub] [spark] otterc commented on a diff in pull request #41489: [SPARK-43987][Shuffle] Separate finalizeShuffleMerge Processing to Dedicated Thread Pools

["otterc (via GitHub)" <gi...@apache.org>]

otterc commented on code in PR #41489:
URL: https://github.com/apache/spark/pull/41489#discussion_r1221921518


##########
common/network-common/src/main/java/org/apache/spark/network/protocol/MessageWithHeader.java:
##########
@@ -35,7 +35,8 @@
  *
  * The header must be a ByteBuf, while the body can be a ByteBuf or a FileRegion.
  */
-class MessageWithHeader extends AbstractFileRegion {
+
+public class MessageWithHeader extends AbstractFileRegion {

Review Comment:
   Note for reviewers: Need to make this public for the UTs added for overridden decoder in `ShuffleTransportContext`



##########
common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java:
##########
@@ -324,6 +326,28 @@ public boolean separateChunkFetchRequest() {
     return conf.getInt("spark.shuffle.server.chunkFetchHandlerThreadsPercent", 0) > 0;
   }
 
+  /**
+   * Number of threads to process FinalizeShuffleMerge. Shuffle server will use a separate
+   * EventLoopGroup to process FinalizeShuffleMerge messages, which are I/O intensive and
+   * could take long time to process due to disk contentions.
+   */
+  public int finalizeShuffleMergeHandlerThreads() {
+    if (!this.getModuleName().equalsIgnoreCase("shuffle")) {
+      return 0;
+    }
+    Preconditions.checkArgument(separateFinalizeShuffleMerge(),
+      "Please set spark.shuffle.server.finalizeShuffleMergeThreads to a positive value");
+    return Integer.parseInt(conf.get("spark.shuffle.server.finalizeShuffleMergeThreads"));
+  }
+

Review Comment:
   Should this be a percentage like chunkFetchHandlerThreads config?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org