[spark] branch master updated: [SPARK-38344][SHUFFLE] Avoid to submit task when there are no requests to push up in push-based shuffle

[mr...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

mridulm80 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 96bcb04  [SPARK-38344][SHUFFLE] Avoid to submit task when there are no requests to push up in push-based shuffle
96bcb04 is described below

commit 96bcb0406ea229b1c5ecbb98a1de6ba0d6a238b4
Author: weixiuli <we...@jd.com>
AuthorDate: Tue Mar 1 20:20:30 2022 -0600

    [SPARK-38344][SHUFFLE] Avoid to submit task when there are no requests to push up in push-based shuffle
    
    ### What changes were proposed in this pull request?
    
    Avoid to submit task when there are no requests to push up in push-based shuffle.
    
    ### Why are the changes needed?
    This is a performance improvement to the existing functionality.
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    Pass GA.
    Existing unittests.
    
    Closes #35675 from weixiuli/SPARK-38344-SBS.
    
    Authored-by: weixiuli <we...@jd.com>
    Signed-off-by: Mridul Muralidharan <mridul<at>gmail.com>
---
 .../main/scala/org/apache/spark/shuffle/ShuffleBlockPusher.scala  | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPusher.scala b/core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPusher.scala
index d6972cd..230ec7e 100644
--- a/core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPusher.scala
+++ b/core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPusher.scala
@@ -118,11 +118,11 @@ private[spark] class ShuffleBlockPusher(conf: SparkConf) extends Logging {
     pushRequests ++= Utils.randomize(requests)
     if (pushRequests.isEmpty) {
       notifyDriverAboutPushCompletion()
+    } else {
+      submitTask(() => {
+        tryPushUpToMax()
+      })
     }
-
-    submitTask(() => {
-      tryPushUpToMax()
-    })
   }
 
   private[shuffle] def tryPushUpToMax(): Unit = {

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org