You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "wuyi (Jira)" <ji...@apache.org> on 2021/08/17 02:01:00 UTC

[jira] [Created] (SPARK-36530) Avoid finalizing when there's no push at all in a shuffle

wuyi created SPARK-36530:
----------------------------

             Summary: Avoid finalizing when there's no push at all in a shuffle
                 Key: SPARK-36530
                 URL: https://issues.apache.org/jira/browse/SPARK-36530
             Project: Spark
          Issue Type: Sub-task
          Components: Spark Core
    Affects Versions: 3.2.0, 3.3.0
            Reporter: wuyi


When all partition data of a map output is bigger than spark.shuffle.push.maxBlockSizeToPush, there will be no push. When all map outputs don't have push, the shuffle doesn't have push. In that case, we don't need to launch the finalizing request.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org