You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "wuyi (Jira)" <ji...@apache.org> on 2021/08/17 02:02:00 UTC
[jira] [Commented] (SPARK-36530) Avoid finalizing when there's no
push at all in a shuffle
[ https://issues.apache.org/jira/browse/SPARK-36530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17400089#comment-17400089 ]
wuyi commented on SPARK-36530:
------------------------------
cc [~mridulm80] [~mshen] any thoughts?
> Avoid finalizing when there's no push at all in a shuffle
> ---------------------------------------------------------
>
> Key: SPARK-36530
> URL: https://issues.apache.org/jira/browse/SPARK-36530
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 3.2.0, 3.3.0
> Reporter: wuyi
> Priority: Major
>
> When all partition data of a map output is bigger than spark.shuffle.push.maxBlockSizeToPush, there will be no push. When all map outputs don't have push, the shuffle doesn't have push. In that case, we don't need to launch the finalizing request.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org