You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2020/12/09 20:30:00 UTC

[jira] [Assigned] (SPARK-32920) Add support in Spark driver to coordinate the finalization of the push/merge phase in push-based shuffle for a given shuffle and the initiation of the reduce stage

     [ https://issues.apache.org/jira/browse/SPARK-32920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apache Spark reassigned SPARK-32920:
------------------------------------

    Assignee:     (was: Apache Spark)

> Add support in Spark driver to coordinate the finalization of the push/merge phase in push-based shuffle for a given shuffle and the initiation of the reduce stage
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-32920
>                 URL: https://issues.apache.org/jira/browse/SPARK-32920
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Shuffle, Spark Core
>    Affects Versions: 3.1.0
>            Reporter: Min Shen
>            Priority: Major
>
> With push-based shuffle, we are currently decoupling map task executions from the shuffle block push process. Thus, when all map tasks finish, we might want to wait for some small extra time to allow more shuffle blocks to get pushed and merged. This requires some extra coordination in the Spark driver when it transitions from a shuffle map stage to the corresponding reduce stage.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org