You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2021/05/16 15:48:00 UTC

[jira] [Assigned] (SPARK-35414) Completely fix the broadcast timeout issue in AQE

     [ https://issues.apache.org/jira/browse/SPARK-35414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apache Spark reassigned SPARK-35414:
------------------------------------

    Assignee: Yu Zhong  (was: Apache Spark)

> Completely fix the broadcast timeout issue in AQE
> -------------------------------------------------
>
>                 Key: SPARK-35414
>                 URL: https://issues.apache.org/jira/browse/SPARK-35414
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>    Affects Versions: 3.0.0, 3.0.1
>            Reporter: Yu Zhong
>            Assignee: Yu Zhong
>            Priority: Major
>
> SPARK-33933 report a issue that in AQE, when the resources is limited, broadcast timeout could happened. 
> [#31269|https://github.com/apache/spark/pull/31269] gives a partial fix by reorder newStages by class type to make sure BroadcastQueryState precede others when calling materialized(). However, it only guarantee that the order of task to be scheduled in normal circumstances, but, the guarantee is not strict since the submit of broadcast job and shuffle map job are in different thread.
> So we need a completely fix to avoid the edge case triggering broadcast timeout.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org