You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "XiDuo You (Jira)" <ji...@apache.org> on 2021/07/12 04:05:00 UTC

[jira] [Created] (SPARK-36085) Make broadcast query stage executionContext isolation from AQE

XiDuo You created SPARK-36085:
---------------------------------

             Summary: Make broadcast query stage executionContext isolation from AQE
                 Key: SPARK-36085
                 URL: https://issues.apache.org/jira/browse/SPARK-36085
             Project: Spark
          Issue Type: Sub-task
          Components: SQL
    Affects Versions: 3.1.2, 3.2.0, 3.3.0
            Reporter: XiDuo You


`AdaptiveSparkPlanExec` shares the execution context with `BroadcastQueryStage`, but it has a small number thread (hard code 16). Unfortunately, it's not enough in some complex queries and no thread is available to cancel the broadcast stage.

 

In normal code path (without AQE), broadcast has it's own execution context with a config `spark.sql.broadcastExchange.maxThreadThreshold`. It should make sense to make broadcast query stage executionContext isolation from AQE and respect the existed thread threshold config.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org