[jira] [Updated] (SPARK-46995) Allow AQE coalesce final stage in SQL cached plan

["Ziqi Liu (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-46995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ziqi Liu updated SPARK-46995:
-----------------------------
    Component/s: SQL

> Allow AQE coalesce final stage in SQL cached plan
> -------------------------------------------------
>
>                 Key: SPARK-46995
>                 URL: https://issues.apache.org/jira/browse/SPARK-46995
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core, SQL
>    Affects Versions: 4.0.0
>            Reporter: Ziqi Liu
>            Priority: Major
>
> [https://github.com/apache/spark/pull/43435] and [https://github.com/apache/spark/pull/43760] are fixing a correctness issue which will be triggered when AQE applied on cached query plan, specifically, when AQE coalescing the final result stage of the cached plan.
>  
> The current semantic of {{spark.sql.optimizer.canChangeCachedPlanOutputPartitioning}}
> ([source code|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala#L403-L411]):
>  * when true, we enable AQE, but disable coalescing final stage ({*}default{*})
>  * when false, we disable AQE
>  
> But let’s revisit the semantic of this config: actually for caller the only thing that matters is whether we change the output partitioning of the cached plan. And we should only try to apply AQE if possible.  Thus we want to modify the semantic of {{spark.sql.optimizer.canChangeCachedPlanOutputPartitioning}}
>  * when true, we enable AQE and allow coalescing final: this might lead to perf regression, because it introduce extra shuffle
>  * when false, we enable AQE, but disable coalescing final stage. *(this is actually the `true` semantic of old behavior)*
> Also, to keep the default behavior unchanged, we might want to flip the default value of {{spark.sql.optimizer.canChangeCachedPlanOutputPartitioning}} to `false`
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org