[jira] [Resolved] (SPARK-39551) Add AQE invalid plan check

["Wenchen Fan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-39551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wenchen Fan resolved SPARK-39551.
---------------------------------
    Fix Version/s: 3.3.1
                   3.4.0
       Resolution: Fixed

Issue resolved by pull request 36953
[https://github.com/apache/spark/pull/36953]

> Add AQE invalid plan check
> --------------------------
>
>                 Key: SPARK-39551
>                 URL: https://issues.apache.org/jira/browse/SPARK-39551
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 3.3.0
>            Reporter: Wei Xue
>            Assignee: Wei Xue
>            Priority: Minor
>             Fix For: 3.3.1, 3.4.0
>
>
> AQE logical optimization rules can lead to invalid physical plans as certain physical plan nodes are not compatible with others. E.g., `BroadcastExchangeExec` can only work as a direct child of broadcast join nodes.
> Logical optimizations, on the other hand, are not (and should not be) aware of such restrictions. So a general solution here is to check for invalid plans and throw exceptions, which can be caught by AQE replanning process. And if such an exception is captured, AQE can void the current replanning result and keep using the latest valid plan.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org