You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Wei Xue (Jira)" <ji...@apache.org> on 2022/06/22 02:50:00 UTC

[jira] [Created] (SPARK-39551) Add AQE invalid plan check

Wei Xue created SPARK-39551:
-------------------------------

             Summary: Add AQE invalid plan check
                 Key: SPARK-39551
                 URL: https://issues.apache.org/jira/browse/SPARK-39551
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.3.0
            Reporter: Wei Xue


AQE logical optimization rules can lead to invalid physical plans as certain physical plan nodes are not compatible with others. E.g., `BroadcastExchangeExec` can only work as a direct child of broadcast join nodes.

Logical optimizations, on the other hand, are not (and should not be) aware of such restrictions. So a general solution here is to check for invalid plans and throw exceptions, which can be caught by AQE replanning process. And if such an exception is captured, AQE can void the current replanning result and keep using the latest valid plan.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org