[jira] [Resolved] (SPARK-41441) Allow Generate with no required child output to host outer references

["Wenchen Fan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-41441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wenchen Fan resolved SPARK-41441.
---------------------------------
    Fix Version/s: 3.4.0
       Resolution: Fixed

Issue resolved by pull request 38968
[https://github.com/apache/spark/pull/38968]

> Allow Generate with no required child output to host outer references
> ---------------------------------------------------------------------
>
>                 Key: SPARK-41441
>                 URL: https://issues.apache.org/jira/browse/SPARK-41441
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>    Affects Versions: 3.4.0
>            Reporter: Allison Wang
>            Assignee: Allison Wang
>            Priority: Major
>             Fix For: 3.4.0
>
>
> Currently, in CheckAnalysis, Spark disallows Generate to host any outer references when it's required child output is not empty. But when the child output is empty, it can host outer references, which DecorrelateInnerQuery does not handle.
> For example,
> {code:java}
> select * from t, lateral (select explode(array(c1, c2))){code}
> This throws an internal error :
> {code:java}
> Caused by: java.lang.AssertionError: assertion failed: Correlated column is not allowed in Generate explode(array(outer(c1#219), outer(c2#220))), false, [col#221] +- OneRowRelation{code}
>  We should support Generate to host outer references when its required child output is empty.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org