[jira] [Assigned] (SPARK-58165) SubqueryExpression drops nested subquery plans in IN values, breaking IN-over-subquery on Spark Connect

["Dongjoon Hyun (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-58165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dongjoon Hyun reassigned SPARK-58165:
-------------------------------------

    Assignee: Yang Jie

> SubqueryExpression drops nested subquery plans in IN values, breaking IN-over-subquery on Spark Connect
> -------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-58165
>                 URL: https://issues.apache.org/jira/browse/SPARK-58165
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.1.2
>            Reporter: Yang Jie
>            Assignee: Yang Jie
>            Priority: Major
>              Labels: pull-request-available
>
> {{SubqueryExpression}} (sql-api ColumnNode) returns {{children = Seq.empty}} even when its
> {{SubqueryType}} is {{IN(values)}}, whose {{values}} may hold nested {{ColumnNode}}s --
> including another {{SubqueryExpression}}. A nested subquery is reachable through the public
> 4.1 API, e.g.
> {code:scala}
> t1.select(max($"v")).scalar().isin(t2.select($"c"))
> // WHERE (SELECT max(v) FROM t1) IN (SELECT c FROM t2)
> {code}
> On Spark Connect, {{SparkSession.newDataset}} collects subquery plan references via
> {{node.collect \{ case _: SubqueryExpression => ... \}}}, which recurses through
> {{children}}. Because {{children}} is empty, the inner scalar subquery's plan is never added
> to {{WithRelations}}, while the proto converter still emits a planId reference to it. The
> server then fails resolving that planId:
> {noformat}
> [CONNECT_INVALID_PLAN.ASSERTION_FAILURE] Missing relation in WithRelations: (N) not in (M)
> {noformat}
> Spark Classic is unaffected: its converter embeds the subquery's logical plan directly and
> does not rely on {{children}}.
> Fix: {{SubqueryExpression.children}} now exposes {{IN.values}}, and {{normalize()}} is
> overridden to mirror sibling nodes (strip {{origin}}, normalize {{values}}).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org